“An Omnistereoscopic Video Pipeline for Capture and Display of Real-World VR” by Schroers, Bazin and Sorkine-Hornung

  • ©

Conference:


Type(s):


Title:

    An Omnistereoscopic Video Pipeline for Capture and Display of Real-World VR

Session/Category Title:   Computational Photos and Videos

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    In this article, we describe a complete pipeline for the capture and display of real-world Virtual Reality video content, based on the concept of omnistereoscopic panoramas. We address important practical and theoretical issues that have remained undiscussed in previous works. On the capture side, we show how high-quality omnistereo video can be generated from a sparse set of cameras (16 in our prototype array) instead of the hundreds of input views previously required. Despite the sparse number of input views, our approach allows for high quality, real-time virtual head motion, thereby providing an important additional cue for immersive depth perception compared to static stereoscopic video. We also provide an in-depth analysis of the required camera array geometry in order to meet specific stereoscopic output constraints, which is fundamental for achieving a plausible and fully controlled VR viewing experience. Finally, we describe additional insights on how to integrate omnistereo video panoramas with rendered CG content. We provide qualitative comparisons to alternative solutions, including depth-based view synthesis and the Facebook Surround 360 system. In summary, this article provides a first complete guide and analysis for reimplementing a system for capturing and displaying real-world VR, which we demonstrate on several real-world examples captured with our prototype.

References:


    • Aseem Agarwala, Maneesh Agrawala, Michael Cohen, David Salesin, and Richard Szeliski. 2006. Photographing long scenes with multi-viewpoint panoramas. TOG (SIGGRAPH) 25, 3 (2006), 853–861. Google ScholarDigital Library
    • Robert Anderson, David Gallup, Jonathan T. Barron, Janne Kontkanen, Noah Snavely, Carlos Hernandez Esteban, Sameer Agarwal, and Steven M. Seitz. 2016. Jump: Virtual reality video. TOG (SIGGRAPH Asia) 35, 6 (2016), 198:1–198:13. http://dl.acm.org/citation.cfm?id=2980257. Google ScholarDigital Library
    • Clemens Birklbauer and Oliver Bimber. 2014. Panorama light-field imaging. CGF (Eurographics) 33, 2 (2014), 43–52. Google ScholarDigital Library
    • Matthew Brown and David G. Lowe. 2007. Automatic panoramic image stitching using invariant features. IJCV 74, 1 (2007), 59–73. Google ScholarDigital Library
    • Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert. 2004. High accuracy optical flow estimation based on a theory for warping. In ECCV. 25–36.Google Scholar
    • Vincent Chapdelaine-Couture and Sébastien Roy. 2013. The omnipolar camera: A new approach to stereo immersive capture. In ICCP. 1–9.Google Scholar
    • Vincent Couture, Michael S. Langer, and Sébastien Roy. 2011. Panoramic stereo video textures. In ICCV. 1251–1258. Google ScholarDigital Library
    • Alexei A. Efros and William T. Freeman. 2001. Image quilting for texture synthesis and transfer. In SIGGRAPH. 341–346. http://portal.acm.org/citation.cfm?id=383259.383296 Google ScholarDigital Library
    • Simon Fuhrmann, Fabian Langguth, and Michael Goesele. 2014. MVE — A multi-view reconstruction environment. In Eurographics Workshop on Graphics and Cultural Heritage. Google ScholarDigital Library
    • Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multiview stereopsis. TPAMI 32, 8 (2010), 1362–1376. Google ScholarDigital Library
    • Silvano Galliani, Katrin Lasinger, and Konrad Schindler. 2015. Massively parallel multiview stereopsis by surface normal diffusion. In ICCV. 873–881. Google ScholarDigital Library
    • Rajiv Gupta and Richard I. Hartley. 1997. Linear pushbroom cameras. TPAMI 19, 9 (1997), 963–975. Google ScholarDigital Library
    • Richard Hartley and Andrew Zisserman. 2004. Multiple View Geometry in Computer Vision. Cambridge University Press. Google ScholarDigital Library
    • Peter Hedman, Suhib Alsisan, Richard Szeliski, and Johannes Kopf. 2017. Casual 3D photography. TOG (SIGGRAPH Asia) 36, 6 (2017), 234:1–234:15. Google ScholarDigital Library
    • Heiko Hirschmüller. 2008. Stereo processing by semiglobal matching and mutual information. TPAMI 30, 2 (2008), 328–341. Google ScholarDigital Library
    • Hiroshi Ishiguro, Masashi Yamamoto, and Saburo Tsuji. 1992. Omni-directional stereo. TPAMI 14, 2 (1992), 257–262. Google ScholarDigital Library
    • Jiaya Jia and Chi-Keung Tang. 2008. Image stitching using structure deformation. TPAMI 30, 4 (2008), 617–631. Google ScholarDigital Library
    • Sing Bing Kang, Richard Szeliski, and Matthew Uyttendaele. 2004. Seamless Stitching using Multi-Perspective Plane Sweep. Technical Report MSR-TR-2004-48. Microsoft Research.Google Scholar
    • Johannes Kopf, Billy Chen, Richard Szeliski, and Michael Cohen. 2010. Street slide: Browsing street level imagery. TOG (SIGGRAPH) 29, 4 (2010), 96:1–8. Google ScholarDigital Library
    • Johannes Kopf, Matthew Uyttendaele, Oliver Deussen, and Michael F. Cohen. 2007. Capturing and viewing gigapixel images. TOG (SIGGRAPH) 26, 3 (2007), 93. Google ScholarDigital Library
    • Jungjin Lee, Bumki Kim, Kyehyun Kim, Younghui Kim, and Jun-yong Noh. 2016. Rich360: Optimized spherical representation from structured panoramic camera arrays. TOG (SIGGRAPH) 35, 4 (2016), 63. Google ScholarDigital Library
    • Kevin Matzen, Michael F. Cohen, Bryce Evans, Johannes Kopf, and Richard Szeliski. 2017. Low-cost 360 stereo photography and video capture. ACM Trans. Graph. 36, 4, Article 148 (2017), 12 pages. Google ScholarDigital Library
    • Shmuel Peleg and Moshe Ben-Ezra. 1999. Stereo panorama with a single camera. In CVPR. 1395–1401.Google Scholar
    • Shmuel Peleg, Moshe Ben-Ezra, and Yael Pritch. 2001. Omnistereo: Panoramic stereo imaging. TPAMI 23, 3 (2001), 279–290. Google ScholarDigital Library
    • Federico Perazzi, Alexander Sorkine-Hornung, Henning Zimmer, Peter Kaufmann, Oliver Wang, S. Watson, and Markus H. Gross. 2015. Panoramic video from unstructured camera arrays. CGF (Eurographics) 34, 2 (2015), 57–68. Google ScholarDigital Library
    • Alex Rav-Acha, Giora Engel, and Shmuel Peleg. 2008. Minimal aspect distortion (MAD) mosaicing of long scenes. IJCV 78, 2–3 (2008), 187–206. Google ScholarDigital Library
    • Christian Richardt, Yael Pritch, Henning Zimmer, and Alexander Sorkine-Hornung. 2013. Megastereo: Constructing high-resolution stereo panoramas. In CVPR. 1256–1263. Google ScholarDigital Library
    • Augusto Román and Hendrik P. A. Lensch. 2006. Automatic multiperspective images. In Eurographics Symposium on Rendering Techniques (EGSR). 83–92. Google ScholarDigital Library
    • Steven M. Seitz and Jiwon Kim. 2002. The space of all stereo images. IJCV 48, 1 (2002), 21–38. Google ScholarDigital Library
    • Heung-Yeung Shum and Li-wei He. 1999. Rendering with concentric mosaics. In SIGGRAPH. 299–306. Google ScholarDigital Library
    • Harry Shum and Sing Bing Kang. 2000. Review of image-based rendering techniques. In Visual Communications and Image Processing. 2–13.Google Scholar
    • Heung-Yeung Shum, King To Ng, and Shing-Chow Chan. 2005. A virtual reality system using the concentric mosaic: Construction, rendering, and data compression. IEEE Transactions on Multimedia 7, 1 (2005), 85–95. Google ScholarDigital Library
    • Heung-Yeung Shum and Richard Szeliski. 2000. Systems and experiment paper: Construction of panoramic image mosaics with global and local alignment. IJCV 36, 2 (2000), 101–130. Google ScholarDigital Library
    • Andreas Simon, Randall C. Smith, and Richard R. Pawlicki. 2004. Omnistereo for panoramic virtual environment display systems. In IEEE VR. 67–74. Google ScholarDigital Library
    • Richard Szeliski. 2006. Image alignment and stitching: A tutorial. Foundations and Trends in Computer Graphics and Vision 2, 1 (2006), 1–104. Google ScholarDigital Library
    • Fan Zhang and Feng Liu. 2014. Parallax-tolerant image stitching. In CVPR. 3262–3269. Google ScholarDigital Library
    • Fan Zhang and Feng Liu. 2015. Casual stereoscopic panorama stitching. In CVPR. 2002–2010.Google Scholar
    • Guofeng Zhang, Jiaya Jia, Tien-Tsin Wong, and Hujun Bao. 2009. Consistent depth maps recovery from a video sequence. TPAMI 31, 6 (2009), 974–988. Google ScholarDigital Library
    • Zhengyou Zhang. 2000. A flexible new technique for camera calibration. TPAMI 22, 11 (2000), 1330–1334. Google ScholarDigital Library
    • Enliang Zheng, Rahul Raguram, Pierre Fite Georgel, and Jan-Michael Frahm. 2011. Efficient generation of multi-perspective panoramas. In International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT). 86–92. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: