“SpinVR: towards live-streaming 3D virtual reality video”
Conference:
Type(s):
Title:
- SpinVR: towards live-streaming 3D virtual reality video
Session/Category Title: AR/VR
Presenter(s)/Author(s):
Abstract:
Streaming of 360° content is gaining attention as an immersive way to remotely experience live events. However live capture is presently limited to 2D content due to the prohibitive computational cost associated with multi-camera rigs. In this work we present a system that directly captures streaming 3D virtual reality content. Our approach does not suffer from spatial or temporal seams and natively handles phenomena that are challenging for existing systems, including refraction, reflection, transparency and speculars. Vortex natively captures in the omni-directional stereo (ODS) format, which is widely supported by VR displays and streaming pipelines. We identify an important source of distortion inherent to the ODS format, and demonstrate a simple means of correcting it. We include a detailed analysis of the design space, including tradeoffs between noise, frame rate, resolution, and hardware complexity. Processing is minimal, enabling live transmission of immersive, 3D, 360° content. We construct a prototype and demonstrate capture of 360° scenes at up to 8192 X 4096 pixels at 5 fps, and establish the viability of operation up to 32 fps.
References:
1. Michael Adam, Christoph Jung, Stefan Roth, and Guido Brunnett. 2009. Real-time Stereo-Image Stitching using GPU-based Belief Propagation. In Vision, Modeling, and Visualization Workshop (VMV). 215–224.
2. Rajat Aggarwal, Amrisha Vohra, and Anoop M. Namboodiri. 2016. Panoramic Stereo Videos With a Single Camera. In Proc. IEEE CVPR.
3. Robert Anderson, David Gallup, Jonathan T. Barron, Janne Kontkanen, Noah Snavely, Carlos Hernández, Sameer Agarwal, and Steven M. Seitz. 2016. Jump: Virtual Reality Video. ACM Trans. Graph. (SIGGRAPH Asia) 35, 6 (2016), 198:1–198:13.
4. Robert G. Batchko. 1994. Three-hundred-sixty degree electroholographic stereogram and volumetric display system. In Proc. SPIE, Vol. 2176. 30–41. Cross Ref
5. R. Benosman, T. Maniere, and J. Devars. 1996. Multidirectional stereovision sensor, calibration and scenes reconstruction. In Proc. ICPR, Vol. 1. 161–165.
6. P. Bourke. 2010. Capturing omni-directional stereoscopic spherical projections with a single camera. In Proc. IEEE VSMM. 179–183. Cross Ref
7. Matthew Brown and David G. Lowe. 2007. Automatic Panoramic Image Stitching Using Invariant Features. IJCV 74, 1 (2007), 59–73.
8. Antoni Buades and Jean-Michel Morel. 2005. A non-local algorithm for image denoising. In Proc. IEEE CVPR.
9. V. Chapdelaine-Couture and S. Roy. 2013. The omnipolar camera: A new approach to stereo immersive capture. In Proc. ICCP. 1–9.
10. Oliver Cossairt, Mohit Gupta, and Shree K. Nayar. 2013. When Does Computational Imaging Improve Performance? IEEE Trans. Im. Proc. 22, 2 (2013), 447–458.
11. Oliver S. Cossairt, Joshua Napoli, Samuel L. Hill, Rick K. Dorval, and Gregg E. Favalora. 2007. Occlusion-capable multiview volumetric three-dimensional display. OSA Appl. Opt. 46, 8 (2007), 1244–1250. Cross Ref
12. Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007. Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering. IEEE Trans. Im. Proc. 16, 8 (2007), 2080–2095.
13. Donald G. Dansereau, Glenn Schuster, Joseph Ford, and Gordon Wetzstein. 2017. A Wide-Field-of-View Monocentric Light Field Camera. In Computer Vision and Pattern Recognition (CVPR). IEEE.
14. Ho-Chao Huang and Yi-Ping Hung. 1998. Panoramic Stereo Imaging System with
15. Automatic Disparity Warping and Seaming. Graph. Models Image Process. 60, 3 (May 1998), 196–208.
16. H. Ishiguro, M. Yamamoto, and S. Tsuji. 1990. Omni-directional stereo for making global map. In Proc. ICCV. 540–547.
17. Andrew Jones, Ian McDowall, Hideshi Yamada, Mark Bolas, and Paul Debevec. 2007. Rendering for an Interactive 360&Deg; Light Field Display. ACM Trans. Graph. (SIGGRAPH) 26, 3 (2007).
18. Kevin Matzen, Michael F. Cohen, Bryce Evans, Johannes Kopf, and Richard Szeliski. 2017. Low-cost 360 Stereo Photography and Video Capture. ACM Trans. Graph. 36, 4 (July 2017), 148:1–148:12.
19. David W. Murray. 1995. Recovering Range Using Virtual Multicamera Stereo. Proc. CVIU 61, 2 (1995), 285 — 291.
20. S. Peleg, M. Ben-Ezra, and Y. Pritch. 2001. Omnistereo: panoramic stereo imaging. IEEE Trans. PAMI 23, 3 (2001), 279–290.
21. C. Richardt, Y. Pritch, H. Zimmer, and A. Sorkine-Hornung. 2013. Megastereo: Constructing High-Resolution Stereo Panoramas. In Proc. IEEE CVPR. 1256–1263.
22. Heung-Yeung Shum and Li-Wei He. 1999. Rendering with Concentric Mosaics. In Proc. SIGGRAPH. 299–306.
23. Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, and Gordon Wetzstein. 2016. Saliency in VR: How do people explore virtual environments?. In arXiv:1612.04335.
24. Richard Szeliski. 2010. Computer Vision: Algorithms and Applications. Springer.
25. Kenji Tanaka, Junya Hayashi, Masahiko Inami, and Susumu Tachi. 2004. TWISTER: An immersive autostereoscopic display. In Proc. IEEE VR. 59–66.
26. K. Tanaka and S. Tachi. 2005. TORNADO: omnistereo video imaging with rotating optics. IEEE TVCG 11, 6 (2005), 614–625.
27. Christian Weissig, Oliver Schreer, Peter Eisert, and Peter Kauff. 2012. The Ultimate Immersive Experience: Panoramic 3D Video Acquisition. In Proc. MMM. 671–681.
