“A system for acquiring, processing, and rendering panoramic light field stills for virtual reality”
Conference:
Type(s):
Title:
- A system for acquiring, processing, and rendering panoramic light field stills for virtual reality
Session/Category Title: Acquisition, rendering and display for virtual reality
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We present a system for acquiring, processing, and rendering panoramic light field still photography for display in Virtual Reality (VR). We acquire spherical light field datasets with two novel light field camera rigs designed for portable and efficient light field acquisition. We introduce a novel real-time light field reconstruction algorithm that uses a per-view geometry and a disk-based blending field. We also demonstrate how to use a light field prefiltering operation to project from a high-quality offline reconstruction model into our real-time model while suppressing artifacts. We introduce a practical approach for compressing light fields by modifying the VP9 video codec to provide high quality compression with real-time, random access decompression.We combine these components into a complete light field system offering convenient acquisition, compact file size, and high-quality rendering while generating stereo views at 90Hz on commodity VR hardware. Using our system, we built a freely available light field experience application called Welcome to Light Fields featuring a library of panoramic light field stills for consumer VR which has been downloaded over 15,000 times.
References:
1. Robert Anderson, David Gallup, Jonathan T. Barron, Janne Kontkanen, Noah Snavely, Carlos Hernández, Sameer Agarwal, and Steven M. Seitz. 2016. Jump: Virtual Reality Video. ACM Trans. Graph. 35, 6 (Nov. 2016), 198:1–198:13. Google ScholarDigital Library
2. Jonathan T Barron, Andrew Adams, YiChang Shih, and Carlos Hernández. 2015. Fast Bilateral-Space Stereo for Synthetic Defocus. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
3. Clemens Birklbauer and Oliver Bimber. 2014. Panorama light-field imaging. Computer Graphics Forum 33, 2 (2014), 43–52. Google ScholarDigital Library
4. Clemens Birklbauer and Oliver Bimber. 2015. Active guidance for light-field photography on smartphones. Computers & Graphics 53 (2015), 127–135. Google ScholarDigital Library
5. Clemens Birklbauer, Simon Opelt, and Oliver Bimber. 2013. Rendering gigaray light fields. Computer Graphics Forum 32, 2pt4 (2013), 469–478.Google Scholar
6. Chip Brown. 2017. Bringing pixels front and center in VR video. (March 2017). https://blog.google/products/google-vr/bringing-pixels-front-and-center-vr-video/Google Scholar
7. Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001. Unstructured lumigraph rendering. In Proceedings of SIGGRAPH 2001. 425–432. Google ScholarDigital Library
8. Fernando Cacciola. 2018. Triangulated Surface Mesh Simplification. In CGAL User and Reference Manual (4.12 ed.). CGAL Editorial Board. https://doc.cgal.org/4.12/Manual/packages.html#PkgSurfaceMeshSimplificationSummaryGoogle Scholar
9. CapturingReality. 2018. (2018). https://www.capturingreality.com/ProductGoogle Scholar
10. Jin-Xiang Chai, Xin Tong, Shing-Chow Chan, and Heung-Yeung Shum. 2000. Plenoptic sampling. In Proceedings of SIGGRAPH 2000. 307–318. Google ScholarDigital Library
11. Gaurav Chaurasia, Sylvain Duchene, Olga Sorkine-Hornung, and George Drettakis. 2013. Depth synthesis and local warps for plausible image-based navigation. ACM Trans. Graph. 32, 3 (July 2013), 30:1–30:12. Google ScholarDigital Library
12. Wei-Chao Chen, Jean-Yves Bouguet, Michael H Chu, and Radek Grzeszczuk. 2002. Light field mapping: efficient representation and hardware rendering of surface light fields. ACM Trans. Graph. 21, 3 (July 2002), 447–456. Google ScholarDigital Library
13. Feng Dai, Jun Zhang, Yike Ma, and Yongdong Zhang. 2015. Lenselet image compression scheme based on subaperture images streaming. In 2015 IEEE International Conference on Image Processing (ICIP). 4733–4737.Google ScholarDigital Library
14. Abe Davis, Marc Levoy, and Fredo Durand. 2012. Unstructured light fields. Computer Graphics Forum 31, 2pt1 (2012), 305–314. Google ScholarDigital Library
15. Paul Debevec, Greg Downing, Mark Bolas, Hsuen-Yueh Peng, and Jules Urbach. 2015. Spherical Light Field Environment Capture for Virtual Reality Using a Motorized Pan/Tilt Head and Offset Camera. In ACM SIGGRAPH 2015 Posters. 30:1–30:1. Google ScholarDigital Library
16. Paul Debevec, Yizhou Yu, and George Borshukov. 1998. Efficient view-dependent image-based rendering with projective texture-mapping. In Rendering Techniques ’98. 105–116.Google ScholarCross Ref
17. Martin Eisemann, Bert De Decker, Marcus Magnor, Philippe Bekaert, Edilson De Aguiar, Naveed Ahmed, Christian Theobalt, and Anita Sellent. 2008. Floating textures. Computer Graphics Forum 27, 2 (2008), 409–418.Google ScholarCross Ref
18. Steven J Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F Cohen. 1996. The lumigraph. In Proceedings of SIGGRAPH 1996. 43–54. Google ScholarDigital Library
19. Markus Gross and Hanspeter Pfister. 2011. Point-based graphics. Morgan Kaufmann.Google Scholar
20. Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel Brostow. 2016. Scalable inside-out image-based rendering. ACM Trans. Graph. 35, 6 (Nov. 2016), 231:1–231:11. Google ScholarDigital Library
21. Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based view synthesis for light field cameras. ACM Trans. Graph. 35, 6 (Nov. 2016), 193:1–193:10. Google ScholarDigital Library
22. Robert Konrad, Donald G Dansereau, Aniq Masood, and Gordon Wetzstein. 2017. SpinVR: towards live-streaming 3D virtual reality video. ACM Trans. Graph. 36, 6 (Nov. 2017), 209:1–209:12. Google ScholarDigital Library
23. Douglas Lanman and David Luebke. 2013. Near-eye light field displays. ACM Trans. Graph. 32, 6 (Nov. 2013), 220:1–220:10. Google ScholarDigital Library
24. Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of SIGGRAPH 1996. 31–42. Google ScholarDigital Library
25. Zhouchen Lin and Heung-Yeung Shum. 2004. A geometric analysis of light field rendering. International Journal of Computer Vision 58, 2 (Jul 2004), 121–138. Google ScholarDigital Library
26. Peter Lindstrom and Greg Turk. 1998. Fast and memory efficient polygonal simplification. In Proceedings of Visualization’98. 279–286. Google ScholarDigital Library
27. Peter Lindstrom and Greg Turk. 1999. Evaluation of memoryless simplification. IEEE Transactions on Visualization & Computer Graphics 5, 2 (1999), 98–115. Google ScholarDigital Library
28. M. Lukacs. 1986. Predictive coding of multi-viewpoint image sets. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP ’86), Vol. 11. 521–524.Google ScholarCross Ref
29. Marcus Magnor and Bernd Girod. 2000. Data compression for light-field rendering. IEEE Transactions on Circuits and Systems for Video Technology 10, 3 (2000), 338–343. Google ScholarDigital Library
30. Tim Milliron, Chrissy Szczupak, and Orin Green. 2017. Hallelujah: the world’s first lytro VR experience. In ACM SIGGRAPH 2017 VR Village. 7:1–7:2. Google ScholarDigital Library
31. Debargha Mukherjee, Jim Bankoski, Adrian Grange, Jingning Han, John Koleszar, Paul Wilkins, Yaowu Xu, and Ronald Bultje. 2013. The latest open-source video codec VP9 – An overview and preliminary results. In 2013 Picture Coding Symposium (PCS). 390–393.Google ScholarCross Ref
32. Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, and Pat Hanrahan. 2005. Light field photography with a hand-held plenoptic camera. Computer Science Technical Report CSTR 2, 11 (2005), 1–11.Google Scholar
33. Eric Penner and Li Zhang. 2017. Soft 3D Reconstruction for View Synthesis. ACM Trans. Graph. 36, 6 (Nov. 2017), 235:1–235:11. Google ScholarDigital Library
34. Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. 1998. Layered depth images. In Proceedings of SIGGRAPH 1998. 231–242. Google ScholarDigital Library
35. Lixin Shi, Haitham Hassanieh, Abe Davis, Dina Katabi, and Fredo Durand. 2014. Light field reconstruction using sparsity in the continuous fourier domain. ACM Trans. Graph. 34, 1 (Dec. 2014), 12:1–12:13. Google ScholarDigital Library
36. Heung-Yeung Shum and Li-Wei He. 1999. Rendering with concentric mosaics. In Proceedings of SIGGRAPH 1999. 299–306. Google ScholarDigital Library
37. Peter-Pike Sloan, Michael F Cohen, and Steven J Gortler. 1997. Time critical lumigraph rendering. In Proceedings of the 1997 Symposium on Interactive 3D Graphics. 17-ff. Google ScholarDigital Library
38. Gary J Sullivan, Jens Ohm, Woo-Jin Han, and Thomas Wiegand. 2012. Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on circuits and systems for video technology 22, 12 (2012), 1649–1668. Google ScholarDigital Library
39. Vaibhav Vaish and Andrew Adams. 2008. The (new) stanford light field archive. Computer Graphics Laboratory, Stanford University (2008). http://lightfield.stanford.edu/Google Scholar
40. Irene Viola, Martin Reřábek, and Touradj Ebrahimi. 2017. Comparison and evaluation of light field image coding approaches. IEEE Journal of Selected Topics in Signal Processing 11, 7 (2017), 1092–1106.Google ScholarCross Ref
41. WebM. 2016. (2016). https://www.webmproject.org/code/Google Scholar
42. WebP. 2016. (2016). https://developers.google.com/speed/webp/Google Scholar
43. Daniel N Wood, Daniel I Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H Salesin, and Werner Stuetzle. 2000. Surface light fields for 3D photography. In Proceedings of SIGGRAPH 2000. 287–296. Google ScholarDigital Library
44. Changchang Wu. 2013. Towards linear-time incremental structure from motion. In 2013 International Conference on 3D Vision. 127–134. Google ScholarDigital Library
45. Gaochang Wu, Belen Masia, Adrian Jarabo, Yuchen Zhang, Liangyong Wang, Qionghai Dai, Tianyou Chai, and Yebin Liu. 2017. Light field image processing: An overview. IEEE Journal of Selected Topics in Signal Processing 11, 7 (Oct 2017), 926–954.Google ScholarCross Ref
46. Cha Zhang and Tsuhan Chen. 2004. A survey on image-based rendering-representation, sampling and compression. Signal Processing: Image Communication 19, 1 (2004), 1–28.Google ScholarDigital Library
47. Cha Zhang and Jin Li. 2000. Compression of lumigraph with multiple reference frame (MRF) prediction and just-in-time rendering. In Proceedings of Data Compression Conference 2000. IEEE, 253–262. Google ScholarDigital Library
48. C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2004. High-quality video view interpolation using a layered representation. In ACM Trans. Graph., Vol. 23. 600–608. Google ScholarDigital Library
49. Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. 2001. Surface splatting. In Proceedings of SIGGRAPH 2001. 371–378. Google ScholarDigital Library
