“Neural 3D holography: learning accurate wave propagation models for 3D holographic virtual and augmented reality displays” by Choi, Gopakumar, Peng, Kim and Wetzstein
Conference:
Type(s):
Title:
- Neural 3D holography: learning accurate wave propagation models for 3D holographic virtual and augmented reality displays
Session/Category Title: Audio and Visual Displays
Presenter(s)/Author(s):
Abstract:
Holographic near-eye displays promise unprecedented capabilities for virtual and augmented reality (VR/AR) systems. The image quality achieved by current holographic displays, however, is limited by the wave propagation models used to simulate the physical optics. We propose a neural network-parameterized plane-to-multiplane wave propagation model that closes the gap between physics and simulation. Our model is automatically trained using camera feedback and it outperforms related techniques in 2D plane-to-plane settings by a large margin. Moreover, it is the first network-parameterized model to naturally extend to 3D settings, enabling high-quality 3D computer-generated holography using a novel phase regularization strategy of the complex-valued wave field. The efficacy of our approach is demonstrated through extensive experimental evaluation with both VR and optical see-through AR display prototypes.
References:
1. Eirikur Agustsson and Radu Timofte. 2017. NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. In CVPR.
2. Francis Bach, Rodolphe Jenatton, Julien Mairal, and Guillaume Obozinski. 2012. Optimization with Sparsity-Inducing Penalties. 4, 1 (2012), 1–106.
3. Yosuke Bando, Henry Holtzman, and Ramesh Raskar. 2013. Near-invariant blur for depth and 2D motion via time-varying light field analysis. ACM Trans. Graph. 32, 2 (2013), 1–15.
4. Stephen A. Benton. 1983. Survey Of Holographic Stereograms. In Proc. SPIE, Vol. 0367.
5. Stephen A. Benton and V. Michael Bove. 2008. Holographic Imaging. Wiley-Interscience.
6. Fergus W Campbell. 1957. The depth of field of the human eye. Optica Acta: International Journal of Optics 4, 4 (1957), 157–164.
7. Praneeth Chakravarthula, Yifan Peng, Joel Kollin, Henry Fuchs, and Felix Heide. 2019. Wirtinger Holography for Near-eye Displays. ACM Trans. Graph. (SIGGRAPH Asia) 38, 6 (2019).
8. Praneeth Chakravarthula, Ethan Tseng, Tarun Srivastava, Henry Fuchs, and Felix Heide. 2020. Learned Hardware-in-the-loop Phase Retrieval for Holographic Near-Eye Displays. ACM Trans. Graph. (SIGGRAPH Asia) 39, 6 (2020), 186.
9. Chenliang Chang, Kiseung Bang, Gordon Wetzstein, Byoungho Lee, and Liang Gao. 2020. Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective. Optica 7, 11 (2020), 1563–1578.
10. Chun Chen, Byounghyo Lee, Nan-Nan Li, Minseok Chae, Di Wang, Qiong-Hua Wang, and Byoungho Lee. 2021. Multi-depth hologram generation using stochastic gradient descent algorithm with complex loss function. OSA Opt. Express 29, 10 (2021), 15089–15103.
11. Jhen-Si Chen and Daping Chu. 2015. Improved layer-based method for rapid hologram generation and real-time interactive holographic display applications. OSA Opt. Express 23, 14 (2015), 18143–18155.
12. Rick H-Y Chen and Timothy D Wilkinson. 2009. Computer generated hologram with geometric occlusion using GPU-accelerated depth buffer rasterization for three-dimensional display. Applied optics 48, 21 (2009), 4246–4255.
13. Wenzheng Chen, Parsa Mirdehghan, Sanja Fidler, and Kiriakos N Kutulakos. 2020. Auto-Tuning Structured Light by Optical Stochastic Gradient Descent. In CVPR. 5970–5980.
14. Suyeon Choi, Jonghyun Kim, Yifan Peng, and Gordon Wetzstein. 2021. Optimizing image quality for holographic near-eye displays with Michelson Holography. Optica 8, 2 (2021), 143–146.
15. Jennifer E Curtis, Brian A Koss, and David G Grier. 2002. Dynamic holographic optical tweezers. Optics communications 207, 1-6 (2002), 169–175.
16. Rainer G Dorsch, Adolf W Lohmann, and Stefan Sinzinger. 1994. Fresnel ping-pong algorithm for two-plane computer-generated hologram display. OSA Applied optics 33, 5 (1994), 869–875.
17. M. Hossein Eybposh, Nicholas W. Caira, Mathew Atisa, Praneeth Chakravarthula, and Nicolas C. Pégard. 2020. DeepCGH: 3D computer-generated holography using deep learning. Opt. Express 28, 18 (2020), 26636–26650.
18. James R Fienup. 1982. Phase retrieval algorithms: a comparison. Applied optics 21, 15 (1982), 2758–2769.
19. Qiankun Gao, Juan Liu, Jian Han, and Xin Li. 2016. Monocular 3D see-through head-mounted display via complex amplitude modulation. OSA Opt. Express 24, 15 (2016).
20. Ralph W Gerchberg. 1972. A practical algorithm for the determination of phase from image and diffraction plane pictures. Optik 35 (1972), 237–246.
21. Joseph W Goodman. 2005. Introduction to Fourier optics. Roberts and Company.
22. Oscar Hernandez, Eirini Papagiakoumou, Dimitrii Tanese, Kevin Fidelin, Claire Wyart, and Valentina Emiliani. 2016. Three-dimensional spatiotemporal focusing of holographic patterns. Nature communications 7, 1 (2016), 1–11.
23. Ryoichi Horisaki, Ryosuke Takagi, and Jun Tanida. 2018. Deep-learning-generated holography. Appl. Opt. 57, 14 (2018), 3859–3863.
24. Chung-Kai Hsueh and Alexander A. Sawchuk. 1978. Computer-generated double-phase holograms. Applied optics 17, 24 (1978), 3874–3883.
25. Hong Hua and Bahram Javidi. 2014. A 3D integral imaging optical see-through head-mounted display. Optics express 22, 11 (2014), 13484–13491.
26. Fu-Chung Huang, Kevin Chen, and Gordon Wetzstein. 2015. The light field stereoscope: immersive computer graphics via factored near-eye light field displays with focus cues. ACM Trans. Graph. (SIGGRAPH) 34, 4 (2015), 60.
27. Changwon Jang, Kiseung Bang, Gang Li, and Byoungho Lee. 2018. Holographic Near-eye Display with Expanded Eye-box. ACM Trans. Graph. (SIGGRAPH Asia) 37, 6 (2018).
28. Changwon Jang, Kiseung Bang, Seokil Moon, Jonghyun Kim, Seungjae Lee, and Byoungho Lee. 2017. Retinal 3D: augmented reality near-eye display via pupil-tracked light field projection on retina. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6 (2017).
29. Hoonjong Kang, Takeshi Yamaguchi, and Hiroshi Yoshikawa. 2008. Accurate phase-added stereogram to improve the coherent stereogram. OSA Appl. Opt. 47, 19 (2008).
30. Grace Kuo, Laura Waller, Ren Ng, and Andrew Maimone. 2020. High Resolution étendue expansion for holographic displays. ACM Trans. Graph. (SIGGRAPH) 39, 4 (2020).
31. Douglas Lanman and David Luebke. 2013. Near-eye light field displays. ACM Trans. Graph. (SIGGRAPH Asia) 32, 6 (2013), 220.
32. Juhyun Lee, Jinsoo Jeong, Jaebum Cho, Dongheon Yoo, Byounghyo Lee, and Byoungho Lee. 2020. Deep neural network for multi-depth hologram generation and its training strategy. Opt. Express 28, 18 (2020), 27137–27154.
33. Wai Hon Lee. 1970. Sampled Fourier transform hologram generated by computer. Applied Optics 9, 3 (1970), 639–643.
34. Gang Li, Dukho Lee, Youngmo Jeong, Jaebum Cho, and Byoungho Lee. 2016. Holographic display for see-through augmented reality using mirror-lens holographic optical element. OSA Opt. Lett. 41, 11 (2016), 2486–2489.
35. Mark Lucente and Tinsley A Galyean. 1995. Rendering interactive holographic images. In ACM SIGGRAPH. 387–394.
36. Andrew Maimone, Andreas Georgiou, and Joel S Kollin. 2017. Holographic near-eye displays for virtual and augmented reality. ACM Trans. Graph. (SIGGRAPH) 36, 4 (2017), 85.
37. Andrew Maimone and Junren Wang. 2020. Holographic Optics for Thin and Lightweight Virtual Reality. ACM Trans. Graph. (SIGGRAPH) 39, 4 (2020).
38. Susana Marcos, Esther Moreno, and Rafael Navarro. 1999. The depth-of-field of the human eye from objective and subjective measurements. Vision research 39, 12 (1999), 2039–2049.
39. Kyoji Matsushima and Sumio Nakahara. 2009. Extremely high-definition full-parallax computer-generated hologram created by the polygon-based method. Applied optics 48, 34 (2009), H54–H63.
40. Eunkyong Moon, Myeongjae Kim, Jinyoung Roh, Hwi Kim, and Joonku Hahn. 2014. Holographic head-mounted display with RGB light emitting diode light source. OSA Opt. Express 22, 6 (2014), 6526–6534.
41. Nitish Padmanaban, Yifan Peng, and Gordon Wetzstein. 2019. Holographic Near-eye Displays Based on Overlap-add Stereograms. ACM Trans. Graph. (SIGGRAPH Asia) 38, 6 (2019).
42. Eirini Papagiakoumou, Francesca Anselmi, Aurélien Bègue, Vincent De Sars, Jesper Glückstad, Ehud Y Isacoff, and Valentina Emiliani. 2010. Scanless two-photon excitation of channelrhodopsin-2. Nature methods 7, 10 (2010), 848–854.
43. Jae-Hyeung Park. 2017. Recent progress in computer-generated holography for three-dimensional scenes. Journal of Information Display 18, 1 (2017), 1–12.
44. Yifan Peng, Suyeon Choi, Jonghyun Kim, and Gordon Wetzstein. 2021. Speckle-free Holography with Partially Coherent Light Sources and Camera-in-the-loop Calibration. Science Advances (2021).
45. Yifan Peng, Suyeon Choi, Nitish Padmanaban, and Gordon Wetzstein. 2020. Neural holography with camera-in-the-loop training. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–14.
46. Yifan Peng, Xiong Dun, Qilin Sun, and Wolfgang Heidrich. 2017. Mix-and-match holography. ACM Trans. Graph. 36, 6 (2017), 191.
47. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241.
48. Liang Shi, Fu-Chung Huang, Ward Lopes, Wojciech Matusik, and David Luebke. 2017. Near-eye Light Field Holographic Rendering with Spherical Waves for Wide Field of View Interactive 3D Computer Graphics. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6, Article 236 (2017), 236:1–236:17 pages.
49. Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer, and Wojciech Matusik. 2021. Towards real-time photorealistic 3D holography with deep neural networks. Nature 591 (2021), 234–239.
50. Maxim Shusteff, Allison E. M. Browar, Brett E. Kelly, Johannes Henriksson, Todd H. Weisgraber, Robert M. Panas, Nicholas X. Fang, and Christopher M. Spadaccini. 2017. One-step volumetric additive manufacturing of complex polymer structures. Science Advances 3, 12 (2017).
51. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2016. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016).
52. Koki Wakunami, Hiroaki Yamashita, and Masahiro Yamaguchi. 2013. Occlusion culling for computer generated hologram based on ray-wavefront conversion. Optics express 21, 19 (2013), 21811–21822.
53. Samuel J. Yang, William E. Allen, Isaac Kauvar, Aaron S. Andalman, Noah P. Young, Christina K. Kim, James H. Marshel, Gordon Wetzstein, and Karl Deisseroth. 2015. Extended field-of-view and increased-signal 3D holographic illumination with time-division multiplexing. Opt. Express 23, 25 (Dec 2015), 32573–32581.
54. Fahri Yaras, Hoonjong Kang, and Levent Onural. 2010. State of the Art in Holographic Displays: A Survey. Journal of Display Technology 6, 10 (2010), 443–454.
55. Han-Ju Yeom, Hee-Jae Kim, Seong-Bok Kim, HuiJun Zhang, BoNi Li, Yeong-Min Ji, Sang-Hoo Kim, and Jae-Hyeung Park. 2015. 3D holographic head mounted display using holographic optical elements with astigmatism aberration compensation. OSA Opt. Express 23, 25 (2015), 32025–32034.
56. Hao Zhang, Liangcai Cao, and Guofan Jin. 2017. Computer-generated hologram with occlusion effect using layer-based processing. Applied optics 56, 13 (2017).
57. Hao Zhang, Neil Collings, Jing Chen, Bill A Crossland, Daping Chu, and Jinghui Xie. 2011. Full parallax three-dimensional display with occlusion effect using computer generated hologram. Optical Engineering 50, 7 (2011), 074003.
58. Zhengyun Zhang and M. Levoy. 2009. Wigner distributions and how they relate to the light field. In Proc. ICCP. 1–10.
59. Remo Ziegler, Simon Bucheli, Lukas Ahrenberg, Marcus Magnor, and Markus Gross. 2007. A Bidirectional Light Field-Hologram Transform. In Computer Graphics Forum (Eurographics), Vol. 26. 435–446.
