“Neural holography with camera-in-the-loop training” by Peng, Choi, Padmanaban and Wetzstein
Conference:
Type(s):
Title:
- Neural holography with camera-in-the-loop training
Session/Category Title: Computational Holography
Presenter(s)/Author(s):
Abstract:
Holographic displays promise unprecedented capabilities for direct-view displays as well as virtual and augmented reality applications. However, one of the biggest challenges for computer-generated holography (CGH) is the fundamental tradeoff between algorithm runtime and achieved image quality, which has prevented high-quality holographic image synthesis at fast speeds. Moreover, the image quality achieved by most holographic displays is low, due to the mismatch between the optical wave propagation of the display and its simulated model. Here, we develop an algorithmic CGH framework that achieves unprecedented image fidelity and real-time framerates. Our framework comprises several parts, including a novel camera-in-the-loop optimization strategy that allows us to either optimize a hologram directly or train an interpretable model of the optical wave propagation and a neural network architecture that represents the first CGH algorithm capable of generating full-color high-quality holographic images at 1080p resolution in real time.
References:
1. Eirikur Agustsson and Radu Timofte. 2017. NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. In CVPR.Google Scholar
2. Kaan Akşit, Ward Lopes, Jonghyun Kim, Peter Shirley, and David Luebke. 2017. Near-eye Varifocal Augmented Reality Display Using See-through Screens. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6 (2017), 189:1–189:13.Google Scholar
3. Kurt Akeley, Simon J. Watt, Ahna Reza Girshick, and Martin S. Banks. 2004. A stereo display prototype with multiple focal distances. ACM Trans. Graph. (SIGGRAPH) 23, 3 (2004), 804–813.Google ScholarDigital Library
4. George Barbastathis, Aydogan Ozcan, and Guohai Situ. 2019. On the use of deep learning for computational imaging. Optica 6, 8 (2019), 921–943.Google ScholarCross Ref
5. Stephen A. Benton. 1983. Survey Of Holographic Stereograms. In Proc. SPIE, Vol. 0367.Google Scholar
6. Stephen A. Benton and V. Michael Bove. 2008. Holographic Imaging. Wiley-Interscience.Google Scholar
7. Max Born and Emil Wolf. 1959. Principles of Optics. Cambridge University Press.Google Scholar
8. Praneeth Chakravarthula, David Dunn, Kaan Akşit, and Henry Fuchs. 2018. Focusar: Auto-focus augmented reality eyeglasses for both real world and virtual imagery. IEEE TVCG (ISMAR) 24, 11 (2018), 2906–2916.Google Scholar
9. Praneeth Chakravarthula, Yifan Peng, Joel Kollin, Henry Fuchs, and Felix Heide. 2019. Wirtinger Holography for Near-eye Displays. ACM Trans. Graph. (SIGGRAPH Asia) 38, 6 (2019).Google Scholar
10. Jen-Hao Rick Chang, B. V. K. Vijaya Kumar, and Aswin C. Sankaranarayanan. 2018. Towards Multifocal Displays with Dense Focal Stacks. ACM Trans. Graph. (SIGGRAPH Asia) 37, 6 (2018), 198:1–198:13.Google Scholar
11. Jhen-Si Chen and Daping Chu. 2015. Improved layer-based method for rapid hologram generation and real-time interactive holographic display applications. OSA Opt. Express 23, 14 (2015), 18143–18155.Google ScholarCross Ref
12. Rick H-Y Chen and Timothy D Wilkinson. 2009. Computer generated hologram with geometric occlusion using GPU-accelerated depth buffer rasterization for three-dimensional display. Applied optics 48, 21 (2009), 4246–4255.Google Scholar
13. Yuxin Chen, Yuejie Chi, Jianqing Fan, and Cong Ma. 2019. Gradient descent with random initialization: fast global convergence for nonconvex phase retrieval. Mathematical Programming 176 (2019), 1436–4646.Google ScholarDigital Library
14. Rainer G Dorsch, Adolf W Lohmann, and Stefan Sinzinger. 1994. Fresnel ping-pong algorithm for two-plane computer-generated hologram display. OSA Applied optics 33, 5 (1994), 869–875.Google Scholar
15. David Dunn, Cary Tippets, Kent Torell, Petr Kellnhofer, Kaan Akşit, Piotr Didyk, Karol Myszkowski, David Luebke, and Henry Fuchs. 2017. Wide Field Of View Varifocal Near-Eye Display Using See-Through Deformable Membrane Mirrors. IEEE TVCG (VR) 23, 4 (2017).Google ScholarDigital Library
16. James R Fienup. 1982. Phase retrieval algorithms: a comparison. Applied optics 21, 15 (1982), 2758–2769.Google Scholar
17. Qiankun Gao, Juan Liu, Jian Han, and Xin Li. 2016. Monocular 3D see-through head-mounted display via complex amplitude modulation. OSA Opt. Express 24, 15 (2016).Google Scholar
18. Ralph W Gerchberg. 1972. A practical algorithm for the determination of phase from image and diffraction plane pictures. Optik 35 (1972), 237–246.Google Scholar
19. Joseph W Goodman. 2005. Introduction to Fourier optics. Roberts and Company.Google Scholar
20. Ryoichi Horisaki, Ryosuke Takagi, and Jun Tanida. 2018. Deep-learning-generated holography. Appl. Opt. 57, 14 (2018), 3859–3863.Google ScholarCross Ref
21. Chung-Kai Hsueh and Alexander A. Sawchuk. 1978. Computer-generated double-phase holograms. Applied optics 17, 24 (1978), 3874–3883.Google Scholar
22. Xinda Hu and Hong Hua. 2014. Design and assessment of a depth-fused multi-focal-plane display prototype. J. Disp. Technol. 10, 4 (2014), 308–316.Google ScholarCross Ref
23. Hong Hua and Bahram Javidi. 2014. A 3D integral imaging optical see-through head-mounted display. Optics express 22, 11 (2014), 13484–13491.Google Scholar
24. Fu-Chung Huang, Kevin Chen, and Gordon Wetzstein. 2015. The light field stereoscope: immersive computer graphics via factored near-eye light field displays with focus cues. ACM Trans. Graph. (SIGGRAPH) 34, 4 (2015), 60.Google ScholarDigital Library
25. Changwon Jang, Kiseung Bang, Gang Li, and Byoungho Lee. 2018. Holographic Near-eye Display with Expanded Eye-box. ACM Trans. Graph. (SIGGRAPH Asia) 37, 6 (2018).Google Scholar
26. Changwon Jang, Kiseung Bang, Seokil Moon, Jonghyun Kim, Seungjae Lee, and Byoungho Lee. 2017. Retinal 3D: augmented reality near-eye display via pupil-tracked light field projection on retina. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6 (2017).Google Scholar
27. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016a. Perceptual losses for real-time style transfer and super-resolution. In ECCV. 694–711.Google Scholar
28. Paul V. Johnson, Jared AQ. Parnell, Joohwan Kim, Christopher D. Saunter, Gordon D. Love, and Martin S. Banks. 2016b. Dynamic lens and monovision 3D displays to improve viewer comfort. OSA Opt. Express 24, 11 (2016), 11808–11827.Google ScholarCross Ref
29. Hoonjong Kang, Takeshi Yamaguchi, and Hiroshi Yoshikawa. 2008. Accurate phase-added stereogram to improve the coherent stereogram. OSA Appl. Opt. 47, 19 (2008).Google Scholar
30. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
31. Robert Konrad, Emily A. Cooper, and Gordon Wetzstein. 2016. Novel Optical Configurations for Virtual Reality: Evaluating User Preference and Performance with Focus-tunable and Monovision Near-eye Displays. In Proc. ACM SIGCHI. 1211–1220.Google ScholarDigital Library
32. Pierre-Yves Laffont, Ali Hasnain, Pierre-Yves Guillemet, Samuel Wirajaya, Joe Khoo, Deng Teng, and Jean-Charles Bazin. 2018. Verifocal: A Platform for Vision Correction and Accommodation in Head-mounted Displays. In ACM SIGGRAPH 2018 Emerging Technologies. 21:1–21:2.Google Scholar
33. Douglas Lanman and David Luebke. 2013. Near-eye light field displays. ACM Trans. Graph. (SIGGRAPH Asia) 32, 6 (2013), 220.Google ScholarDigital Library
34. Wai Hon Lee. 1970. Sampled Fourier transform hologram generated by computer. Applied Optics 9, 3 (1970), 639–643.Google ScholarCross Ref
35. Gang Li, Dukho Lee, Youngmo Jeong, Jaebum Cho, and Byoungho Lee. 2016. Holographic display for see-through augmented reality using mirror-lens holographic optical element. OSA Opt. Lett. 41, 11 (2016), 2486–2489.Google ScholarCross Ref
36. Sheng Liu, Dewen Cheng, and Hong Hua. 2008. An optical see-through head mounted display with addressable focal planes. In Proc. IEEE ISMAR. 33–42.Google Scholar
37. Patrick Llull, Noah Bedard, Wanmin Wu, Ivana Tosic, Kathrin Berkner, and Nikhil Balram. 2015. Design and optimization of a near-eye multifocal display system for augmented reality. In OSA Imaging Appl. Opt.Google Scholar
38. Gordon D. Love, David M. Hoffman, Philip J. W. Hands, James Gao, Andrew K. Kirby, and Martin S. Banks. 2009. High-speed switchable lens enables the development of a volumetric stereoscopic display. Opt. Express 17, 18 (2009), 15716–25.Google ScholarCross Ref
39. Mark Lucente and Tinsley A Galyean. 1995. Rendering interactive holographic images. In ACM SIGGRAPH. 387–394.Google Scholar
40. Andrew Maimone, Andreas Georgiou, and Joel S Kollin. 2017. Holographic near-eye displays for virtual and augmented reality. ACM Trans. Graph. (SIGGRAPH) 36, 4 (2017), 85.Google ScholarDigital Library
41. Andrew Maimone and Junren Wang. 2020. Holographic Optics for Thin and Lightweight Virtual Reality. ACM Trans. Graph. (SIGGRAPH) 39, 4 (2020).Google ScholarDigital Library
42. Kyoji Matsushima and Sumio Nakahara. 2009. Extremely high-definition full-parallax computer-generated hologram created by the polygon-based method. Applied optics 48, 34 (2009), H54–H63.Google Scholar
43. Kyoji Matsushima and Tomoyoshi Shimobaba. 2009. Band-limited angular spectrum method for numerical simulation of free-space propagation in far and near fields. Optics express 17, 22 (2009), 19662–19673.Google Scholar
44. Olivier Mercier, Yusufu Sulai, Kevin Mackenzie, Marina Zannoli, James Hillis, Derek Nowrouzezahrai, and Douglas Lanman. 2017. Fast Gaze-contingent Optimal Decompositions for Multifocal Displays. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6 (2017), 237:1–237:15.Google Scholar
45. Eunkyong Moon, Myeongjae Kim, Jinyoung Roh, Hwi Kim, and Joonku Hahn. 2014. Holographic head-mounted display with RGB light emitting diode light source. OSA Opt. Express 22, 6 (2014), 6526–6534.Google ScholarCross Ref
46. Rahul Narain, Rachel A. Albert, Abdullah Bulbul, Gregory J. Ward, Martin S. Banks, and James F. O’Brien. 2015. Optimal Presentation of Imagery with Focus Cues on Multi-plane Displays. ACM Trans. Graph. (SIGGRAPH) 34, 4 (2015), 59:1–59:12.Google ScholarDigital Library
47. Nitish Padmanaban, Robert Konrad, Tal Stramer, Emily A. Cooper, and Gordon Wetzstein. 2017. Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays. PNAS 114 (2017), 2183–2188. Issue 9.Google ScholarCross Ref
48. Nitish Padmanaban, Yifan Peng, and Gordon Wetzstein. 2019. Holographic Near-eye Displays Based on Overlap-add Stereograms. ACM Trans. Graph. (SIGGRAPH Asia) 38, 6 (2019).Google Scholar
49. Jae-Hyeung Park. 2017. Recent progress in computer-generated holography for three-dimensional scenes. Journal of Information Display 18, 1 (2017), 1–12.Google ScholarCross Ref
50. Yifan Peng, Xiong Dun, Qilin Sun, and Wolfgang Heidrich. 2017. Mix-and-match holography. ACM Trans. Graph. 36, 6 (2017), 191.Google ScholarDigital Library
51. K. Rathinavel, H. Wang, A. Blate, and H. Fuchs. 2018. An Extended Depth-at-Field Volumetric Near-Eye Augmented Reality Display. IEEE TVCG 24, 11 (2018).Google Scholar
52. Yair Rivenson, Yichen Wu, and Aydogan Ozcan. 2019. Deep learning in holography and coherent imaging. Light: Science & Applications 8, 85 (2019).Google Scholar
53. Yair Rivenson, Yibo Zhang, Harun Günaydın, Da Teng, and Aydogan Ozcan. 2018. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light: Science & Applications 7, 2 (2018), 17141.Google ScholarCross Ref
54. Jannick P. Rolland, Myron W. Krueger, and Alexei Goon. 2000. Multifocal planes head-mounted displays. Appl. Opt. 39, 19 (2000), 3209–3215.Google ScholarCross Ref
55. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241.Google ScholarCross Ref
56. Yoav Shechtman, Yonina C. Eldar, Oren Cohen, Henry Nicholas Chapman, Jianwei Miao, and Mordechai Segev. 2015. Phase Retrieval with Application to Optical Imaging: A contemporary overview. IEEE Signal Processing Magazine 32, 3 (2015), 87–109.Google ScholarCross Ref
57. Liang Shi, Fu-Chung Huang, Ward Lopes, Wojciech Matusik, and David Luebke. 2017. Near-eye Light Field Holographic Rendering with Spherical Waves for Wide Field of View Interactive 3D Computer Graphics. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6, Article 236 (2017), 236:1–236:17 pages.Google Scholar
58. Shinichi Shiwa, Katsuyuki Omura, and Fumio Kishino. 1996. Proposal for a 3-D display with accommodative compensation: 3DDAC. Journal of the Society for Information Display 4, 4 (1996), 255–261.Google ScholarCross Ref
59. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. In CVPR.Google Scholar
60. Ayan Sinha, Justin Lee, Shuai Li, and George Barbastathis. 2017. Lensless computational imaging through deep learning. Optica 4, 9 (2017), 1117–1125.Google ScholarCross Ref
61. MJ Townson, OJD Farley, G Orban de Xivry, J Osborn, and AP Reeves. 2019. AOtools: a Python package for adaptive optics modelling and analysis. Optics express 27, 22 (2019), 31316–31329.Google Scholar
62. Ethan Tseng, Felix Yu, Yuting Yang, Fahim Mannan, Karl ST Arnaud, Derek Nowrouzezahrai, Jean-François Lalonde, and Felix Heide. 2019. Hyperparameter optimization in black-box image processing using differentiable proxies. ACM Trans. Graph. (SIGGRAPH) 38, 4 (2019), 1–14.Google ScholarDigital Library
63. Koki Wakunami, Hiroaki Yamashita, and Masahiro Yamaguchi. 2013. Occlusion culling for computer generated hologram based on ray-wavefront conversion. Optics express 21, 19 (2013), 21811–21822.Google Scholar
64. Fahri Yaras, Hoonjong Kang, and Levent Onural. 2010. State of the Art in Holographic Displays: A Survey. Journal of Display Technology 6, 10 (2010), 443–454.Google ScholarCross Ref
65. Han-Ju Yeom, Hee-Jae Kim, Seong-Bok Kim, HuiJun Zhang, BoNi Li, Yeong-Min Ji, Sang-Hoo Kim, and Jae-Hyeung Park. 2015. 3D holographic head mounted display using holographic optical elements with astigmatism aberration compensation. OSA Opt. Express 23, 25 (2015), 32025–32034.Google ScholarCross Ref
66. Hao Zhang, Liangcai Cao, and Guofan Jin. 2017. Computer-generated hologram with occlusion effect using layer-based processing. Applied optics 56, 13 (2017).Google Scholar
67. Hao Zhang, Neil Collings, Jing Chen, Bill A Crossland, Daping Chu, and Jinghui Xie. 2011. Full parallax three-dimensional display with occlusion effect using computer generated hologram. Optical Engineering 50, 7 (2011), 074003.Google ScholarCross Ref
68. Zhengyun Zhang and M. Levoy. 2009. Wigner distributions and how they relate to the light field. In Proc. ICCP. 1–10.Google Scholar
69. Remo Ziegler, Simon Bucheli, Lukas Ahrenberg, Marcus Magnor, and Markus Gross. 2007. A Bidirectional Light Field-Hologram Transform. In Computer Graphics Forum (Eurographics), Vol. 26. 435–446.Google ScholarCross Ref
