“Learned hardware-in-the-loop phase retrieval for holographic near-eye displays” by Chakravarthula, Tseng, Srivastava, Fuchs and Heide
Conference:
Type(s):
Title:
- Learned hardware-in-the-loop phase retrieval for holographic near-eye displays
Session/Category Title: Computational Holography
Presenter(s)/Author(s):
Abstract:
Holography is arguably the most promising technology to provide wide field-of-view compact eyeglasses-style near-eye displays for augmented and virtual reality. However, the image quality of existing holographic displays is far from that of current generation conventional displays, effectively making today’s holographic display systems impractical. This gap stems predominantly from the severe deviations in the idealized approximations of the “unknown” light transport model in a real holographic display, used for computing holograms.In this work, we depart from such approximate “ideal” coherent light transport models for computing holograms. Instead, we learn the deviations of the real display from the ideal light transport from the images measured using a display-camera hardware system. After this unknown light propagation is learned, we use it to compensate for severe aberrations in real holographic imagery. The proposed hardware-in-the-loop approach is robust to spatial, temporal and hardware deviations, and improves the image quality of existing methods qualitatively and quantitatively in SNR and perceptual quality. We validate our approach on a holographic display prototype and show that the method can fully compensate unknown aberrations and erroneous and non-linear SLM phase delays, without explicitly modeling them. As a result, the proposed method significantly outperforms existing state-of-the-art methods in simulation and experimentation – just by observing captured holographic images.
References:
1. Eirikur Agustsson and Radu Timofte. 2017. NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.Google Scholar
2. Lukas Ahrenberg, Philip Benzie, Marcus Magnor, and John Watson. 2008. Computer generated holograms from three dimensional meshes using an analytic light transport model. Applied optics 47, 10 (2008), 1567–1574.Google Scholar
3. Sohail Bahmani and Justin Romberg. 2017. Phase Retrieval Meets Statistical Learning Theory: A Flexible Convex Relaxation. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Vol. 54. PMLR, 252–260.Google Scholar
4. George Barbastathis, Aydogan Ozcan, and Guohai Situ. 2019. On the use of deep learning for computational imaging. Optica 6, 8 (2019), 921–943.Google ScholarCross Ref
5. Heinz H Bauschke, Patrick L Combettes, and D Russell Luke. 2003. Hybrid projection-reflection method for phase retrieval. JOSA A 20, 6 (2003), 1025–1034.Google ScholarCross Ref
6. Muharrem Bayraktar and Meriç Özcan. 2010. Method to calculate the far field of three-dimensional objects for computer-generated holography. Applied optics 49, 24 (2010), 4647–4654.Google Scholar
7. Vittorio Bianco, Pasquale Memmolo, Marco Leo, Silvio Montresor, Cosimo Distante, Melania Paturzo, Pascal Picart, Bahram Javidi, and Pietro Ferraro. 2018. Strategies for reducing speckle noise in digital holography. Light: Science & Applications 7, 1 (2018), 1–16.Google ScholarCross Ref
8. Vittorio Bianco, Pasquale Memmolo, Melania Paturzo, Andrea Finizio, Bahram Javidi, and Pietro Ferraro. 2016. Quasi noise-free digital holography. Light: Science & Applications 5, 9 (2016), e16142.Google ScholarCross Ref
9. Emmanuel J Candes, Thomas Strohmer, and Vladislav Voroninski. 2013. Phaselift: Exact and stable signal recovery from magnitude measurements via convex programming. Communications on Pure and Applied Mathematics 66, 8 (2013), 1241–1274.Google ScholarCross Ref
10. Praneeth Chakravarthula, David Dunn, Kaan Akşit, and Henry Fuchs. 2018. Focusar: Auto-focus augmented reality eyeglasses for both real world and virtual imagery. IEEE transactions on visualization and computer graphics 24, 11 (2018), 2906–2916.Google Scholar
11. Praneeth Chakravarthula, Yifan Peng, Joel Kollin, Henry Fuchs, and Felix Heide. 2019. Wirtinger holography for near-eye displays. ACM Transactions on Graphics (TOG) 38, 6 (2019), 213.Google ScholarDigital Library
12. Praneeth Chakravarthula, Yifan Peng, Joel Kollin, Felix Heide, and Henry Fuchs. 2020. Computing high quality phase-only holograms for holographic displays. In Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR), Vol. 11310. International Society for Optics and Photonics, 1131006.Google Scholar
13. JS Chen and DP Chu. 2015. Improved layer-based method for rapid hologram generation and real-time interactive holographic display applications. Optics express 23, 14 (2015), 18143–18155.Google Scholar
14. Rick H-Y Chen and Timothy D Wilkinson. 2009. Computer generated hologram from point cloud using graphics processor. Applied optics 48, 36 (2009), 6841–6850.Google Scholar
15. Steven A Cholewiak, Gordon D Love, Pratul P Srinivasan, Ren Ng, and Martin S Banks. 2017. ChromaBlur: Rendering chromatic eye aberration improves accommodation and realism. ACM Transactions on Graphics (TOG) 36, 6 (2017), 210.Google ScholarDigital Library
16. J Christopher Dainty. 1977. I The statistics of speckle patterns. In Progress in optics. Vol. 14. Elsevier, 1–46.Google Scholar
17. WJ Dallas and AW Lohmann. 1972. Phase quantization in holograms-depth effects. Applied optics 11, 1 (1972), 192–194.Google Scholar
18. James R Fienup. 1982. Phase retrieval algorithms: a comparison. Applied optics 21, 15 (1982), 2758–2769.Google Scholar
19. James R Fienup. 1993. Phase-retrieval algorithms for a complicated optical system. Applied optics 32, 10 (1993), 1737–1746.Google Scholar
20. Ralph W Gerchberg. 1972. A practical algorithm for the determination of the phase from image and diffraction plane pictures. Optik 35 (1972), 237–246.Google Scholar
21. Lior Golan and Shy Shoham. 2009. Speckle elimination using shift-averaging in highrate holographic projection. Optics express 17, 3 (2009), 1330–1339.Google Scholar
22. Tom Goldstein and Christoph Studer. 2018. PhaseMax: Convex phase retrieval via basis pursuit. IEEE Transactions on Information Theory (2018).Google Scholar
23. RA Gonsalves. 1976. Phase retrieval from modulus data. JOSA 66, 9 (1976), 961–964.Google ScholarCross Ref
24. Joseph W Goodman. 2005. Introduction to Fourier optics. Roberts and Company Publishers.Google Scholar
25. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2016. Image-to-Image Translation with Conditional Adversarial Networks. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 5967–5976.Google Scholar
26. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
27. Changwon Jang, Kiseung Bang, Gang Li, and Byoungho Lee. 2019. Holographic near-eye display with expanded eye-box. ACM Transactions on Graphics (TOG) 37, 6 (2019), 195.Google ScholarDigital Library
28. Changwon Jang, Kiseung Bang, Seokil Moon, Jonghyun Kim, Seungjae Lee, and Byoungho Lee. 2017. Retinal 3D: augmented reality near-eye display via pupil-tracked light field projection on retina. ACM Transactions on Graphics (TOG) 36, 6 (2017), 190.Google ScholarDigital Library
29. Jia Jia, Juan Liu, Guofan Jin, and Yongtian Wang. 2014. Fast and effective occlusion culling for 3D holographic displays by inverse orthographic projection with low angular sampling. Applied optics 53, 27 (2014), 6287–6293.Google Scholar
30. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In European Conference on Computer Vision (ECCV).Google Scholar
31. Xin Kang. 2008. An effective method for reducing speckle noise in digital holography. Chinese Optics Letters 6, 2 (2008), 100–103.Google ScholarCross Ref
32. Hwi Kim, Joonku Hahn, and Byoungho Lee. 2008. Mathematical modeling of triangle-mesh-modeled three-dimensional surface objects for digital holography. Applied optics 47, 19 (2008), D117–D127.Google Scholar
33. RG Lane. 1991. Phase retrieval using conjugate gradient minimization. Journal of Modern Optics 38, 9 (1991), 1797–1813.Google ScholarCross Ref
34. Douglas Lanman and David Luebke. 2013. Near-eye light field displays. ACM Transactions on Graphics (TOG) 32, 6 (2013), 220.Google ScholarDigital Library
35. Detlef Leseberg and Christian Frère. 1988. Computer-generated holograms of 3-D objects composed of tilted planar segments. Applied optics 27, 14 (1988), 3020–3024.Google Scholar
36. LB Lesem, PM Hirsch, and JA Jordan. 1969. The kinoform: a new wavefront reconstruction device. IBM Journal of Research and Development 13, 2 (1969), 150–155.Google ScholarDigital Library
37. Anat Levin, Haggai Maron, and Michal Yarom. 2016. Passive light and viewpoint sensitive display of 3D content. In 2016 IEEE International Conference on Computational Photography (ICCP). IEEE, 1–15.Google ScholarCross Ref
38. Gang Li, Dukho Lee, Youngmo Jeong, Jaebum Cho, and Byoungho Lee. 2016. Holographic display for see-through augmented reality using mirror-lens holographic optical element. Optics letters 41, 11 (2016), 2486–2489.Google Scholar
39. Mark Lucente and Tinsley A Galyean. 1995. Rendering interactive holographic images. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. ACM, 387–394.Google ScholarDigital Library
40. Mark E Lucente. 1993. Interactive computation of holograms using a look-up table. Journal of Electronic Imaging 2, 1 (1993), 28–34.Google ScholarCross Ref
41. Andrew Maimone, Andreas Georgiou, and Joel S Kollin. 2017. Holographic near-eye displays for virtual and augmented reality. ACM Transactions on Graphics (TOG) 36, 4 (2017), 85.Google ScholarDigital Library
42. Stefano Marchesini, Yu-Chao Tu, and Hau-tieng Wu. 2016. Alternating projection, ptychographic imaging and phase synchronization. Applied and Computational Harmonic Analysis 41, 3 (2016), 815–851.Google ScholarCross Ref
43. Nobuyuki Masuda, Tomoyoshi Ito, Takashi Tanaka, Atsushi Shiraki, and Takashige Sugie. 2006. Computer generated holography using a graphics processing unit. Optics Express 14, 2 (2006), 603–608.Google ScholarCross Ref
44. Kyoji Matsushima. 2005. Computer-generated holograms for three-dimensional surface objects with shade and texture. Applied optics 44, 22 (2005), 4607–4614.Google Scholar
45. Kyoji Matsushima and Sumio Nakahara. 2009. Extremely high-definition full-parallax computer-generated hologram created by the polygon-based method. Applied optics 48, 34 (2009), H54–H63.Google Scholar
46. Kyoji Matsushima, Masaki Nakamura, and Sumio Nakahara. 2014. Silhouette method for hidden surface removal in computer holography and its acceleration using the switch-back technique. Optics express 22, 20 (2014), 24450–24465.Google Scholar
47. Nitish Padmanaban, Yifan Peng, and Gordon Wetzstein. 2019. Holographic near-eye displays based on overlap-add stereograms. ACM Transactions on Graphics (TOG) 38, 6 (2019), 214.Google ScholarDigital Library
48. Yifan Peng, Xiong Dun, Qilin Sun, and Wolfgang Heidrich. 2017. Mix-and-match holography. ACM Transactions on Graphics (2017).Google Scholar
49. Christoph Petz and Marcus Magnor. 2003. Fast hologram synthesis for 3D geometry models using graphics hardware. In Proc. SPIE, Vol. 5005. 266–275.Google ScholarCross Ref
50. Stephan Reichelt, Ralf Häussler, Gerald Fütterer, Norbert Leister, Hiromi Kato, Naru Usukura, and Yuuichi Kanbayashi. 2012. Full-range, complex spatial light modulator for real-time holography. Optics letters 37, 11 (2012), 1955–1957.Google Scholar
51. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241.Google ScholarCross Ref
52. Liang Shi, Fu-Chung Huang, Ward Lopes, Wojciech Matusik, and David Luebke. 2017. Near-eye light field holographic rendering with spherical waves for wide field of view interactive 3d computer graphics. ACM Transactions on Graphics (TOG) 36, 6 (2017), 236.Google ScholarDigital Library
53. Quinn YJ Smithwick, James Barabas, Daniel E Smalley, and V Michael Bove. 2010. Interactive holographic stereograms with accommodation cues. In Practical Holography XXIV: Materials and Applications, Vol. 7619. International Society for Optics and Photonics, 761903.Google Scholar
54. Tullio Tommasi and Bruno Bianco. 1993. Computer-generated holograms of tilted planes by a spatial frequency approach. JOSA A 10, 2 (1993), 299–305.Google ScholarCross Ref
55. Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2017. High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 8798–8807.Google Scholar
56. Zhou Wang, Eero P Simoncelli, and Alan C Bovik. 2003. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, Vol. 2. Ieee, 1398–1402.Google ScholarCross Ref
57. James P Waters. 1966. Holographic image synthesis utilizing theoretical methods. Applied physics letters 9, 11 (1966), 405–407.Google Scholar
58. Zaiwen Wen, Chao Yang, Xin Liu, and Stefano Marchesini. 2012. Alternating direction methods for classical and ptychographic phase retrieval. Inverse Problems 28, 11 (2012), 115010.Google ScholarCross Ref
59. Gordon Wetzstein, Douglas Lanman, Matthew Hirsch, and Ramesh Raskar. 2012. Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. (2012).Google Scholar
60. Masahiro Yamaguchi, Hideshi Hoshino, Toshio Honda, and Nagaaki Ohyama. 1993. Phase-added stereogram: calculation of hologram using computer graphics technique. In Proc. SPIE, Vol. 1914. 25–31.Google ScholarCross Ref
61. Han-Ju Yeom, Hee-Jae Kim, Seong-Bok Kim, HuiJun Zhang, BoNi Li, Yeong-Min Ji, Sang-Hoo Kim, and Jae-Hyeung Park. 2015. 3D holographic head mounted display using holographic optical elements with astigmatism aberration compensation. Optics express 23, 25 (2015), 32025–32034.Google Scholar
62. Hiroshi Yoshikawa, Takeshi Yamaguchi, and Hiroki Uetake. 2016. Image quality evaluation and control of computer-generated holograms. In Practical Holography XXX: Materials and Applications, Vol. 9771. International Society for Optics and Photonics, 97710N.Google Scholar
63. Hao Zhang, Yan Zhao, Liangcai Cao, and Guofan Jin. 2015. Fully computed holographic stereogram based algorithm for computer-generated holograms with accurate depth cues. Optics express 23, 4 (2015), 3901–3913.Google Scholar
64. Hao Zhang, Yan Zhao, Liangcai Cao, and Guofan Jin. 2016. Layered holographic stereogram based on inverse Fresnel diffraction. Applied optics 55, 3 (2016), A154–A159.Google Scholar
65. Jingzhao Zhang, Nicolas Pégard, Jingshan Zhong, Hillel Adesnik, and Laura Waller. 2017. 3D computer-generated holography by non-convex optimization. Optica 4, 10 (2017), 1306–1313.Google ScholarCross Ref
66. Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
67. Yan Zhao, Liangcai Cao, Hao Zhang, Dezhao Kong, and Guofan Jin. 2015. Accurate calculation of computer-generated holograms using angular-spectrum layer-oriented method. Optics express 23, 20 (2015), 25440–25449.Google Scholar
68. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. IEEE International Conference on Computer Vision (ICCV) (2017), 2242–2251.Google Scholar
