“Simultaneous Color Computer Generated Holography” by Matsuda, Markley, Schiffers, Coissart and Kuo
Conference:
Type(s):
Title:
- Simultaneous Color Computer Generated Holography
Session/Category Title: Holography
Presenter(s)/Author(s):
Abstract:
Computer generated holography has long been touted as the future of augmented and virtual reality (AR/VR) displays, but has yet to be realized in practice. Previous high-quality, color holographic displays have made either a 3x sacrifice on frame rate by using a sequential color illumination scheme or used more than one spatial light modulator (SLM) and/or bulky, complex optical setups. The reduced frame rate of sequential color introduces distracting judder and color fringing in the presence of head motion while the form factor of current simultaneous color systems is incompatible with a head-mounted display. In this work, we propose a framework for simultaneous color holography that allows the use of the full SLM frame rate while maintaining a compact and simple optical setup. Simultaneous color holograms are optimized through the use of a perceptual loss function, a physics-based neural network wavefront propagator, and a camera-calibrated forward model. We measurably improve hologram quality compared to other simultaneous color methods and move one step closer to the realization of color holographic displays for AR/VR.
References:
[1]
Boris Apter, Uzi Efron, and Eldad Bahat-Treidel. 2004. On the fringing-field effect in liquid-crystal beam-steering devices. Applied optics 43, 1 (2004), 11–19.
[2]
Ayan Chakrabarti. 2016. Learning sensor multiplexing design through back-propagation. Advances in Neural Information Processing SystemsNips (2016), 3089–3097.
[3]
Praneeth Chakravarthula, Seung-Hwan Baek, Ethan Tseng, Andrew Maimone, Grace Kuo, Florian Schiffers, Nathan Matsuda, Oliver Cossairt, Douglas Lanman, and Felix Heide. 2022. Pupil-aware Holography. arXiv preprint arXiv:2203.14939 (2022).
[4]
Praneeth Chakravarthula, Yifan Peng, Joel Kollin, Henry Fuchs, and Felix Heide. 2019. Wirtinger holography for near-eye displays. ACM Transactions on Graphics 38, 6 (2019). https://doi.org/10.1145/3355089.3356539
[5]
Praneeth Chakravarthula, Ethan Tseng, Tarun Srivastava, Henry Fuchs, and Felix Heide. 2020. Learned hardware-in-the-loop phase retrieval for holographic near-eye displays. ACM Transactions on Graphics 39, 6 (2020). https://doi.org/10.1145/3414685.3417846
[6]
Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, Matthew O’Toole, and Gordon Wetzstein. 2022. Time-multiplexed Neural Holography: A Flexible Framework for Holographic Near-eye Displays with Fast Heavily-quantized Spatial Light Modulators. (2022), 1–9. https://doi.org/10.1145/3528233.3530734
[7]
Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, and Gordon Wetzstein. 2021a. Neural 3D Holography: Learning Accurate Wave Propagation Models for 3D Holographic Virtual and Augmented Reality Displays. ACM Transactions on Graphics 40, 6 (2021). https://doi.org/10.1145/3478513.3480542
[8]
Suyeon Choi, Jonghyun Kim, Yifan Peng, and Gordon Wetzstein. 2021b. Optimizing image quality for holographic near-eye displays with michelson holography. Optica 8, 2 (2021), 143–146.
[9]
Yuanbo Deng and Daping Chu. 2017. Coherence properties of different light sources and their effect on the image sharpness and speckle of holographic displays. Scientific Reports 7, 1 (2017), 1–12. https://doi.org/10.1038/s41598-017-06215-x
[10]
Peter Duerr, Andreas Neudert, Christoph Hohle, Hagen Stolle, Johannes Pleikies, and Hagen Sahm. 2021. MEMS Spatial Light Modulators for Real Holographic 3D Displays. In MikroSystemTechnik Congress 2021; Congress. 1–4.
[11]
M Hossein Eybposh, Nicholas W Caira, Praneeth Chakravarthula, Mathew Atisa, and Nicolas C Pégard. 2020. High-speed computer-generated holography using convolutional neural networks. In Optics and the Brain. Optica Publishing Group, BTu2C–2.
[12]
Ralph W Gerchberg. 1972. A practical algorithm for the determination of plane from image and diffraction pictures. Optik 35, 2 (1972), 237–246.
[13]
Joseph W Goodman. 2005. Introduction to fourier optics. Roberts & Company Publishers.
[14]
Manu Gopakumar, Jonghyun Kim, Suyeon Choi, Yifan Peng, and Gordon Wetzstein. 2021. Unfiltered holography: optimizing high diffraction orders without optical filtering for compact holographic displays. Optics Letters 46, 23 (2021), 5822. https://doi.org/10.1364/ol.442851
[15]
Tobias Haist and Wolfgang Osten. 2015. Holography using pixelated spatial light modulators—part 1: theory and basic considerations. Journal of Micro/Nanolithography, MEMS, and MOEMS 14, 4 (2015), 041310.
[16]
Changwon Jang, Kiseung Bang, Gang Li, and Byoungho Lee. 2018. Holographic near-eye display with expanded eye-box. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1–14.
[17]
Alexander Jesacher, Stefan Bernet, and Monika Ritsch-Marte. 2014. Colour hologram projection with an SLM by exploiting its full phase modulation range. Optics Express 22, 17 (8 2014), 20530. https://doi.org/10.1364/oe.22.020530
[18]
Koray Kavaklı, Liang Shi, Hakan Ürey, Wojciech Matusik, and Kaan Akşit. 2023. HoloHDR: Multi-color Holograms improve Dynamic Range. arXiv preprint arXiv:2301.09950 (2023).
[19]
Koray Kavaklı, Hakan Urey, and Kaan Akşit. 2022. Learned holographic light transport. Applied Optics 61, 5 (2022), B50–B55.
[20]
Dongyeon Kim, Seung-Woo Nam, Byounghyo Lee, Jong-Mo Seo, and Byoungho Lee. 2022b. Accommodative holography: improving accommodation response for perceptually realistic holographic displays. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–15.
[21]
Jonghyun Kim, Manu Gopakumar, Suyeon Choi, Yifan Peng, Ward Lopes, and Gordon Wetzstein. 2022a. Holographic glasses for virtual reality. In ACM SIGGRAPH 2022 Conference Proceedings. 1–9.
[22]
Tomasz Kozacki and Maksymilian Chlipala. 2016. Color holographic display with white light LED source and single phase only SLM. Optics Express 24, 3 (feb 2016), 2189. https://doi.org/10.1364/oe.24.002189
[23]
Grace Kuo, Laura Waller, Ren Ng, and Andrew Maimone. 2020. High resolution étendue expansion for holographic displays. ACM Transactions on Graphics (TOG) 39, 4 (2020), 66–1.
[24]
Byounghyo Lee, Dongyeon Kim, Seungjae Lee, Chun Chen, and Byoungho Lee. 2022. High-contrast, speckle-free, true 3D holography via binary CGH optimization. Scientific reports 12, 1 (2022), 1–12.
[25]
Byounghyo Lee, Dongheon Yoo, Jinsoo Jeong, Seungjae Lee, Dukho Lee, and Byoungho Lee. 2020. Wide-angle speckleless DMD holographic display using structured illumination with temporal multiplexing. Optics letters 45, 8 (2020), 2148–2151.
[26]
Gang Li, Dukho Lee, Youngmo Jeong, Jaebum Cho, and Byoungho Lee. 2016. Holographic display for see-through augmented reality using mirror-lens holographic optical element. Optics letters 41, 11 (2016), 2486–2489.
[27]
Shu-Feng Lin, Hong-Kun Cao, and Eun-Soo Kim. 2019. Single SLM full-color holographic three-dimensional video display based on image and frequency-shift multiplexing. Opt. Express 27, 11 (may 2019), 15926–15942. https://doi.org/10.1364/OE.27.015926
[28]
Shu-Feng Lin and Eun-Soo Kim. 2017. Single SLM full-color holographic 3-D display based on sampling and selective frequency-filtering methods. Opt. Express 25, 10 (may 2017), 11389–11404. https://doi.org/10.1364/OE.25.011389
[29]
Andrew Maimone, Andreas Georgiou, and Joel S Kollin. 2017. Holographic near-eye displays for virtual and augmented reality. ACM Transactions on Graphics (TOG) 36, 4 (2017), 85.
[30]
Michal Makowski, Izabela Ducin, Karol Kakarenko, Jaroslaw Suszek, Maciej Sypek, Andrzej Kolodziejczyk, P.-H Yao, C.-H Chen, J.-N 4 Kuo, and H.-W Wu. 2011. Simple holographic projection in color. Technical Report. 7–9 pages. www.osiris-project.eu
[31]
Michal Makowski, Izabela Ducin, Maciej Sypek, Agnieszka Siemion, Andrzej Siemion, Jaroslaw Suszek, and Andrzej Kolodziejczyk. 2010. Color image projection based on Fourier holograms. Technical Report 8. 1227 pages.
[32]
Michal Makowski, Mciej Sypek, Izabela Ducin, Agnieszka Fajst, Andrzej Siemion, Jaroslaw Suszek, and Andrzej Kolodzijczyk. 2009. Experimental evaluation of a full-color compact lensless holographic display. Opt. Express 17, 23 (2009), 20840–20846. https://doi.org/10.1364/OE.17.020840
[33]
Michal Makowski, Maciej Sypek, and Andrzej Kolodziejczyk. 2008. Colorful reconstructions from a thin multi-plane phase hologram. Technical Report.
[34]
Simon Moser, Monika Ritsch-Marte, and Gregor Thalhammer. 2019. Model-based compensation of pixel crosstalk in liquid crystal spatial light modulators. Optics express 27, 18 (2019), 25046–25063.
[35]
Kathy T Mullen. 1985. The contrast sensitivity of human colour vision to red-green and blue-yellow chromatic gratings.The Journal of physiology 359, 1 (1985), 381–400.
[36]
Hirotaka Nakayama, Naoki Takada, Yasuyuki Ichihashi, Shin Awazu, Tomoyoshi Shimobaba, Nobuyuki Masuda, and Tomoyoshi Ito. 2010. Real-time color electroholography using multiple graphics processing units and multiple high-definition liquid-crystal display panels. Technical Report.
[37]
Yifan Peng, Suyeon Choi, Jonghyun Kim, and Gordon Wetzstein. 2021. Speckle-free holography with partially coherent light sources and camera-in-the-loop calibration. Science Advances 7, 46 (2021). https://doi.org/10.1126/sciadv.abg5040
[38]
Yifan Peng, Suyeon Choi, Nitish Padmanaban, and Gordon Wetzstein. 2020. Neural holography with camera-in-the-loop training. ACM Transactions on Graphics 39, 6 (11 2020). https://doi.org/10.1145/3414685.3417802
[39]
William B Pennebaker and Joan L Mitchell. 1992. JPEG: Still image data compression standard. Springer Science & Business Media.
[40]
Martin Persson, David Engström, and Mattias Goksör. 2012. Reducing the effect of pixel crosstalk in phase only spatial light modulators. Optics express 20, 20 (2012), 22334–22343.
[41]
Dapu Pi, Juan Liu, and Yongtian Wang. 2022. Review of computer-generated hologram algorithms for color dynamic holographic three-dimensional display. https://doi.org/10.1038/s41377-022-00916-3
[42]
John C Platt. 2000. Optimal filtering for patterned displays. IEEE Signal Processing Letters 7, 7 (2000), 179–181.
[43]
Bernhard E Riecke, Hans-Günther Nusseck, and Jörg Schulte-Pelkum. 2006. Selected technical and perceptual aspects of virtual reality displays. (2006).
[44]
Steve Serati, Xiaowei Xia, Owais Mughal, and Anna Linnenberger. 2003. High-resolution phase-only spatial light modulators with submillisecond response. In SPIE Proceedings, Vol. 5106. Boulder Nonlinear Systems, Inc., 450 Courtney Way, #107, Lafayette, Colorado 80026.
[45]
Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer, and Wojciech Matusik. 2021. Towards real-time photorealistic 3D holography with deep neural networks. Nature 591, 7849 (2021), 234–239.
[46]
Atsushi Shiraki, Naoki Takada, Masashi Niwa, Yasuyuki Ichihashi, Tomoyoshi Shimobaba, Nobuyuki Masuda, Tomoyoshi Ito, P S Hilaire, S A Benton, M Lucente, M L Jepsen, J Kollin, H Yoshikawa, and J Underkoffler. 2009. Simplified electroholographic color reconstruction system using graphics processing unit and liquid crystal display projector References and links. Technical Report. http://www.opticsinfobase.org/oe/abstract.cfm?id=81092.
[47]
JMP Van Waveren. 2016. The asynchronous time warp for virtual reality on consumer hardware. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology. 37–46.
[48]
David R Walton, Koray Kavaklı, Rafael Kuffner Dos Anjos, David Swapp, Tim Weyrich, Hakan Urey, Anthony Steed, Tobias Ritschel, and Kaan Akşit. 2022. Metameric Varifocal Holograms. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 746–755.
[49]
Brian A Wandell. 1995. Foundations of vision. Sinauer Associates.
[50]
Gaolei Xue, Juan Liu, Xin Li, Jia Jia, Zhao Zhang, Bin Hu, and Yongtian Wang. 2014. Multiplexing encoding method for full-color dynamic 3D holographic display. Opt. Express 22, 15 (Jul 2014), 18473–18482. https://doi.org/10.1364/OE.22.018473
[51]
Daeho Yang, Wontaek Seo, Hyeonseung Yu, Sun Il Kim, Bongsu Shin, Chang-Kun Lee, Seokil Moon, Jungkwuen An, Jong-Young Hong, Geeyoung Sung, 2022. Diffraction-engineered holography: Beyond the depth representation limit of holographic displays. Nature Communications 13, 1 (2022), 1–11.
[52]
Lei Yang, Jun Xia, Chenliang Chang, Xiaobing Zhang, Zhiming Yang, and Jianhong Chen. 2015. Nonlinear dynamic phase response calibration by digital holographic microscopy. Applied optics 54, 25 (2015), 7799–7806.
[53]
Xin Yang, Ping Song, HongBo Zhang, and Qiong-Hua Wang. 2019. Full-color computer-generated holographic near-eye display based on white light illumination. Optics Express 27, 26 (2019), 38236–38249.
[54]
Fahri Yaraş, Hoonjong Kang, and Levent Onural. 2009. Real-time phase-only color holographic video display system using LED illumination. Technical Report.
[55]
Kun Yin, En Lin Hsiang, Junyu Zou, Yannanqi Li, Zhiyong Yang, Qian Yang, Po Cheng Lai, Chih Lung Lin, and Shin Tson Wu. 2022. Advanced liquid crystal devices for augmented reality and virtual reality displays: principles and applications. Light: Science and Applications 11, 1 (2022). https://doi.org/10.1038/s41377-022-00851-3
[56]
Jingzhao Zhang, Nicolas Pégard, Jingshan Zhong, Hillel Adesnik, and Laura Waller. 2017. 3D computer-generated holography by non-convex optimization. Optica 4, 10 (2017), 1306–1313.
[57]
Zichen Zhang, Zheng You, and Daping Chu. 2014. Fundamentals of phase-only liquid crystal on silicon (LCOS) devices. Light: Science & Applications 3, 10 (2014), e213–e213.
[58]
David J Zielinski, Hrishikesh M Rao, Mark A Sommer, and Regis Kopper. 2015. Exploring the effects of image persistence in low frame rate virtual environments. In 2015 IEEE Virtual Reality (VR). IEEE, 19–26.
