“Accommodative holography: improving accommodation response for perceptually realistic holographic displays” by Kim, Nam, Lee, Seo and Lee

Holographic displays have gained unprecedented attention as next-generation virtual and augmented reality applications with recent achievements in the realization of a high-contrast image through computer-generated holograms (CGHs). However, these holograms show a high energy concentration in a limited angular spectrum, whereas the holograms with uniformly distributed angular spectrum suffer from a severe speckle noise in the reconstructed images. In this study, we claim that these two physical phenomena attributed to the existing CGHs significantly limit the support of accommodation cues, which is known as one of the biggest advantages of holographic displays. To support the statement, we analyze and evaluate various CGH algorithms with contrast gradients – a change of contrast over the change of the focal diopter of the eye – simulated based on the optical configuration of the display system and human visual perception models. We first introduce two approaches to improve monocular accommodation response in holographic viewing experience; optical and computational approaches to provide holographic images with sufficient contrast gradients. We design and conduct user experiments with our prototype of holographic near-eye displays, validating the deficient support of accommodation cues in the existing CGH algorithms and demonstrating the feasibility of the proposed solutions with significant improvements on accommodative gains.

1. Eirikur Agustsson and Radu Timofte. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 126–135.Google ScholarCross Ref
2. Kurt Akeley, Simon J. Watt, Ahna Reza Girshick, and Martin S. Banks. 2004. A Stereo Display Prototype with Multiple Focal Distances. ACM Trans. Graph. 23, 3 (2004), 804–813.Google ScholarDigital Library
3. Martin S Banks, Wilson S Geisler, and Patrick J Bennett. 1987. The physical limits of grating visibility. Vision research 27, 11 (1987), 1915–1924.Google Scholar
4. Peter GJ Barten. 1999. Contrast sensitivity of the human eye and its effects on image quality. SPIE press.Google Scholar
5. Stephen A Benton and V Michael Bove Jr. 2008. Holographic imaging. John Wiley & Sons.Google Scholar
6. Rafal Bogacz, Eric Brown, Jeff Moehlis, Philip Holmes, and Jonathan D Cohen. 2006. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychological Review 113, 4 (2006), 700.Google ScholarCross Ref
7. SE Broomfield, MAA Neil, EGS Paige, and GG Yang. 1992. Programmable binary phase-only optical device based on ferroelectric liquid crystal SLM. Electronics Letters 28, 1 (1992), 26–28.Google ScholarCross Ref
8. Praneeth Chakravarthula, Yifan Peng, Joel Kollin, Henry Fuchs, and Felix Heide. 2019. Wirtinger Holography for Near-Eye Displays. 38, 6, Article 213 (2019), 13 pages.Google Scholar
9. Praneeth Chakravarthula, Ethan Tseng, Tarun Srivastava, Henry Fuchs, and Felix Heide. 2020. Learned Hardware-in-the-Loop Phase Retrieval for Holographic near-Eye Displays. ACM Trans. Graph. 39, 6, Article 186 (2020), 18 pages.Google ScholarDigital Library
10. Praneeth Chakravarthula, Zhan Zhang, Okan Tursun, Piotr Didyk, Qi Sun, and Henry Fuchs. 2021. Gaze-contingent retinal speckle suppression for perceptually-matched foveated holographic displays. IEEE Transactions on Visualization and Computer Graphics 27, 11 (2021), 4194–4203.Google ScholarDigital Library
11. Chenliang Chang, Kiseung Bang, Gordon Wetzstein, Byoungho Lee, and Liang Gao. 2020. Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective. Optica 7, 11 (2020), 1563–1578.Google ScholarCross Ref
12. Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, and Gordon Wetzstein. 2021. Neural 3D Holography: Learning Accurate Wave Propagation Models for 3D Holographic Virtual and Augmented Reality Displays. ACM Trans. Graph. 40, 6, Article 240 (2021), 12 pages.Google ScholarDigital Library
13. Steven A. Cholewiak, Gordon D. Love, Pratul P. Srinivasan, Ren Ng, and Martin S. Banks. 2017. Chromablur: Rendering Chromatic Eye Aberration Improves Accommodation and Realism. ACM Trans. Graph. 36, 6, Article 210 (2017), 12 pages.Google ScholarDigital Library
14. Bruce G Cumming and Stuart J Judge. 1986. Disparity-induced and blur-induced convergence eye movement and accommodation in the monkey. Journal of Neurophysiology 55, 5 (1986), 896–914.Google ScholarCross Ref
15. Vincent R Curtis, Nicholas W Caira, Jiayi Xu, Asha Gowda Sata, and Nicolas C Pégard. 2021. DCGH: Dynamic computer generated holography for speckle-free, high fidelity 3D displays. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). IEEE, 1–9.Google Scholar
16. Scott J Daly. 1992. Visible differences predictor: an algorithm for the assessment of image fidelity. In Human Vision, Visual Processing, and Digital Display III, Vol. 1666. International Society for Optics and Photonics, 2–15.Google Scholar
17. Antonio J Del Águila-Carrasco, Iván Marín-Franch, Paula Bernal-Molina, José J Esteve-Taboada, Philip B Kruger, Robert Montés-Micó, and Norberto López-Gil. 2017. Accommodation responds to optical vergence and not defocus blur alone. Investigative ophthalmology & visual science 58, 3 (2017), 1758–1763.Google Scholar
18. Yuanbo Deng and Daping Chu. 2017. Coherence properties of different light sources and their effect on the image sharpness and speckle of holographic displays. Scientific Reports 7, 1 (2017), 1–12.Google Scholar
19. James R Fienup. 1982. Phase retrieval algorithms: a comparison. Applied Optics 21, 15 (1982), 2758–2769.Google ScholarCross Ref
20. Edgar F Fincham. 1951. The accommodation reflex and its stimulus. The British Journal of Ophthalmology 35, 7 (1951), 381.Google ScholarCross Ref
21. Ralph W Gerchberg. 1972. A practical algorithm for the determination of phase from image and diffraction plane pictures. Optik 35 (1972), 237–246.Google Scholar
22. Joseph W Goodman. 2005. Introduction to Fourier optics. Roberts and Company Publishers.Google Scholar
23. Joseph W Goodman. 2007. Speckle phenomena in optics: theory and applications. Roberts and Company Publishers.Google Scholar
24. Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1–10.Google ScholarDigital Library
25. David M Hoffman, Ahna R Girshick, Kurt Akeley, and Martin S Banks. 2008. Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. Journal of Vision 8, 3 (2008), 33–33.Google ScholarCross Ref
26. Ryoichi Horisaki, Ryosuke Takagi, and Jun Tanida. 2018. Deep-learning-generated holography. Applied Optics 57, 14 (2018), 3859–3863.Google ScholarCross Ref
27. Chung-Kai Hsueh and Alexander A Sawchuk. 1978. Computer-generated double-phase holograms. Applied Optics 17, 24 (1978), 3874–3883.Google ScholarCross Ref
28. Fu-Chung Huang, David P Luebke, and Gordon Wetzstein. 2015. The light field stereoscope.. In SIGGRAPH emerging technologies. 24–1.Google Scholar
29. Changwon Jang, Kiseung Bang, Gang Li, and Byoungho Lee. 2018. Holographic Near-Eye Display with Expanded Eye-Box. ACM Trans. Graph. 37, 6, Article 195 (dec 2018), 14 pages.Google ScholarDigital Library
30. Changwon Jang, Kiseung Bang, Seokil Moon, Jonghyun Kim, Seungjae Lee, and Byoungho Lee. 2017. Retinal 3D: Augmented Reality near-Eye Display via Pupil-Tracked Light Field Projection on Retina. ACM Trans. Graph. 36, 6, Article 190 (2017), 13 pages.Google ScholarDigital Library
31. Donald H Kelly. 1979. Motion and vision. II. Stabilized spatio-temporal threshold surface. Josa 69, 10 (1979), 1340–1349.Google ScholarCross Ref
32. Dongyeon Kim, Seung-Woo Nam, Kiseung Bang, Byounghyo Lee, Seungjae Lee, Youngmo Jeong, Jong-Mo Seo, and Byoungho Lee. 2021. Vision-correcting holographic display: evaluation of aberration correcting hologram. Biomedical Optics Express 12, 8 (2021), 5179–5195.Google ScholarCross Ref
33. Robert Konrad, Nitish Padmanaban, Keenan Molner, Emily A. Cooper, and Gordon Wetzstein. 2017. Accommodation-Invariant Computational near-Eye Displays. ACM Trans. Graph. 36, 4, Article 88 (2017), 12 pages.Google ScholarDigital Library
34. George-Alex Koulieris, Bee Bui, Martin S. Banks, and George Drettakis. 2017. Accommodation and Comfort in Head-Mounted Displays. ACM Trans. Graph. 36, 4, Article 87 (2017), 11 pages.Google ScholarDigital Library
35. Brooke Krajancich, Petr Kellnhofer, and Gordon Wetzstein. 2021. A Perceptual Model for Eccentricity-dependent Spatio-temporal Flicker Fusion and its Applications to Foveated Graphics. ACM Trans. Graph. 40 (2021). Issue 4.Google Scholar
36. Philip B Kruger, Steven Mathews, Karan R Aggarwala, and Nivian Sanchez. 1993. Chromatic aberration and ocular focus: Fincham revisited. Vision Research 33, 10 (1993), 1397–1411.Google ScholarCross Ref
37. Grace Kuo, Laura Waller, RenNg, and AndrewMaimone. 2020. High Resolution éTendue Expansion for Holographic Displays. ACM Trans. Graph. 39, 4, Article 66 (2020), 14 pages.Google ScholarDigital Library
38. Byounghyo Lee, Dongyeon Kim, Seungjae Lee, Chun Chen, and Byoungho Lee. 2022. High-contrast, speckle-free, true 3D holography via binary CGH optimization. Scientific Reports 12, 1 (18 Feb 2022), 2811. Google ScholarCross Ref
39. Byounghyo Lee, Dongheon Yoo, Jinsoo Jeong, Seungjae Lee, Dukho Lee, and Byoungho Lee. 2020c. Wide-angle speckleless DMD holographic display using structured illumination with temporal multiplexing. Optics Letters 45, 8 (2020), 2148–2151.Google ScholarCross Ref
40. Juhyun Lee, Jinsoo Jeong, Jaebum Cho, Dongheon Yoo, Byounghyo Lee, and Byoungho Lee. 2020a. Deep neural network for multi-depth hologram generation and its training strategy. Optics Express 28, 18 (2020), 27137–27154.Google ScholarCross Ref
41. Seungjae Lee, Youngjin Jo, Dongheon Yoo, Jaebum Cho, Dukho Lee, and Byoungho Lee. 2019. Tomographic near-eye displays. Nature communications 10, 1 (2019), 1–10.Google Scholar
42. Seungjae Lee, Dongyeon Kim, Seung-Woo Nam, Byounghyo Lee, Jaebum Cho, and Byoungho Lee. 2020b. Light source optimization for partially coherent holographic displays with consideration of speckle contrast, resolution, and depth of field. Scientific Reports 10, 1 (2020), 1–12.Google Scholar
43. Wai Hon Lee. 1970. Sampled Fourier transform hologram generated by computer. Applied Optics 9, 3 (1970), 639–643.Google ScholarCross Ref
44. Gang Li, Dukho Lee, Youngmo Jeong, Jaebum Cho, and Byoungho Lee. 2016. Holographic display for see-through augmented reality using mirror-lens holographic optical element. Optics Letters 41, 11 (2016), 2486–2489.Google ScholarCross Ref
45. Yongjun Lim, Keehoon Hong, Hwi Kim, Hyun-Eui Kim, Eun-Young Chang, Soohyun Lee, Taeone Kim, Jeho Nam, Hyon-Gon Choo, Jinwoong Kim, et al. 2016. 360-degree tabletop electronic holographic display. Optics Express 24, 22 (2016), 24999–25009.Google ScholarCross Ref
46. Kevin J MacKenzie, David M Hoffman, and Simon J Watt. 2010. Accommodation to multiple-focal-plane displays: Implications for improving stereoscopic displays and for accommodation control. Journal of Vision 10, 8 (2010), 22–22.Google ScholarCross Ref
47. Andrew Maimone, Andreas Georgiou, and Joel S Kollin. 2017. Holographic near-eye displays for virtual and augmented reality. ACM Transactions on Graphics (Tog) 36, 4 (2017), 1–16.Google ScholarDigital Library
48. Rafał Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in All Luminance Conditions. ACM Trans. Graph. 30, 4, Article 40 (2011), 14 pages.Google ScholarDigital Library
49. Rafał K. Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney. 2021. FovVideoVDP: A Visible Difference Predictor for Wide Field-of-View Video. ACM Trans. Graph. 40, 4, Article 49 (2021), 19 pages.Google ScholarDigital Library
50. Theodore G Martens and Kenneth N Ogle. 1959. Observations on accommodative convergence: Especially its nonlinear relationships. American Journal of Ophthalmology 47, 1 (1959), 455–463.Google ScholarCross Ref
51. Steven Mathews and Philip B Kruger. 1994. Spatiotemporal transfer function of human accommodation. Vision Research 34, 15 (1994), 1965–1980.Google ScholarCross Ref
52. Arian Mehrfard, Javad Fotouhi, Giacomo Taylor, Tess Forster, Nassir Navab, and Bernhard Fuerst. 2019. A comparative analysis of virtual reality head-mounted display systems. arXiv preprint arXiv:1912.02913 (2019).Google Scholar
53. Rafael Navarro, Pablo Artal, and David R Williams. 1993. Modulation transfer of the human eye as a function of retinal eccentricity. JOSA A 10, 2 (1993), 201–212.Google ScholarCross Ref
54. Aya Nozaki, Masaya Mitobe, Fumio Okuyama, and Yuji Sakamoto. 2017. Dynamic visual responses of accommodation and vergence to electro-holographic images. Optics Express 25, 4 (2017), 4542–4551.Google ScholarCross Ref
55. Ryuichi Ohara, Masanobu Kurita, Takuo Yoneyama, Fumio Okuyama, and Yuji Sakamoto. 2015. Response of accommodation and vergence to electro-holographic images. Applied Optics 54, 4 (2015), 615–621.Google ScholarCross Ref
56. International Commission on Non-Ionizing Radiation Protection et al. 1996. Guidelines on limits of exposure to laser radiation of wavelengths between 180 nm and 1,000 μm. Health Physics 71, 5 (1996), 804–819.Google Scholar
57. DA Owens. 1980. A comparison of accommodative responsiveness and contrast sensitivity for sinusoidal gratings. Vision Research 20, 2 (1980), 159–167.Google ScholarCross Ref
58. Nitish Padmanaban, Robert Konrad, Tal Stramer, Emily A Cooper, and Gordon Wetzstein. 2017. Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays. Proceedings of the National Academy of Sciences 114, 9 (2017), 2183–2188.Google ScholarCross Ref
59. Jongchan Park, KyeoReh Lee, and YongKeun Park. 2019. Ultrathin wide-angle large-area digital 3D holographic display using a non-periodic photon sieve. Nature Communications 10, 1 (2019), 1–8.Google ScholarCross Ref
60. Yifan Peng, Suyeon Choi, Jonghyun Kim, and Gordon Wetzstein. 2021. Speckle-free holography with partially coherent light sources and camera-in-the-loop calibration. Science Advances 7, 46 (2021), eabg5040.Google Scholar
61. Yifan Peng, Suyeon Choi, Nitish Padmanaban, and Gordon Wetzstein. 2020. Neural Holography with Camera-in-the-Loop Training. ACM Trans. Graph. 39, 6, Article 185 (nov 2020), 14 pages.Google ScholarDigital Library
62. Maria Perez-Ortiz and Rafal K Mantiuk. 2017. A practical guide and software for analysing pairwise comparison experiments. arXiv preprint arXiv:1712.03686 (2017).Google Scholar
63. Sowmya Ravikumar, Kurt Akeley, and Martin S Banks. 2011. Creating effective focus cues in multi-plane 3D displays. Optics Express 19, 21 (2011), 20940–20952.Google ScholarCross Ref
64. Stephan Reichelt, Ralf Häussler, Gerald Fütterer, and Norbert Leister. 2010. Depth cues in human visual perception and their realization in 3D displays. In Three-Dimensional Imaging, Visualization, and Display 2010 and Display Technologies and Applications for Defense, Security, and Avionics IV, Vol. 7690. International Society for Optics and Photonics, 76900B.Google Scholar
65. Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer, and Wojciech Matusik. 2021. Towards real-time photorealistic 3D holography with deep neural networks. Nature 591, 7849 (2021), 234–239.Google Scholar
66. Yasuhiro Takaki and Naoya Okada. 2009. Hologram generation by horizontal scanning of a high-speed spatial light modulator. Applied Optics 48, 17 (2009), 3255–3260.Google ScholarCross Ref
67. Yasuhiro Takaki and Masahito Yokouchi. 2012. Accommodation measurements of horizontally scanning holographic display. Optics Express 20, 4 (2012), 3918–3931.Google ScholarCross Ref
68. Larry N Thibos, Xin Hong, Arthur Bradley, and Xu Cheng. 2002. Statistical variation of aberration structure and image quality in a normal population of healthy eyes. JOSA A 19, 12 (2002), 2329–2348.Google ScholarCross Ref
69. Christopher W Tyler and Russell D Hamer. 1990. Analysis of visual modulation sensitivity. IV. Validity of the Ferry-Porter law. JOSA A 7, 4 (1990), 743–758.Google ScholarCross Ref
70. Koki Wakunami, Po-Yuan Hsieh, Ryutaro Oi, Takanori Senoh, Hisayuki Sasaki, Yasuyuki Ichihashi, Makoto Okui, Yi-Pai Huang, and Kenji Yamamoto. 2016. Projection-type see-through holographic three-dimensional display. Nature Communications 7, 1 (2016), 1–7.Google ScholarCross Ref
71. Andrew B Watson. 2013. A formula for the mean human optical modulation transfer function as a function of pupil size. Journal of Vision 13, 6 (2013), 18–18.Google ScholarCross Ref
72. Gordon Wetzstein, Douglas R Lanman, Matthew Waggener Hirsch, and Ramesh Raskar. 2012. Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. (2012).Google Scholar
73. Han-Ju Yeom, Hee-Jae Kim, Seong-Bok Kim, HuiJun Zhang, BoNi Li, Yeong-Min Ji, Sang-Hoo Kim, and Jae-Hyeung Park. 2015. 3D holographic head mounted display using holographic optical elements with astigmatism aberration compensation. Optics Express 23, 25 (2015), 32025–32034.Google ScholarCross Ref