“Focal surface displays”

  • ©Nathan Matsuda, Alexander Fix, and Douglas Lanman




    Focal surface displays

Session/Category Title: Time to Focus




    Conventional binocular head-mounted displays (HMDs) vary the stimulus to vergence with the information in the picture, while the stimulus to accommodation remains fixed at the apparent distance of the display, as created by the viewing optics. Sustained vergence-accommodation conflict (VAC) has been associated with visual discomfort, motivating numerous proposals for delivering near-correct accommodation cues. We introduce focal surface displays to meet this challenge, augmenting conventional HMDs with a phase-only spatial light modulator (SLM) placed between the display screen and viewing optics. This SLM acts as a dynamic freeform lens, shaping synthesized focal surfaces to conform to the virtual scene geometry. We introduce a framework to decompose target focal stacks and depth maps into one or more pairs of piecewise smooth focal surfaces and underlying display images. We build on recent developments in “optimized blending” to implement a multifocal display that allows the accurate depiction of occluding, semi-transparent, and reflective objects. Practical benefits over prior accommodation-supporting HMDs are demonstrated using a binocular focal surface display employing a liquid crystal on silicon (LCOS) phase SLM and an organic light-emitting diode (OLED) display.


    1. Sameer Agarwal and Others. 2012. Ceres Solver. http://ceres-solver.org. (2012).Google Scholar
    2. Kurt Akeley, Simon J. Watt, Ahna R. Girshick, and Martin S. Banks. 2004. A Stereo Display Prototype with Multiple Focal Distances. ACM Trans. Graph. 23, 3 (2004). Google ScholarDigital Library
    3. Barry Blundell and Adam Schwartz. 1999. Volumetric Three-Dimensional Display Systems. Wiley-IEEE Press.Google Scholar
    4. V. Michael Bove. 2012. Display Holography’s Digital Second Act. Proc. IEEE 100, 4 (2012), 918–928. Google ScholarCross Ref
    5. Stephen A. Burns and Robert H. Webb. 2010. Optical Generation of the Visual Stimulus. In Handbook of Optics, Third Edition Volume III, Michael Bass (Ed.). McGraw-Hill.Google Scholar
    6. Ozan Cakmakci and Jannick Rolland. 2006. Head-Worn Displays: A Review. Journal of Display Technology 2, 3 (2006), 199–216. Google ScholarCross Ref
    7. John D. Cook. 2010. How to compute the soft maximum. http://www.johndcook.com/blog/2010/01/20. (2010).Google Scholar
    8. Gerwin Damberg, James Gregson, and Wolfgang Heidrich. 2016. High Brightness HDR Projection Using Dynamic Freeform Lensing. ACM Trans. Graph. 35, 3 (2016). Google ScholarDigital Library
    9. Andrew T. Duchowski and Others. 2014. Reducing Visual Discomfort of 3D Stereoscopic Displays with Gaze-contingent Depth-of-field. In ACM Symposium on Applied Perception. 39–46. Google ScholarDigital Library
    10. David Dunn, Cary Tippets, Kent Torell, Petr Kellnhofer, Kaan Akşit, Piotr Didyk, Karol Myszkowski, David Luebke, and Henry Fuchs. 2017. Wide Field of View Varifocal Near-Eye Display using See-through Deformable Membrane Mirrors. IEEE TVCG 23, 4 (2017), 1322–1331. Google ScholarDigital Library
    11. Enrique J. Fernández, Pedro M. Prieto, and Pablo Artal. 2010. Adaptive optics binocular visual simulator to study stereopsis in the presence of aberrations. J. Optical Society of America A 27, 11 (2010). Google ScholarCross Ref
    12. Enrique J. Fernandez, Pedro M. Prieto, Emmanuel Chirre, and Pablo Artal. 2013. Performance of a 6-Pi liquid crystal on silicon (LCoS) spatial light modulator under white light illumination for visual applications. Imaging and Applied Optics. Google ScholarCross Ref
    13. Daniel Glasner, Todd Zickler, and Anat Levin. 2014. A Reflectance Display. ACM Trans. Graph. (2014).Google Scholar
    14. Sébastien Hillaire, Anatole Lécuyer, Rémi Cozot, and Géry Casiez. 2008. Using an Eye-Tracking System to Improve Camera Motions and Depth-of-Field Blur Effects in Virtual Environments. In IEEE Virtual Reality. 47–50. Google ScholarCross Ref
    15. David M. Hoffman, R. Girshick, Kurt Akeley, and Martin S. Banks. 2008. Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. Journal of Vision 8, 3 (2008), 33.Google ScholarCross Ref
    16. Xinda Hu and Hong Hua. 2014. High-resolution optical see-through multi-focal-plane head-mounted display using freeform optics. Optics Express 22, 11 (2014). Google ScholarCross Ref
    17. Hong Hua and Bahram Javidi. 2014. A 3D integral imaging optical see-through head-mounted display. Optics Express 22, 11 (2014). Google ScholarCross Ref
    18. Fu-Chung Huang, Kevin Chen, and Gordon Wetzstein. 2015. The Light Field Stereoscope: Immersive Computer Graphics via Factored Near-eye Light Field Displays with Focus Cues. ACM Trans. Graph. 34, 4, Article 60 (2015), 12 pages.Google ScholarDigital Library
    19. Fu-Chung Huang, Douglas Lanman, Brian Barsky, and Ramesh Raskar. 2012. Correcting for Optical Aberrations using Multilayer Displays. ACM Trans. Graph. 31, 6 (2012). Google ScholarDigital Library
    20. Robert J. Jacobs, Ian L. Bailey, and Mark A. Bullimore. 1992. Artificial pupils and Maxwellian view. Applied Optics 31, 19 (1992). Google ScholarCross Ref
    21. Paul V. Johnson, Jared A. Parnell, Joohwan Kim, Christopher Saunter, Martin S. Banks, and Gordon D. Love. 2016. Assessing Visual Discomfort using Dynamic Lens and Monovision Displays. Imaging and Applied Optics 2016.Google Scholar
    22. Robert Konrad, Emily A. Cooper, and Gordon Wetzstein. 2016. Novel Optical Configurations for Virtual Reality: Evaluating User Preference and Performance with Focus-tunable and Monovision Near-eye Displays. ACM CHI (2016), 1211–1220.Google Scholar
    23. John C. Kotulak and Clifton M. Schor. 1986. The accommodative response to subthreshold blur and to perceptual fading during the Troxler phenomenon. Perception 15, 1 (1986), 7–15. Google ScholarCross Ref
    24. Gregory Kramida. 2016. Resolving the Vergence-Accommodation Conflict in Head-Mounted Displays. IEEE TVCG 22, 7 (2016), 1912–1931. Google ScholarCross Ref
    25. Bernard Kress and Thad Starner. 2013. A review of head-mounted displays (HMD) technologies and applications for consumer electronics. SPIE 8720 (2013).Google Scholar
    26. Douglas Lanman and David Luebke. 2013. Near-Eye Light Field Displays. ACM Trans. Graph. 32, 6, Article 220 (2013), 10 pages.Google ScholarDigital Library
    27. Vincent Laude. 1998. Twisted-nematic liquid-crystal pixelated active lens. Optics Communications 153 (1998), 134–152. Google ScholarCross Ref
    28. Anat Levin, Haggai Maron, and Michal Yarom. 2016. Passive light and viewpoint sensitive display of 3D content. In IEEE Int. Conf. on Computational Photography. Google ScholarCross Ref
    29. Sheng Liu, Hong Hua, and Dewen Cheng. 2010. A Novel Prototype for an Optical See-Through Head-Mounted Display with Addressable Focus Cues. IEEE TVCG 16, 3 (2010), 381–393.Google Scholar
    30. Patrick Llull, Noah Bedard, Wanmin Wu, Ivana Tošić, Kathrin Berkner, and Nikhil Balram. 2015. Design and optimization of a near-eye multifocal display system for augmented reality. Imaging and Applied Optics. Google ScholarCross Ref
    31. Gordon D. Love, David M. Hoffman, Philip J.W. Hands, James Gao, Andrew K. Kirby, and Martin S. Banks. 2009. High-speed switchable lens enables the development of a volumetric stereoscopic display. Optics Express 17, 18 (2009). Google ScholarCross Ref
    32. Kevin J. MacKenzie, Ruth A. Dickson, and Simon J. Watt. 2012. Vergence and accommodation to multiple-image-plane stereoscopic displays: “real world” responses with practical image-plane separations? Journal of Electronic Imaging 21, 1 (2012). Google ScholarCross Ref
    33. Guido Maiello, Manuela Chessa, Fabio Solari, and Peter J. Bex. 2015. The (In)Effectiveness of Simulated Blur for Depth Perception in Naturalistic Images. PLOS ONE 10, 10 (2015). Google ScholarCross Ref
    34. Andrew Maimone and Henry Fuchs. 2013. Computational augmented reality eyeglasses. In IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 29–38. Google ScholarCross Ref
    35. Rados Mantiuk, Bartosz Bazyluk, and Anna Tomaszewska. 2011a. Gaze-Dependent Depth-of-field Effect Rendering in Virtual Environments. In Serious Games Development and Applications. Springer-Verlag, 1–12. Google ScholarDigital Library
    36. Rafal Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011b. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in All Luminance Conditions. ACM Trans. Graph. 30, 4, Article 40 (2011). Google ScholarDigital Library
    37. Andrés Márquez, Claudio Iemmi, Juan Campos, and María J. Yzuel. 2006. Achromatic diffractive lens written onto a liquid crystal display. Optics Letters 31, 3 (2006). Google ScholarCross Ref
    38. Lynn Marran and Clifton Schor. 1997. Multiaccommodative Stimuli in VR Systems: Problems & Solutions. Human Factors 39, 3 (1997), 382–388. Google ScholarCross Ref
    39. Sarah C. McQuaide, Eric J. Seibel, John P. Kelly, Brian T. Schowengerdt, and Thomas A. Furness III. 2003. A retinal scanning display system that produces multiple focal planes with a deformable membrane mirror. Displays 24, 2 (2003). Google ScholarCross Ref
    40. Eunkyong Moon, Myeongjae Kim, Jinyoung Roh, Hwi Kim, and Joonku Hahn. 2014. Holographic head-mounted display with RGB light emitting diode light source. Optics Express 22, 6 (2014), 6526–6534. Google ScholarCross Ref
    41. Rahul Narain, Rachel A. Albert, Abdullah Bulbul, Gregory J. Ward, Martin S. Banks, and James F. O’Brien. 2015. Optimal Presentation of Imagery with Focus Cues on Multi-plane Displays. ACM Trans. Graph. 34, 4, Article 59 (2015). Google ScholarDigital Library
    42. Mark A. A. Neil, Edward G. S. Paige, and Leon O. D. Sucharov. 1997. Spatial-light-modulator-based three-dimensional multiplanar display. SPIE 3012 (1997), 337–341. Google ScholarCross Ref
    43. Jorge Nocedal. 1980. Updating quasi-Newton matrices with limited storage. Math. Comp. 35, 151 (1980), 773–782. Google ScholarCross Ref
    44. Nitish Padmanaban, Robert Konrad, Tal Stramer, Emily A. Cooper, and Gordon Wetzstein. 2017. Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays. Proc. of the National Academy of Sciences 114, 9 (2017). Google ScholarCross Ref
    45. Marios Papas, Wojciech Jarosz, Wenzel Jakob, Szymon Rusinkiewicz, Wojciech Matusik, and Tim Weyrich. 2011. Goal-based Caustics. Computer Graphics Forum 30, 2 (2011). Google ScholarCross Ref
    46. Eli Peli. 1999. Optometric and perceptual issues with head-mounted displays. In Visual Instrumentation: Optical Design and Principles, P. Mouroulis (Ed.). McGraw-Hill.Google Scholar
    47. Sowmya Ravikumar, Kurt Akeley, and Martin S. Banks. 2011. Creating effective focus cues in multi-plane 3D displays. Optics Express 19, 21 (2011). Google ScholarCross Ref
    48. Jannick P. Rolland, Myron W. Krueger, and Alexei Goon. 2000. Multifocal Planes Head-Mounted Displays. Applied Optics 39 (2000), 3209–3215. Google ScholarCross Ref
    49. Daniel Scharstein, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nesic, Xi Wang, and Porter Westling. 2014. High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth. In GCPR (LNCS), Vol. 8753. Springer. Google ScholarCross Ref
    50. Takashi Shibata, Joohwan Kim, David M. Hoffman, and Martin S. Banks. 2011. The zone of comfort: Predicting visual discomfort with stereo displays. Journal of Vision 11, 8 (2011), 11.Google ScholarCross Ref
    51. Shinichi Shiwa, Katsuyuki Omura, and Fumio Kishino. 1996. Proposal for a 3-D display with accommodative compensation: 3DDAC. J. Soc. Information Display 4, 4 (1996). Google ScholarCross Ref
    52. William W. Sprague, Emily A. Cooper, Ivana Tošić, and Martin S. Banks. 2015. Stereopsis is adaptive for the natural environment. Science Advances 1, 4 (2015). Google ScholarCross Ref
    53. Toshiaki Sugihara and Tsutomu Miyasato. 1998. System development of fatigue-less HMD system 3DDAC (3D Display with Accommodative Compensation): System implementation of Mk.4 in light-weight HMD. Image Engineering 97, 467 (1998).Google Scholar
    54. Erik S. Viirre, Homer Pryor, Satoru Nagata, and Thomas A. Furness III. 1998. The virtual retinal display. In Proceedings of Medicine Meets Virtual Reality. 252–257.Google Scholar
    55. Eloy A. Villegas, Concepcion González, Bernard Bourdoncle, Thierry Bonnin, and Pablo Artal. 2002. Correlation between optical and psychophysical parameters as a function of defocus. Optometry & Vision Science 79, 1 (2002), 60–67. Google ScholarCross Ref
    56. David G. Voelz. 2011. Computational Fourier Optics: A MATLAB Tutorial. SPIE Press.Google Scholar
    57. Marc von Waldkirch. 2005. Retinal projection displays for accommodation-insensitive viewing. Ph.D. Dissertation. ETH Zurich.Google Scholar
    58. Marc von Waldkirch, Paul Lukowicz, and Gerhard Tröster. 2004. Spectacle-Based Design of Wearable See-Through Display for Accommodation-Free Viewing. In Pervasive Computing. Google ScholarCross Ref
    59. Wanmin Wu, Patrick Llull, Ivana Tošić, Noah Bedard, Kathrin Berkner, and Nikhil Balram. 2016. Content-adaptive focus configuration for near-eye multi-focal displays. In IEEE Multimedia and Expo. Google ScholarCross Ref
    60. Yonghao Yue, Kei Iwasaki, Bing-Yu Chen, Yoshinori Dobashi, and Tomoyuki Nishita. 2014. Poisson-based Continuous Surface Generation for Goal-Based Caustics. ACM Trans. Graph. 33, 3 (2014), 7. Google ScholarDigital Library
    61. Zeev Zalevsky. 2010. Extended depth of focus imaging: A review. Journal of Photonics for Energy (2010).Google Scholar
    62. Marina Zannoli, Gordon D. Love, Rahul Narain, and Martin S. Banks. 2016. Blur and the perception of depth at occlusions. Journal of Vision 16, 6 (2016), 17.Google ScholarCross Ref

ACM Digital Library Publication: