“Optimal presentation of imagery with focus cues on multi-plane displays” by Narain, Albert, Bulbul, Banks, O’Brien, et al. …

  • ©Rahul Narain, Rachel Albert, M. Abdullah Bulbul, Martin (Marty) S. Banks, and James F. O'Brien



Session Title:

    VR, Display, and Interaction


    Optimal presentation of imagery with focus cues on multi-plane displays




    We present a technique for displaying three-dimensional imagery of general scenes with nearly correct focus cues on multi-plane displays. These displays present an additive combination of images at a discrete set of optical distances, allowing the viewer to focus at different distances in the simulated scene. Our proposed technique extends the capabilities of multi-plane displays to general scenes with occlusions and non-Lambertian effects by using a model of defocus in the eye of the viewer. Requiring no explicit knowledge of the scene geometry, our technique uses an optimization algorithm to compute the images to be displayed on the presentation planes so that the retinal images when accommodating to different distances match the corresponding retinal images of the input scene as closely as possible. We demonstrate the utility of the technique using imagery acquired from both synthetic and real-world scenes, and analyze the system’s characteristics including bounds on achievable resolution.


    1. Akeley, K., Watt, S. J., Girshick, A. R., and Banks, M. S. 2004. A stereo display prototype with multiple focal distances. ACM Trans. Graph. 23, 3 (Aug.), 804–813. Google ScholarDigital Library
    2. Andersen, A., and Kak, A. 1984. Simultaneous algebraic reconstruction technique (SART): A superior implementation of the ART algorithm. Ultrasonic Imaging 6, 1, 81–94.Google ScholarCross Ref
    3. Banks, M. S., Bulbul, A., Albert, R. A., Narain, R., O’Brien, J. F., and Ward, G. 2014. The perception of surface material from disparity and focus cues. In Proc. Vision Sciences Society 14th Annual Meeting.Google Scholar
    4. Buckley, D., and Frisby, J. P. 1993. Interaction of stereo, texture and outline cues in the shape perception of three-dimensional ridges. Vision Research 33, 7, 919–933.Google ScholarCross Ref
    5. Campbell, F. 1957. The depth of field of the human eye. Optica Acta: International Journal of Optics 4, 4, 157–164.Google ScholarCross Ref
    6. Chambolle, A., and Pock, T. 2011. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision 40, 1, 120–145. Google ScholarDigital Library
    7. Coelho, J. M. P., Baião, A., and Vieira, P. 2013. Development of an optical simulator of the human eye. Proc. SPIE 8785, 8785CS–8785CS–8.Google Scholar
    8. Cossairt, O. S., Napoli, J., Hill, S. L., Dorval, R. K., and Favalora, G. E. 2007. Occlusion-capable multiview volumetric three-dimensional display. Appl. Opt. 46, 8 (Mar), 1244–1250.Google ScholarCross Ref
    9. Du, S.-P., Masia, B., Hu, S.-M., and Gutierrez, D. 2013. A metric of visual comfort for stereoscopic motion. ACM Trans. Graph. 32, 6 (Nov.), 222:1–222:9. Google ScholarDigital Library
    10. Emoto, M., Niida, T., and Okano, F. 2005. Repeated vergence adaptation causes the decline of visual functions in watching stereoscopic television. Display Technology, Journal of 1, 2, 328–340.Google ScholarCross Ref
    11. Esser, E., Zhang, X., and Chan, T. 2010. A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM Journal on Imaging Sciences 3, 4, 1015–1046. Google ScholarDigital Library
    12. Favalora, G. E., Napoli, J., Hall, D. M., Dorval, R. K., Giovinco, M., Richmond, M. J., and Chun, W. S. 2002. 100-million-voxel volumetric display. In Proc. SPIE, vol. 4712, 300–312.Google Scholar
    13. Granger, E., and Cupery, K. 1972. Optical merit function (SQF), which correlates with subjective image judgments. Photographic Science and Engineering 16, 3.Google Scholar
    14. He, B., and Yuan, X. 2012. Convergence analysis of primal-dual algorithms for a saddle-point problem: From contraction perspective. SIAM Journal on Imaging Sciences 5, 1, 119–149. Google ScholarDigital Library
    15. Held, R. T., Cooper, E. A., O’Brien, J. F., and Banks, M. S. 2010. Using blur to affect perceived distance and size. ACM Transactions on Graphics 29, 2 (Mar.), 19:1–16. Google ScholarDigital Library
    16. Hirsch, M., and Lanman, D., 2010. Build your own 3D display. ACM SIGGRAPH ASIA Course Notes. Google ScholarDigital Library
    17. Hoffman, D. M., Girshick, A. R., Akeley, K., and Banks, M. S. 2008. Vergence–accommodation conflicts hinder visual performance and cause visual fatigue. Journal of Vision 8, 3.Google ScholarCross Ref
    18. Hu, X., and Hua, H. 2013. An optical see-through multi-focal-plane stereoscopic display prototype enabling nearly correct focus cues. vol. 8648, 86481A.Google Scholar
    19. Hu, X., and Hua, H. 2014. Design and assessment of a depth-fused multi-focal-plane display prototype. J. Display Technol. 10, 4 (Apr), 308–316.Google ScholarCross Ref
    20. Hu, X., and Hua, H. 2014. High-resolution optical see-through multi-focal-plane head-mounted display using freeform optics. Opt. Express 22, 11 (Jun), 13896–13903.Google ScholarCross Ref
    21. Huang, F.-C., Lanman, D., Barsky, B. A., and Raskar, R. 2012. Correcting for optical aberrations using multilayer displays. ACM Transaction on Graphics 31 (Nov.). Google ScholarDigital Library
    22. Huang, F.-C., Wetzstein, G., Barsky, B. A., and Raskar, R. 2014. Eyeglasses-free display: Towards correcting visual aberrations with computational light field displays. ACM Trans. Graph. 33, 4 (July), 59:1–59:12. Google ScholarDigital Library
    23. Ives, F., 1903. Parallax stereogram and process of making same., Apr. 14. US Patent 725,567.Google Scholar
    24. Jakob, W., 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.Google Scholar
    25. Jones, A., McDowall, I., Yamada, H., Bolas, M., and Debevec, P. 2007. Rendering for an interactive 360° light field display. ACM Trans. Graph. 26, 3 (July). Google ScholarDigital Library
    26. Lambooij, M., Fortuin, M., Heynderickx, I., and Ijsselsteijn, W. 2009. Visual discomfort and visual fatigue of stereoscopic displays: A review. Journal of Imaging Science and Technology 53, 3, 30201-1–30201-14.Google ScholarCross Ref
    27. Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., and Gross, M. 2010. Nonlinear disparity mapping for stereoscopic 3D. ACM Trans. Graph. 29, 4 (July), 75:1–75:10. Google ScholarDigital Library
    28. Lanman, D., and Luebke, D. 2013. Near-eye light field displays. ACM Trans. Graph. 32, 6 (Nov.), 220:1–220:10. Google ScholarDigital Library
    29. Lanman, D., Hirsch, M., Kim, Y., and Raskar, R. 2010. Content-adaptive parallax barriers: Optimizing dual-layer 3D displays using low-rank light field factorization. ACM Trans. Graph. 29, 6 (Dec.), 163:1–163:10. Google ScholarDigital Library
    30. Lanman, D., Wetzstein, G., Hirsch, M., Heidrich, W., and Raskar, R. 2011. Polarization fields: Dynamic light field display using multi-layer LCDs. ACM Trans. Graph. 30, 6. Google ScholarDigital Library
    31. Levoy, M., Ng, R., Adams, A., Footer, M., and Horowitz, M. 2006. Light field microscopy. ACM Trans. Graph. 25, 3 (July), 924–934. Google ScholarDigital Library
    32. Lippmann, G. 1908. Épreuves réversibles donnant la sensation du relief. J. Phys. Theor. Appl. 7, 1, 821–825.Google ScholarCross Ref
    33. Liu, S., and Hua, H. 2010. A systematic method for designing depth-fused multi-focal plane three-dimensional displays. Opt. Express 18, 11 (May), 11562–11573.Google Scholar
    34. Liu, S., Hua, H., and Cheng, D. 2010. A novel prototype for an optical see-through head-mounted display with addressable focus cues. IEEE Transactions on Visualization and Computer Graphics 16, 3, 381–393. Google ScholarDigital Library
    35. Love, G. D., Hoffman, D. M., Hands, P. J., Gao, J., Kirby, A. K., and Banks, M. S. 2009. High-speed switchable lens enables the development of a volumetric stereoscopic display. Opt. Express 17, 18 (Aug), 15716–15725.Google ScholarCross Ref
    36. MacKenzie, K. J., Hoffman, D. M., and Watt, S. J. 2010. Accommodation to multiple-focal-plane displays: Implications for improving stereoscopic displays and for accommodation control. Journal of Vision 10, 8.Google ScholarCross Ref
    37. MacKenzie, K., Dickson, R., and Watt, S. 2012. Vergence and accommodation to multiple-image-plane stereoscopic displays: “real world” responses with practical image-plane separations? Journal of Electronic Imaging 21, 1.Google ScholarCross Ref
    38. Maimone, A., Wetzstein, G., Lanman, D., Hirsch, M., Raskar, R., and Fuchs, H. 2013. Focus 3D: Compressive accommodation display. ACM Trans. Graph. 32, 5, 1–13. Google ScholarDigital Library
    39. Mantiuk, R., Kim, K. J., Rempel, A. G., and Heidrich, W. 2011. HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. 30, 4 (July), 40:1–40:14. Google ScholarDigital Library
    40. Marshall, J. A., Burbeck, C. A., Ariely, D., Rolland, J. P., and Martin, K. E. 1996. Occlusion edge blur: a cue to relative visual depth. Journal of the Optical Society of America. A, Optics, image science, and vision 13, 4 (Apr.), 681–8.Google ScholarCross Ref
    41. Masia, B., Wetzstein, G., Didyk, P., and Gutierrez, D. 2013. A survey on computational displays: Pushing the boundaries of optics, computation, and perception. Computers and Graphics 37, 8, 1012–1038. Google ScholarDigital Library
    42. Mather, G., and Smith, D. R. R. 2002. Blur discrimination and its relation to blur-mediated depth perception. Perception 31, 10, 1211–1219.Google ScholarCross Ref
    43. Mathews, S., and Kruger, P. 1994. Spatiotemporal transfer function of human accommodation. Vision Research 34, 15.Google ScholarCross Ref
    44. Matusik, W., and Pfister, H. 2004. 3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans. Graph. 23, 3 (Aug.), 814–824. Google ScholarDigital Library
    45. Mendiburu, B. 2009. 3D movie making: Stereoscopic digital cinema from script to screen. Focal Press, Elsevier.Google Scholar
    46. Navarro, R. 2009. The optical design of the human eye: a critical review. Journal of Optometry 2, 1, 3–18.Google ScholarCross Ref
    47. Owens, D. 1980. A comparison of accommodative responsiveness and contrast sensitivity for sinusoidal gratings. Vision Research 20, 2, 159–167.Google ScholarCross Ref
    48. Palmer, S. E., and Brooks, J. L. 2008. Edge-region grouping in figure-ground organization and depth perception. Journal of Experimental Psychology: Human Perception and Performance 34, 6 (Dec), 1353–1371.Google ScholarCross Ref
    49. Pamplona, V. F., Oliveira, M. M., Aliaga, D. G., and Raskar, R. 2012. Tailored displays to compensate for visual aberrations. ACM Trans. Graph. 31, 4 (July), 81:1–81:12. Google ScholarDigital Library
    50. Perlin, K., Paxia, S., and Kollin, J. S. 2000. An autostereoscopic display. In Proc. 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’00, 319–326. Google ScholarDigital Library
    51. Ravikumar, S., Akeley, K., and Banks, M. S. 2011. Creating effective focus cues in multi-plane 3D displays. Opt. Express 19, 21 (Oct), 20940–20952.Google ScholarCross Ref
    52. Ryan, L., MacKenzie, K., and Watt, S. 2012. Multiple-focal-planes 3D displays: A practical solution to the vergence-accommodation conflict? In 3D Imaging (IC3D), 2012 International Conference on, 1–6.Google Scholar
    53. Shibata, T., Kim, J., Hoffman, D. M., and Banks, M. S. 2011. The zone of comfort: Predicting visual discomfort with stereo displays. Journal of Vision 11, 8.Google ScholarCross Ref
    54. Spring, K., and Stiles, W. S. 1948. Variation of pupil size with change in the angle at which the light stimulus strikes the retina. British J. Ophthalmol. 32, 6, 340–346.Google ScholarCross Ref
    55. Sullivan, A. 2004. DepthCube solid-state 3D volumetric display. In Proc. SPIE, vol. 5291, 279–284.Google Scholar
    56. Takaki, Y., Tanaka, K., and Nakamura, J. 2011. Super multi-view display with a lower resolution flat-panel display. Opt. Express 19, 5 (Feb), 4129–4139.Google ScholarCross Ref
    57. Takaki, Y. 2006. High-density directional display for generating natural three-dimensional images. Proc. IEEE 94, 3, 654–663.Google ScholarCross Ref
    58. van Ee, R., Banks, M. S., and T., B. B. 1999. An analysis of binocular slant contrast. Perception 28, 9, 1121–1145.Google ScholarCross Ref
    59. Watt, S. J., Akeley, K., Ernst, M. O., and Banks, M. S. 2005. Focus cues affect perceived depth. Journal of Vision 5, 10.Google ScholarCross Ref
    60. Wetzstein, G., Lanman, D., Heidrich, W., and Raskar, R. 2011. Layered 3D: Tomographic image synthesis for attenuation-based light field and high dynamic range displays. ACM Trans. Graph. 30, 4. Google ScholarDigital Library
    61. Wetzstein, G., Lanman, D., Gutierrez, D., and Hirsch, M., 2012. Computational displays: Combining optical fabrication, computational processing, and perceptual tricks to build the displays of the future. ACM SIGGRAPH Course Notes. Google ScholarDigital Library
    62. Wetzstein, G., Lanman, D., Hirsch, M., and Raskar, R. 2012. Tensor displays: Compressive light field synthesis using multilayer displays with directional backlighting. ACM Trans. Graph. (Proc. SIGGRAPH) 31, 4, 1–11. Google ScholarDigital Library
    63. Zannoli, M., Albert, R. A., Bulbul, A., Narain, R., O’Brien, J. F., and Banks, M. S. 2014. Correct blur and accommodation information is a reliable cue to depth ordering. In Proc. Vision Sciences Society 14th Annual Meeting.Google Scholar
    64. Zhu, M., and Chan, T. 2008. An efficient primal-dual hybrid gradient algorithm for total variation image restoration. Tech. rep., University of California, Los Angeles.Google Scholar
    65. Zwicker, M., Matusik, W., Durand, F., Pfister, H., and Forlines, C. 2006. Antialiasing for automultiscopic 3D displays. In ACM SIGGRAPH 2006 Sketches, SIGGRAPH ’06. Google ScholarDigital Library

ACM Digital Library Publication: