“Perceptually-guided foveation for light field displays”
Conference:
Type(s):
Title:
- Perceptually-guided foveation for light field displays
Session/Category Title: Displays
Presenter(s)/Author(s):
Abstract:
A variety of applications such as virtual reality and immersive cinema require high image quality, low rendering latency, and consistent depth cues. 4D light field displays support focus accommodation, but are more costly to render than 2D images, resulting in higher latency.The human visual system can resolve higher spatial frequencies in the fovea than in the periphery. This property has been harnessed by recent 2D foveated rendering methods to reduce computation cost while maintaining perceptual quality. Inspired by this, we present foveated 4D light fields by investigating their effects on 3D depth perception. Based on our psychophysical experiments and theoretical analysis on visual and display bandwidths, we formulate a content-adaptive importance model in the 4D ray space. We verify our method by building a prototype light field display that can render only 16% — 30% rays without compromising perceptual quality.
References:
1. Kurt Akeley, Simon J. Watt, Ahna Reza Girshick, and Martin S. Banks. 2004. A Stereo Display Prototype with Multiple Focal Distances. ACM Trans. Graph. 23, 3 (2004), 804–813.
2. Jin-Xiang Chai, Xin Tong, Shing-Chow Chan, and Heung-Yeung Shum. 2000. Plenoptic Sampling. In SIGGRAPH ’00. 307–318.
3. WN Charman and J Tucker. 1978. Accommodation as a function of object form. Optometry & Vision Science 55, 2 (1978), 84–92. Cross Ref
4. Christine A. Curcio and Kimberly A. Allen. 1990. Topography of ganglion cells in human retina. The Journal of Comparative Neurology 300, 1 (1990), 5–25. Cross Ref
5. Donald G Dansereau, Glenn Schuster, Joseph Ford, and Gordon Wetzstein. 2017. A Wide-Field-of-View Monocentric Light Field Camera. In CVPR ’17. Cross Ref
6. Neville Drasdo, C. Leigh Millican, Charles R. Katholi, and Christine A. Curcio. 2007. The length of Henle fibers in the human retina and a model of ganglion receptive field density in the visual field. Vision Research 47, 22 (2007), 2901 — 2911.
7. Greg Egan. 1994. Permutation City. Millennium Orion Publishing Group.
8. Kevin Egan, Frédo Durand, and Ravi Ramamoorthi. 2011a. Practical Filtering for Efficient Ray-traced Directional Occlusion. ACM Trans. Graph. 30, 6, Article 180 (2011), 10 pages.
9. Kevin Egan, Florian Hecht, Frédo Durand, and Ravi Ramamoorthi. 2011b. Frequency Analysis and Sheared Filtering for Shadow Light Fields of Complex Occluders. ACM Trans. Graph. 30, 2, Article 9 (2011), 13 pages.
10. Kevin Egan, Yu-Ting Tseng, Nicolas Holzschuch, Frédo Durand, and Ravi Ramamoorthi. 2009. Frequency Analysis and Sheared Reconstruction for Rendering Motion Blur. ACM Trans. Graph. 28, 3, Article 93 (2009), 13 pages.
11. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The Lumigraph. In SIGGRAPH ’96. 43–54.
12. Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D Graphics. ACM Trans. Graph. 31, 6, Article 164 (2012), 10 pages.
13. Toshiya Hachisuka, Wojciech Jarosz, Richard Peter Weistroffer, Kevin Dale, Greg Humphreys, Matthias Zwicker, and Henrik Wann Jensen. 2008. Multidimensional Adaptive Sampling and Reconstruction for Ray Tracing. ACM Trans. Graph. 27, 3, Article 33 (2008), 10 pages.
14. Felix Heide, Gordon Wetzstein, Ramesh Raskar, and Wolfgang Heidrich. 2013. Adaptive Image Synthesis for Compressive Displays. ACM Trans. Graph. 32, 4, Article 132 (2013), 12 pages.
15. Fu-Chung Huang, Kevin Chen, and Gordon Wetzstein. 2015. The Light Field Stereoscope: Immersive Computer Graphics via Factored Near-eye Light Field Displays with Focus Cues. ACM Trans. Graph. 34, 4, Article 60 (2015), 12 pages.
16. Fu-Chung Huang, Gordon Wetzstein, Brian A. Barsky, and Ramesh Raskar. 2014. Eyeglasses-free Display: Towards Correcting Visual Aberrations with Computational Light Field Displays. ACM Trans. Graph. 33, 4, Article 59 (2014), 12 pages.
17. Julian Iseringhausen, Bastian Goldlücke, Nina Pesheva, Stanimir Iliev, Alexander Wender, Martin Fuchs, and Matthias B. Hullin. 2017. 4D Imaging Through Spray-on Optics. ACM Trans. Graph. 36, 4, Article 35 (2017), 11 pages.
18. Moritz Kassner, William Patera, and Andreas Bulling. 2014. Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction. In UbiComp ’14 Adjunct. 1151–1160.
19. Joohwan Kim, Qi Sun, Fu-Chung Huang, Li-Yi Wei, David Luebke, and Arie Kaufman. 2017. Perceptual Studies for Foveated Light Field Displays. CoRR abs/1708.06034 (2017).
20. Matias Koskela, Timo Viitanen, Pekka Jääskeläinen, and Jarmo Takala. 2016. Foveated Path Tracing. In ISVC ’16. 723–732.
21. Douglas Lanman and David Luebke. 2013. Near-eye Light Field Displays. ACM Trans. Graph. 32, 6, Article 220 (2013), 10 pages.
22. Jaakko Lehtinen, Timo Aila, Jiawen Chen, Samuli Laine, and Frédo Durand. 2011. Temporal Light Field Reconstruction for Rendering Distribution Effects. ACM Trans. Graph. 30, 4, Article 55 (2011), 12 pages.
23. Jaakko Lehtinen, Timo Aila, Samuli Laine, and Frédo Durand. 2012. Reconstructing the Indirect Light Field for Global Illumination. ACM Trans. Graph. 31, 4, Article 51 (2012), 10 pages.
24. Anat Levin, Samuel W. Hasinoff, Paul Green, Frédo Durand, and William T. Freeman. 2009. 4D Frequency Analysis of Computational Cameras for Depth of Field Extension. ACM Trans. Graph. 28, 3, Article 97 (2009), 14 pages.
25. Marc Levoy and Pat Hanrahan. 1996. Light Field Rendering. In SIGGRAPH ’96. 31–42.
26. Lester C. Loschky and Gary S. Wolverton. 2007. How Late Can You Update Gaze-contingent Multiresolutional Displays Without Detection? ACM Trans. Multimedia Comput. Commun. Appl. 3, 4, Article 7 (2007), 10 pages.
27. Andrew Maimone and Henry Fuchs. 2013. Computational augmented reality eyeglasses. In ISMAR ’13. 29–38. Cross Ref
28. Andrew Maimone, Gordon Wetzstein, Matthew Hirsch, Douglas Lanman, Ramesh Raskar, and Henry Fuchs. 2013. Focus 3D: Compressive Accommodation Display. ACM Trans. Graph. 32, 5, Article 153 (2013), 13 pages.
29. Soham Mehta, Brandon Wang, and Ravi Ramamoorthi. 2012. Axis-Aligned Filtering for Interactive Sampled Soft Shadows. ACM Trans. Graph. 31, 6 (2012), 163:1–163:10.
30. Rahul Narain, Rachel A. Albert, Abdullah Bulbul, Gregory J. Ward, Martin S. Banks, and James F. O’Brien. 2015. Optimal Presentation of Imagery with Focus Cues on Multi-plane Displays. ACM Trans. Graph. 34, 4, Article 59 (2015), 12 pages.
31. Ren Ng. 2005. Fourier Slice Photography. ACM Trans. Graph. 24, 3 (2005), 735–744.
32. Vitor F. Pamplona, Manuel M. Oliveira, Daniel G. Aliaga, and Ramesh Raskar. 2012. Tailored Displays to Compensate for Visual Aberrations. ACM Trans. Graph. 31, 4, Article 81 (2012), 12 pages.
33. Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards Foveated Rendering for Gaze-tracked Virtual Reality. ACM Trans. Graph. 35, 6, Article 179 (2016), 12 pages.
34. Anjul Patney, Marina Zannoli, George-Alex Koulieris, Joohwan Kim, Gordon Wetzstein, and Frank Steinicke. 2017. Applications of Visual Perception to Virtual Reality Rendering. In SIGGRAPH ’17 Courses. Article 1, 38 pages.
35. Yuyang Qiu and Ling Zhu. 2010. The best approximation of the sinc function by a polynomial of degree with the square norm. Journal of Inequalities and Applications 2010, 1 (2010), 1–12. Cross Ref
36. Jonathan Ragan-Kelley, Jaakko Lehtinen, Jiawen Chen, Michael Doggett, and Frédo Durand. 2011. Decoupled Sampling for Graphics Pipelines. ACM Trans. Graph. 30, 3, Article 17 (2011), 17 pages.
37. V. Ramachandra, K. Hirakawa, M. Zwicker, and T. Nguyen. 2011. Spatioangular Pre-filtering for Multiview 3D Displays. IEEE Transactions on Visualization and Computer Graphics 17, 5 (2011), 642–654.
38. John Siderov and Ronald S Harwerth. 1995. Stereopsis, spatial frequency and retinal eccentricity. Vision research 35, 16 (1995), 2329–2337.
39. Y. Takaki. 2006. High-Density Directional Display for Generating Natural Three-Dimensional Images. Proc. IEEE 94, 3 (2006), 654–663. Cross Ref
40. Yasuhiro Takaki, Kosuke Tanaka, and Junya Nakamura. 2011. Super multi-view display with a lower resolution flat-panel display. Opt. Express 19, 5 (2011), 4129–4139. Cross Ref
41. LN Thibos, FE Cheney, and DJ Walsh. 1987. Retinal limits to the detection and resolution of gratings. JOSA A 4, 8 (1987), 1524–1529. Cross Ref
42. Christopher W Tyler. 1987. Analysis of visual modulation sensitivity. III. Meridional variations in peripheral flicker sensitivity. JOSA A 4, 8 (1987), 1612–1619. Cross Ref
43. K. Vaidyanathan, M. Salvi, R. Toth, T. Foley, T. Akenine-Möller, J. Nilsson, J. Munkberg, J. Hasselgren, M. Sugihara, P. Clarberg, T. Janczak, and A. Lefohn. 2014. Coarse Pixel Shading. In HPG ’14. 9–18.
44. Thomas S. A. Wallis, Matthias Bethge, and Felix A. Wichmann. 2016. Testing models of peripheral encoding using metamerism in an oddity paradigm. Journal of Vision 16, 2 (2016), 4. Cross Ref
45. Andrew B. Watson. 2014. A formula for human retinal ganglion cell receptive field density as a function of visual field location. Journal of Vision 14, 7 (2014), 15. Cross Ref
46. Andrew B. Watson and Albert J. Ahumada. 2011. Blur clarified: A review and synthesis of blur discrimination. Journal of Vision 11, 5 (2011), 10. Cross Ref
47. Li-Yi Wei, Chia-Kai Liang, Graham Myhre, Colvin Pitts, and Kurt Akeley. 2015. Improving Light Field Camera Sample Design with Irregularity and Aberration. ACM Trans. Graph. 34, 4, Article 152 (2015), 11 pages.
48. Alexander Wender, Julian Iseringhausen, Bastian Goldluecke, Martin Fuchs, and Matthias B. Hullin. 2015. Light Field Imaging through Household Optics. In Vision, Modeling & Visualization.
49. Gordon Wetzstein, Douglas Lanman, Wolfgang Heidrich, and Ramesh Raskar. 2011. Layered 3D: Tomographic Image Synthesis for Attenuation-based Light Field and High Dynamic Range Displays. ACM Trans. Graph. 30, 4, Article 95 (2011), 12 pages.
50. Gordon Wetzstein, Douglas Lanman, Matthew Hirsch, and Ramesh Raskar. 2012. Tensor Displays: Compressive Light Field Synthesis Using Multilayer Displays with Directional Backlighting. ACM Trans. Graph. 31, 4, Article 80 (2012), 11 pages.
51. Ling-Qi Yan, Soham Uday Mehta, Ravi Ramamoorthi, and Fredo Durand. 2015. Fast 4D Sheared Filtering for Interactive Rendering of Distribution Effects. ACM Trans. Graph. 35, 1, Article 7 (2015), 13 pages.
52. Marina Zannoli, Gordon D. Love, Rahul Narain, and Martin S. Banks. 2016. Blur and the perception of depth at occlusions. Journal of Vision 16, 6 (2016), 17. Cross Ref
53. Matthias Zwicker, Wojciech Matusik, Frédo Durand, Hanspeter Pfister, and Clifton Forlines. 2006. Antialiasing for Automultiscopic 3D Displays. In SIGGRAPH ’06 Sketches. Article 107.
