“A perceptual model for eccentricity-dependent spatio-temporal flicker fusion and its applications to foveated graphics” by Krajancich, Kellnhofer and Wetzstein

Virtual and augmented reality (VR/AR) displays strive to provide a resolution, framerate and field of view that matches the perceptual capabilities of the human visual system, all while constrained by limited compute budgets and transmission bandwidths of wearable computing systems. Foveated graphics techniques have emerged that could achieve these goals by exploiting the falloff of spatial acuity in the periphery of the visual field. However, considerably less attention has been given to temporal aspects of human vision, which also vary across the retina. This is in part due to limitations of current eccentricity-dependent models of the visual system. We introduce a new model, experimentally measuring and computationally fitting eccentricity-dependent critical flicker fusion thresholds jointly for both space and time. In this way, our model is unique in enabling the prediction of temporal information that is imperceptible for a certain spatial frequency, eccentricity, and range of luminance levels. We validate our model with an image quality user study, and use it to predict potential bandwidth savings 7X higher than those afforded by current spatial-only foveated models. As such, this work forms the enabling foundation for new temporally foveated graphics techniques.

1. Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency requirements for foveated rendering in virtual reality. ACM Transactions on Applied Perception (TAP) 14, 4 (2017), 1–13.Google ScholarDigital Library
2. MR Ali and T Amir. 1991. Critical flicker frequency under monocular and binocular conditions. Perceptual and motor skills 72, 2 (1991), 383–386.Google Scholar
3. Dale Allen and Robert F Hess. 1992. Is the visual field temporally homogeneous? Vision research 32, 6 (1992), 1075–1084.Google Scholar
4. Elena Arabadzhiyska, Okan Tarhan Tursun, Karol Myszkowski, Hans-Peter Seidel, and Piotr Didyk. 2017. Saccade landing position prediction for gaze-contingent rendering. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–12.Google ScholarDigital Library
5. Mark F Bradshaw and Brian J Rogers. 1999. Sensitivity to horizontal and vertical corrugations defined by binocular disparity. Vision research 39, 18 (1999), 3049–3056.Google Scholar
6. David Carmel, Nilli Lavie, and Geraint Rees. 2006. Conscious awareness of flicker in humans involves frontal and parietal cortex. Current biology 16, 9 (2006), 907–911.Google Scholar
7. Hanfeng Chen, Sung-Soo Kim, Sung-Hee Lee, Oh-Jae Kwon, and Jun-Ho Sung. 2005. Nonlinearity compensated smooth frame insertion for motion-blur reduction in LCD. In 2005 IEEE 7th Workshop on Multimedia Signal Processing. IEEE, 1–4.Google ScholarCross Ref
8. Kajal T. Claypool and Mark Claypool. 2007. On frame rate and player performance in first person shooter games. Multimedia Systems 13, 1 (Sept. 2007), 3–17.Google ScholarDigital Library
9. Mark Claypool and Kajal Claypool. 2009. Perspectives, Frame Rates and Resolutions: It’s All in the Game. In Proc. Int. Conference on Foundations of Digital Games. Association for Computing Machinery, New York, NY, USA, 42–49.Google ScholarDigital Library
10. Christine A Curcio and Kimberly A Allen. 1990. Topography of ganglion cells in human retina. Journal of comparative Neurology 300, 1 (1990), 5–25.Google ScholarCross Ref
11. John G Daugman. 1985. Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. JOSA A 2, 7 (1985), 1160–1169.Google ScholarCross Ref
12. James Davis, Yi-Hsuan Hsieh, and Hung-Chi Lee. 2015. Humans perceive flicker artifacts at 500 Hz. Scientific Reports 5, 1 (Feb. 2015), 7861.Google ScholarCross Ref
13. H de Lange Dzn. 1958. Research into the dynamic nature of the human fovea→cortex systems with intermittent and modulated light. I. Attenuation characteristics with white and colored light. Josa 48, 11 (1958), 777–784.Google ScholarCross Ref
14. Michael F Deering. 1998. The limits of human vision. In 2nd International Immersive Projection Technology Workshop, Vol. 2. 1.Google Scholar
15. Gyorgy Denes, Akshay Jindal, Aliaksei Mikhailiuk, and Rafał K. Mantiuk. 2020. A perceptual model of motion quality for rendering with adaptive refresh-rate and resolution. ACM Transactions on Graphics 39, 4 (July 2020), 133:133:1–133:133:17.Google ScholarDigital Library
16. Gyorgy Denes, Kuba Maruszczyk, George Ash, and Rafał K. Mantiuk. 2019. Temporal Resolution Multiplexing: Exploiting the limitations of spatio-temporal vision for more efficient VR rendering. IEEE Transactions on Visualization and Computer Graphics 25, 5 (May 2019), 2072–2082.Google ScholarCross Ref
17. Piotr Didyk, Elmar Eisemann, Tobias Ritschel, Karol Myszkowski, and Hans-Peter Seidel. 2010. Perceptually-motivated Real-time Temporal Upsampling of 3D Content for High-refresh-rate Displays. Computer Graphics Forum (Eurographics) 29, 2 (2010), 713–722.Google ScholarCross Ref
18. Andrew T. Duchowski, Nathan Cournia, and Hunter A. Murphy. 2004. Gaze-Contingent Displays: A Review. Cyberpsychology & behavior 7 (2004), 621–34.Google Scholar
19. Auria Eisen-Enosh, Nairouz Farah, Zvia Burgansky-Eliash, Idit Maharshak, Uri Polat, and Yossi Mandel. 2020. A dichoptic presentation device and a method for measuring binocular temporal function in the visual system. Experimental Eye Research 201 (2020), 108290.Google ScholarCross Ref
20. Auria Eisen-Enosh, Nairouz Farah, Zvia Burgansky-Eliash, Uri Polat, and Yossi Mandel. 2017. Evaluation of critical flicker-fusion frequency measurement methods for the investigation of visual temporal resolution. Scientific reports 7, 1 (2017), 1–9.Google Scholar
21. Sebastian Friston, Tobias Ritschel, and Anthony Steed. 2019. Perceptual Rasterization for Head-mounted Display Image Synthesis. ACM Trans. Graph. (Proc. SIGGRAPH 2019) 38, 4 (2019).Google Scholar
22. Wilson S. Geisler and Jeffrey S. Perry. 1998. Real-time foveated multiresolution system for low-bandwidth video communication. In Human Vision and Electronic Imaging III, Vol. 3299. International Society for Optics and Photonics, 294–305.Google Scholar
23. Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D Graphics. ACM Trans. Graph. (SIGGRAPH Asia) (2012).Google Scholar
24. E Hartmann, B Lachenmayr, and H Brettel. 1979. The peripheral critical flicker frequency. Vision Research 19, 9 (1979), 1019–1023.Google ScholarCross Ref
25. Chia-Chiang Ho, Ja-Ling Wu, and Wen-Huang Cheng. 2005. A practical foveation-based rate-shaping mechanism for MPEG videos. IEEE transactions on circuits and systems for video technology 15, 11 (2005), 1365–1372.Google Scholar
26. J-K Kamarainen, Ville Kyrki, and Heikki Kalviainen. 2006. Invariance properties of Gabor filter-based features-overview and applications. IEEE Transactions on image processing 15, 5 (2006), 1088–1099.Google ScholarDigital Library
27. Anton S. Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. 2019. DeepFovea: Neural Reconstruction for Foveated Rendering and Video Compression Using Learned Statistics of Natural Videos. ACM Trans. Graph. 38, 6, Article 212 (Nov. 2019), 13 pages.Google ScholarDigital Library
28. D. H. Kelly. 1979. Motion and vision. II. Stabilized spatio-temporal threshold surface. JOSA 69, 10 (Oct. 1979), 1340–1349.Google ScholarCross Ref
29. Jonghyun Kim, Youngmo Jeong, Michael Stengel, Kaan Akşit, Rachel Albert, Ben Boudaoud, Trey Greer, Joohwan Kim, Ward Lopes, Zander Majercik, et al. 2019. Foveated AR: dynamically-foveated augmented reality display. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–15.Google ScholarDigital Library
30. JJ Koenderink and AJ Van Doorn. 1978. Visual detection of spatial contrast; influence of location in the visual field, target extent and illuminance level. Biological Cybernetics 30, 3 (1978), 157–167.Google ScholarCross Ref
31. Jan J Koenderink, Maarten A Bouman, Albert E Bueno de Mesquita, and Sybe Slappendel. 1978c. Perimetry of contrast detection thresholds of moving spatial sine wave patterns. II. The far peripheral visual field (eccentricity 0°-50°). JOSA 68, 6 (1978), 850–854.Google ScholarCross Ref
32. Jan J Koenderink, Maarten A Bouman, Albert E Bueno de Mesquita, and Sybe Slappendel. 1978b. Perimetry of contrast detection thresholds of moving spatial sine wave patterns. III. The target extent as a sensitivity controlling parameter. JOSA 68, 6 (1978), 854–860.Google ScholarCross Ref
33. Jan J Koenderink, Maarten A Bouman, Albert E Bueno de Mesquita, and Sybe Slappendel. 1978a. Perimetry of contrast detection thresholds of moving spatial sine wave patterns. IV. The influence of the mean retinal illuminance. JOSA 68, 6 (1978), 860–865.Google ScholarCross Ref
34. George Alex Koulieris, Kaan Akşit, Michael Stengel, Rafał K Mantiuk, Katerina Mania, and Christian Richardt. 2019. Near-eye display and tracking technologies for virtual and augmented reality. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 493–519.Google Scholar
35. Brooke Krajancich, Petr Kellnhofer, and Gordon Wetzstein. 2020. Optimizing depth perception in virtual and augmented reality through gaze-contingent stereo rendering. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–10.Google ScholarDigital Library
36. Zhi Li, Anne Aaron, Ioannis Katsavounidis, Anush Moorthy, and Megha Manohara. 2016. Toward a practical perceptual video quality metric. The Netflix Tech Blog 6, 2 (2016).Google Scholar
37. David Luebke and Benjamin Hallen. 2001. Perceptually driven simplification for interactive rendering. In Proceedings of the 12th Eurographics conference on Rendering (EGWR’01). Eurographics Association, Goslar, DEU, 223–234.Google ScholarDigital Library
38. S Marĉelja. 1980. Mathematical description of the responses of simple cortical cells. JOSA 70, 11 (1980), 1297–1300.Google ScholarCross Ref
39. Michael Mauderer, Simone Conte, Miguel A Nacenta, and Dhanraj Vishwanath. 2014. Depth perception with gaze-contingent depth of field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 217–226.Google ScholarDigital Library
40. John D. McCarthy, M. Angela Sasse, and Dimitrios Miras. 2004. Sharp or smooth? comparing the effects of quantization vs. frame rate for streamed video. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’04). Association for Computing Machinery, New York, NY, USA, 535–542.Google Scholar
41. Arian Mehrfard, Javad Fotouhi, Giacomo Taylor, Tess Forster, Nassir Navab, and Bernhard Fuerst. 2019. A comparative analysis of virtual reality head-mounted display systems. arXiv preprint arXiv:1912.02913 (2019).Google Scholar
42. Hunter Murphy and Andrew T. Duchowski. 2001. Gaze-Contingent Level Of Detail Rendering. (2001).Google Scholar
43. Toshikazu Ohshima, Hiroyuki Yamamoto, and Hideyuki Tamura. 1996. Gaze-directed adaptive rendering for interacting with virtual space. In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium. IEEE, 103–110.Google ScholarCross Ref
44. Nitish Padmanaban, Robert Konrad, Tal Stramer, Emily A Cooper, and Gordon Wetzstein. 2017. Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays. Proceedings of the National Academy of Sciences 114, 9 (2017), 2183–2188.Google ScholarCross Ref
45. Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics (TOG) 35, 6 (Nov. 2016), 179:1–179:12.Google ScholarDigital Library
46. Jonathan W Peirce. 2007. PsychoPy-psychophysics software in Python. Journal of neuroscience methods 162, 1-2 (2007), 8–13.Google ScholarCross Ref
47. Denis G Pelli and Peter Bex. 2013. Measuring contrast sensitivity. Vision research 90 (2013), 10–14.Google Scholar
48. JG Robson. 1993. Contrast sensitivity: One hundred years of clinical measurement in Proceedings of the Retina Research Foundation Symposia, Vol. 5 Eds R. Shapley, D. Man-Kit Lam.Google Scholar
49. John G Robson. 1966. Spatial and temporal contrast-sensitivity functions of the visual system. Josa 56, 8 (1966), 1141–1142.Google ScholarCross Ref
50. J. Gordon Robson and Norma Graham. 1981. Probability summation and regional variation in contrast sensitivity across the visual field. Vision research 21, 3 (1981), 409–418.Google Scholar
51. Manuel Romero-Gómez, Juan Córdoba, Rodrigo Jover, Juan A Del Olmo, Marta Ramírez, Ramón Rey, Enrique De Madaria, Carmina Montoliu, David Nuñez, Montse Flavia, et al. 2007. Value of the critical flicker frequency in patients with minimal hepatic encephalopathy. Hepatology 45, 4 (2007), 879–885.Google ScholarCross Ref
52. Ruth Rosenholtz. 2016. Capabilities and limitations of peripheral vision. Annual Review of Vision Science 2 (2016), 437–457.Google Scholar
53. Jyrki Rovamo and Antti Raninen. 1984. Critical flicker frequency and M-scaling of stimulus size and retinal illuminance. Vision research 24, 10 (1984), 1127–1131.Google Scholar
54. Jyrki Rovamo and Antti Raninen. 1988. Critical flicker frequency as a function of stimulus area and luminance at various eccentricities in human cone vision: a revision of Granit-Harper and Ferry-Porter laws. Vision research 28, 7 (1988), 785–790.Google Scholar
55. Theodore C. Ruch and John F. Fulton. 1960. Medical Physiology and Biophysics. Journal of Medical Education 35 (1960), 1067. Issue 11.Google Scholar
56. Daniel Scherzer, Lei Yang, Oliver Mattausch, Diego Nehab, Pedro V. Sander, Michael Wimmer, and Elmar Eisemann. 2012. Temporal Coherence Methods in Real-Time Rendering. Computer Graphics Forum 31, 8 (2012), 2378–2408.Google ScholarDigital Library
57. Kalpana Seshadrinathan, Rajiv Soundararajan, Alan Conrad Bovik, and Lawrence K Cormack. 2010. Study of subjective and objective quality assessment of video. IEEE transactions on Image Processing 19, 6 (2010), 1427–1441.Google Scholar
58. Jie Shen, Christopher A Clark, P Sarita Soni, and Larry N Thibos. 2010. Peripheral refraction with and without contact lens correction. Optometry and vision science: official publication of the American Academy of Optometry 87, 9 (2010), 642.Google Scholar
59. Raunak Sinha, Mrinalini Hoon, Jacob Baudin, Haruhisa Okawa, Rachel OL Wong, and Fred Rieke. 2017. Cellular and circuit mechanisms shaping the perceptual properties of the primate fovea. Cell 168, 3 (2017), 413–426.Google ScholarCross Ref
60. Jan P Springer, Stephan Beck, Felix Weiszig, Dirk Reiners, and Bernd Froehlich. 2007. Multi-frame rate rendering and display. In 2007 IEEE Virtual Reality Conference. IEEE, 195–202.Google ScholarCross Ref
61. Philip A Stanley and A Kelvin Davies. 1995. The effect of field of view size on steady-state pupil diameter. Ophthalmic and Physiological Optics 15, 6 (1995), 601–603.Google ScholarCross Ref
62. Qi Sun, Fu-Chung Huang, Joohwan Kim, Li-Yi Wei, David Luebke, and Arie Kaufman. 2017. Perceptually-guided foveation for light field displays. ACM Transactions on Graphics 36, 6 (Nov. 2017), 192:1–192:13.Google ScholarDigital Library
63. LN Thibos, DJ Walsh, and FE Cheney. 1987. Vision beyond the resolution limit: aliasing in the periphery. Vision Research 27, 12 (1987), 2193–2197.Google ScholarCross Ref
64. Okan Tarhan Tursun, Elena Arabadzhiyska-Koleva, Marek Wernikowski, Radosław Mantiuk, Hans-Peter Seidel, Karol Myszkowski, and Piotr Didyk. 2019. Luminance-contrast-aware foveated rendering. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–14.Google ScholarDigital Library
65. Christopher W Tyler. 1987. Analysis of visual modulation sensitivity. III. Meridional variations in peripheral flicker sensitivity. JOSA A 4, 8 (1987), 1612–1619.Google ScholarCross Ref
66. Christopher W Tyler and Russell D Hamer. 1990. Analysis of visual modulation sensitivity. IV. Validity of the Ferry-Porter law. JOSA A 7, 4 (1990), 743–758.Google ScholarCross Ref
67. Christopher W Tyler and Russell D Hamer. 1993. Eccentricity and the Ferry-Porter law. JOSA A 10, 9 (1993), 2084–2087.Google ScholarCross Ref
68. Floris L Van Nes and Maarten A Bouman. 1967. Spatial modulation transfer in the human eye. JOSA 57, 3 (1967), 401–406.Google ScholarCross Ref
69. Zhou Wang, Ligang Lu, and Alan C Bovik. 2003. Foveation scalable video coding with automatic fixation selection. IEEE Transactions on Image Processing 12, 2 (2003), 243–254.Google ScholarDigital Library
70. Andrew B Watson. 2018. The Field of View, the Field of Resolution, and the Field of Contrast Sensitivity. Journal of Perceptual Imaging 1, 1 (2018), 10505–1.Google ScholarCross Ref
71. Andrew B Watson and Albert J Ahumada. 2016. The pyramid of visibility. Electronic Imaging 2016, 16 (2016), 1–6.Google Scholar
72. Andrew B Watson and Denis G Pelli. 1983. QUEST: A Bayesian adaptive psychometric method. Perception & psychophysics 33, 2 (1983), 113–120.Google Scholar
73. Thomas P Weldon, William E Higgins, and Dennis F Dunn. 1996. Efficient Gabor filter design for texture segmentation. Pattern recognition 29, 12 (1996), 2005–2015.Google Scholar
74. Lei Xiao, Salah Nouri, Matt Chapman, Alexander Fix, Douglas Lanman, and Anton Kaplanyan. 2020. Neural Supersampling for Real-Time Rendering. ACM Trans. Graph. 39, 4, Article 142 (July 2020), 12 pages.Google ScholarDigital Library