“Noise-based enhancement for foveated rendering” by Tariq, Tursun and Didyk

Human visual sensitivity to spatial details declines towards the periphery. Novel image synthesis techniques, so-called foveated rendering, exploit this observation and reduce the spatial resolution of synthesized images for the periphery, avoiding the synthesis of high-spatial-frequency details that are costly to generate but not perceived by a viewer. However, contemporary techniques do not make a clear distinction between the range of spatial frequencies that must be reproduced and those that can be omitted. For a given eccentricity, there is a range of frequencies that are detectable but not resolvable. While the accurate reproduction of these frequencies is not required, an observer can detect their absence if completely omitted. We use this observation to improve the performance of existing foveated rendering techniques. We demonstrate that this specific range of frequencies can be efficiently replaced with procedural noise whose parameters are carefully tuned to image content and human perception. Consequently, these frequencies do not have to be synthesized during rendering, allowing more aggressive foveation, and they can be replaced by noise generated in a less expensive post-processing step, leading to improved performance of the rendering system. Our main contribution is a perceptually-inspired technique for deriving the parameters of the noise required for the enhancement and its calibration. The method operates on rendering output and runs at rates exceeding 200 FPS at 4K resolution, making it suitable for integration with real-time foveated rendering systems for VR and AR devices. We validate our results and compare them to the existing contrast enhancement technique in user experiments.

1. Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency requirements for foveated rendering in virtual reality. ACM Trans. Appl. Percept. 14, 4 (2017).Google ScholarDigital Library
2. Roger S. Anderson, Margarita B. Zlatkova, and Shaban Demirel. 2002. What limits detection and resolution of short-wavelength sinusoidal gratings across the retina? Vision Res. 42, 8 (2002), 981–990.Google ScholarCross Ref
3. Stephen J. Anderson, Kathy T. Mullen, and Robert F. Hess. 1991. Human peripheral spatial resolution for achromatic and chromatic stimuli: limits imposed by optical and retinal factors. J. Physiol. (Lond.) 442, 1 (1991), 47–64.Google ScholarCross Ref
4. H. R. Aubert and C. F. R. Foerster. 1857. Beitrage zur kenntnisse der indirecten sehens [Translation: Contributions of knowledge to indirect vision]. Graefes Archiv fur Ophthalmologie 3 (1857).Google Scholar
5. Horace B. Barlow. 1961. Possible principles underlying the transformation of sensory messages. Sensory communication 1, 01 (1961).Google Scholar
6. Ryan Beams, Brendan Collins, Andrea S. Kim, and Aldo Badano. 2020. Angular dependence of the spatial resolution in virtual reality displays. In IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 836–841.Google ScholarCross Ref
7. Pierre Bénard, Ares Lagae, Peter Vangorp, Sylvain Lefebvre, George Drettakis, and Joëlle Thollot. 2010. A dynamic noise primitive for coherent stylization. Comput Graph Forum 29, 4 (2010), 1497–1506.Google ScholarDigital Library
8. G. Browder and W. Chambers. 1988. Eye-enslaved area-of-interest display systems. In Proc. of Flight Simulation Technologies Conference.Google Scholar
9. Christine A. Curcio, Kenneth R. Sloan, Robert E. Kalina, and Anita E. Hendrickson. 1990. Human photoreceptor topography. Journal of comparative neurology 292, 4 (1990), 497–523.Google ScholarCross Ref
10. Russell L. De Valois, Duane G. Albrecht, and Lisa G. Thorell. 1982. Spatial frequency selectivity of cells in macaque visual cortex. Vision Res. 22, 5 (1982), 545–559.Google ScholarCross Ref
11. Arturo Deza, Aditya Jonnalagadda, and Miguel P. Eckstein. 2019. Towards metamerism via foveated style transfer. In International Conference on Learning Representations. Openreview.net, New Orleans, USA.Google Scholar
12. Piotr Didyk, Tobias Ritschel, Elmar Eisemann, Karol Myszkowski, and Hans-Peter Seidel. 2010. Adaptive image-space stereo view synthesis. In VMV. 299–306.Google Scholar
13. A. Eugen Fick. 1898. Ueber stäbchensehschärfe und zapfensehschärfe. Albrecht von Graefes Archiv für Ophthalmologie 45, 2 (1898), 336–356.Google Scholar
14. Jeremy Freeman and Eero P. Simoncelli. 2011. Metamers of the ventral stream. Nature neuroscience 14, 9 (2011), 1195–1201.Google Scholar
15. William E. Glenn. 1994. Real-time display systems, present and future. In Visual Science and Engineering: Models and Applications. Marcel Dekker, 387–413.Google Scholar
16. Daniel G. Green. 1970. Regional variations in the visual acuity for interference fringes on the retina. J. Physiol. (Lond.) 207, 2 (1970), 351–356.Google ScholarCross Ref
17. Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Trans. Graphics 31, 6 (2012).Google ScholarDigital Library
18. Andrew M. Haun. 2021. What is visible across the visual field? Neuroscience of Consciousness 2021, 1 (06 2021).Google Scholar
19. E. Hering. 1899. Concerning the limits of visual acuity. Ber. d. math.-phys. Kl. d. K. Sachs. Gesellsch. d. Wissensch. zu Leipzig (1899).Google Scholar
20. Joy Hirsch and Christine A. Curcio. 1989. The spatial resolution capacity of human foveal retina. Vision Res. 29, 9 (1989), 1095–1101.Google ScholarCross Ref
21. David Hoffman, Zoe Meraz, and Eric Turner. 2018. Limits of peripheral acuity and implications for VR system design. Journal of the Society for Information Display 26, 8 (2018), 483–495.Google ScholarCross Ref
22. Aapo Hyvärinen, Jarmo Hurri, and Patrick O. Hoyer. 2009. Natural image statistics: A probabilistic approach to early computational vision. Vol. 39. Springer Science & Business Media.Google Scholar
23. Anton Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. 2019. DeepFovea: Neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. ACM Trans. Graphics 38, 6 (2019).Google ScholarDigital Library
24. Philip Kortum and Wilson S. Geisler. 1996. Implementation of a foveated image coding system for image bandwidth reduction. In Human Vision and Electronic Imaging, Vol. 2657. International Society for Optics and Photonics, 350–360.Google Scholar
25. Ares Lagae, Sylvain Lefebvre, Rob Cook, Tony DeRose, George Drettakis, David S. Ebert, John P. Lewis, Ken Perlin, and Matthias Zwicker. 2010. A survey of procedural noise functions. Comput Graph Forum 29, 8 (2010), 2579–2600.Google ScholarCross Ref
26. Ares Lagae, Sylvain Lefebvre, George Drettakis, and Philip Dutré. 2009. Procedural noise using sparse Gabor convolution. ACM Trans. Graphics 28, 3 (2009).Google ScholarDigital Library
27. Gordon E. Legge and Daniel Kersten. 1987. Contrast discrimination in peripheral vision. J. Opt. Soc. Am. A 4, 8 (Aug 1987), 1594–1598.Google Scholar
28. Jerome Y. Lettvin et al. 1976. On seeing sidelong. The Sciences 16, 4 (1976), 10–20.Google ScholarCross Ref
29. Dennis M. Levi and Stanley A. Klein. 1986. Sampling in spatial vision. Nature 320, 6060 (1986), 360–362.Google ScholarCross Ref
30. Marc Levoy and Ross Whitaker. 1990. Gaze-directed volume rendering. SIGGRAPH 24, 2 (feb 1990), 217–223.Google ScholarDigital Library
31. Rafał Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graphics 30, 4 (2011).Google ScholarDigital Library
32. Rafał K. Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney. 2021. FovVideoVDP: A visible difference predictor for wide field-of-view video. ACM Trans. Graphics 40, 4 (2021).Google ScholarDigital Library
33. Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney. 2018. Kernel foveated rendering. Proc. of the ACM on Computer Graphics and Interactive Techniques 1, 1 (2018).Google ScholarDigital Library
34. Michel Millodot, Chris A. Johnson, Anne Lamont, and Herschel W. Leibowitz. 1975. Effect of dioptrics on peripheral visual acuity. Vision Res. 15, 12 (1975), 1357–1362.Google ScholarCross Ref
35. Bipul Mohanto, A. B. M. Tariqul Islam, Enrico Gobbetti, and Oliver Staadt. 2021. An integrative view of foveated rendering. Computers & Graphics (2021).Google Scholar
36. G. A. Østerberg. 1935. Topography of the layer of the rods and cones in the human retina. Acta ophthalmol 13, 6 (1935).Google Scholar
37. Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Trans. Graphics 35, 6 (2016).Google ScholarDigital Library
38. Eli Peli. 1990. Contrast in complex images. J. Opt. Soc. Am. A, Optics and image science 7 10 (1990), 2032–2040.Google ScholarCross Ref
39. Javier Portilla and Eero P. Simoncelli. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. Int J Comput Vision 40, 1 (2000), 49–70.Google ScholarDigital Library
40. Ruth Rosenholtz. 2016. Capabilities and limitations of peripheral vision. Annual Review of Vision Science 2, 1 (2016), 437–457.Google ScholarCross Ref
41. Ethan A. Rossi and Austin Roorda. 2010. The relationship between visual resolution and cone spacing in the human fovea. Nature neuroscience 13, 2 (2010), 156–157.Google Scholar
42. Jyrki Rovamo and Veijo Virsu. 1979. An estimation and application of the human cortical magnification factor. Experimental brain research 37, 3 (1979), 495–510.Google Scholar
43. Jyrki Rovamo, Veijo Virsu, and Risto Näsänen. 1978. Cortical magnification factor predicts the photopic contrast sensitivity of peripheral vision. Nature 271, 5640 (1978), 54–56.Google Scholar
44. Daniel L. Ruderman. 1994. The statistics of natural images. Network: computation in neural systems 5, 4 (1994), 517–548.Google Scholar
45. Heiko H. Schütt and Felix A. Wichmann. 2017. An image-computable psychophysical spatial vision model. J. Vis. 17, 12 (2017).Google ScholarCross Ref
46. Boubakar Sere, Christian Marendaz, and Jeanny Herault. 2000. Nonhomogeneous resolution of images of natural scenes. Perception 29, 12 (2000), 1403–1412.Google ScholarCross Ref
47. Eero P. Simoncelli and Bruno A. Olshausen. 2001. Natural image statistics and neural representation. Annual review of neuroscience 24, 1 (2001), 1193–1216.Google Scholar
48. Michael Stengel, Steve Grogorick, Martin Eisemann, and Marcus Magnor. 2016. Adaptive image-space sampling for gaze-contingent real-time rendering. In Comput Graph Forum, Vol. 35. Wiley Online Library, 129–139.Google Scholar
49. Hans Strasburger, Ingo Rentschler, and Martin Jüttner. 2011. Peripheral vision and pattern recognition: A review. J. Vis. 11, 5 (12 2011).Google ScholarCross Ref
50. Nicholas T. Swafford, José A. Iglesias-Guitian, Charalampos Koniaris, Bochang Moon, Darren Cosker, and Kenny Mitchell. 2016. User, metric, and computational evaluation of foveated rendering methods. In Proc. of the ACM Symposium on Applied Perception. 7–14.Google ScholarDigital Library
51. L. N. Thibos. 1998. Acuity perimetry and the sampling theory of visual resolution. Optometry and vision science: official publication of the American Academy of Optometry 75, 6 (1998), 399–406.Google Scholar
52. L. N. Thibos, F. E. Cheney, and D. J. Walsh. 1987a. Retinal limits to the detection and resolution of gratings. J. Opt. Soc. Am. A 4, 8 (1987), 1524–1529.Google ScholarCross Ref
53. L. N. Thibos, D. L. Still, and A. Bradley. 1996. Characterization of spatial aliasing and contrast sensitivity in peripheral vision. Vision Res. 36, 2 (1996), 249–258.Google ScholarCross Ref
54. L. N. Thibos and D. J. Walsh. 1985. Detection of high frequency gratings in the periphery. J. Opt. Soc. Am. A 2 (1985).Google Scholar
55. L. N. Thibos, D. J. Walsh, and F. E. Cheney. 1987b. Vision beyond the resolution limit: aliasing in the periphery. Vision Res. 27, 12 (1987), 2193–2197.Google ScholarCross Ref
56. H. M. Tong and R. A. Fisher. 1984. Progress report on an eye-slaved area-of-interest visual display. Technical Report. Singer Co Silver Spring Md Link Simul. Systems Div.Google Scholar
57. Norimichi Tsumura, Chizuko Endo, Hideaki Haneishi, and Yoichi Miyake. 1996. Image compression and decompression based on gazing area. In Human Vision and Electronic Imaging, Vol. 2657. International Society for Optics and Photonics, 361–367.Google ScholarCross Ref
58. Okan Tarhan Tursun, Elena Arabadzhiyska-Koleva, Marek Wernikowski, Radosław Mantiuk, Hans-Peter Seidel, Karol Myszkowski, and Piotr Didyk. 2019. Luminance-contrast-aware foveated rendering. ACM Trans. Graphics 38, 4 (2019).Google ScholarDigital Library
59. Veijo Virsu, Risto Näsänen, and Kari Osmoviita. 1987. Cortical magnification and peripheral vision. J. Opt. Soc. Am. A 4, 8 (Aug 1987), 1568–1578.Google ScholarCross Ref
60. David R. Walton, Rafael Kuffner Dos Anjos, Sebastian Friston, David Swapp, Kaan Akşit, Anthony Steed, and Tobias Ritschel. 2021. Beyond blur: real-time ventral metamers for foveated rendering. ACM Trans. Graphics 40, 4 (2021).Google ScholarDigital Library
61. Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Processing 13, 4 (2004), 600–612.Google ScholarDigital Library
62. Andrew B. Watson. 2018. The field of view, the field of resolution, and the field of contrast sensitivity. Electronic Imaging 2018, 14 (2018).Google Scholar
63. Andrew B. Watson and Albert J. Ahumada. 2016. The pyramid of visibility. Electronic Imaging 2016, 16 (2016).Google Scholar
64. T. H. Wertheim. 1894. Uber die indirekte sehscharfe. Zeitschrift fur Psychologie 7 (1894), 172–187.Google Scholar
65. David R. Williams. 1985a. Aliasing in human foveal vision. Vision Res. 25, 2 (1985), 195–205.Google ScholarCross Ref
66. David R. Williams. 1985b. Visibility of interference fringes near the resolution limit. J. Opt. Soc. Am. A 2, 7 (1985), 1087–1093.Google ScholarCross Ref