“Perceptual rasterization for head-mounted display image synthesis” by Friston, Ritschel and Steed
Conference:
Type:
Title:
- Perceptual rasterization for head-mounted display image synthesis
Session/Category Title: VR and AR
Presenter(s)/Author(s):
Abstract:
We suggest a rasterization pipeline tailored towards the needs of HMDs, where latency and field-of-view requirements pose new challenges beyond those of traditional desktop displays. Instead of image warping for low latency, or using multiple passes for foveation, we show how both can be produced directly in a single perceptual rasterization pass. We do this with per-fragment ray-casting. This is enabled by derivations of tight space-time-fovea pixel bounds, introducing just enough flexibility for the requisite geometric tests, but retaining most of the simplicity and efficiency of the traditional rasterizaton pipeline. To produce foveated images, we rasterize to an image with spatially varying pixel density. To compensate for latency, we extend the image formation model to directly produce “rolling” images where the time at each pixel depends on its display location. Our approach overcomes limitations of warping with respect to disocclusions, object motion and view-dependent shading, as well as geometric aliasing artifacts in other foveated rendering techniques. A set of perceptual user studies demonstrates the efficacy of our approach.
References:
1. Tomas Akenine-Möller, Jacob Munkberg, and Jon Hasselgren. 2007. Stochastic rasterization using time-continuous triangles. In Proc. Graphics Hardware. 9. Google ScholarDigital Library
2. Tomas Akenine-Möller, Robert Toth, Jacob Munkberg, and Jon Hasselgren. 2012. Efficient Depth of Field Rasterization Using a Tile Test Based on Half-Space Culling. Comp. Graph. Forum 31, 1 (2012). Google ScholarDigital Library
3. Michael Antonov. 2015. https://developer3.oculus.com/blog/asynchronous-timewarp-examined/.Google Scholar
4. Stefan Brabec, Thomas Annen, and Hans-Peter Seidel. 2002. Shadow mapping for hemispherical and omnidirectional light sources. Advances in Modelling, Animation and Rendering (2002), 397–408.Google Scholar
5. John Brosz, Faramarz F Samavati, M Sheelagh T Carpendale, and Mario Costa Sousa. 2007. Single camera flexible projection. In Proc. NPAR. 33–42. Google ScholarDigital Library
6. J. S. Brunhaver, K. Fatahalian, and P. Hanrahan. 2010. Hardware Implementation of Micropolygon Rasterization with Motion and Defocus Blur. In Proc. HPG. 1–9. Google ScholarDigital Library
7. Timothy J. Buker, Dennis A. Vincenzi, and John E. Deaton. 2012. The Effect of Apparent Latency on Simulator Sickness While Using a See-Through Helmet-Mounted Display: Reducing Apparent Latency With Predictive Compensation. Human Factors 54, 2 (2012), 235–249.Google ScholarCross Ref
8. Shenchang Eric Chen and Lance Williams. 1993. View interpolation for image synthesis. In Proc. SIGGRAPH. 279–88. Google ScholarDigital Library
9. PM Daniel and D Whitteridge. 1961. The representation of the visual field on the cerebral cortex in monkeys. J Physiology 159, 2 (1961), 203–21.Google ScholarCross Ref
10. Piotr Didyk, Elmar Eisemann, Tobias Ritschel, Karol Myszkowski, and Hans-Peter Seidel. 2010. Perceptually-motivated Real-time Temporal Upsampling of 3D Content for High-refresh-rate Displays. Comp. Graph. Forum 29, 2 (2010), 713–22.Google ScholarCross Ref
11. S.R. Ellis, B.D. Adelstein, S. Baumeler, G.J. Jense, and R.H. Jacoby. 1999. Sensor spatial distortion, visual latency, and update rate effects on 3D tracking in virtual environments. In Proc. VR. 218–21. Google ScholarDigital Library
12. Kayvon Fatahalian, Edward Luong, Solomon Boulos, Kurt Akeley, William R Mark, and Pat Hanrahan. 2009. Data-parallel rasterization of micropolygons with defocus and motion blur. In Proc. HPG. 59–68. Google ScholarDigital Library
13. FOVE. 2018. https://www.getfove.com/.Google Scholar
14. Sebastian Friston, Anthony Steed, Simon Tilbury, and Georgi Gaydadjiev. 2016. Construction and Evaluation of an Ultra Low Latency Frameless Renderer for VR. IEEE TVCG 22, 4 (2016), 1377–86. Google ScholarDigital Library
15. George W Furnas. 1986. Generalized fisheye views. In Proc. CHI. Google ScholarDigital Library
16. Jean-Dominique Gascuel, Nicolas Holzschuch, Gabriel Fournier, and Bernard Peroche. 2008. Fast non-linear projections using graphics hardware. In Proc. i3D. 107–14. Google ScholarDigital Library
17. Ragnar Granit and Phyllis Harper. 1930. Comparative studies on the peripheral and central retina. J Physiology 95, 1 (1930), 211–28.Google Scholar
18. Gerard E Grossman, R John Leigh, LA Abel, Douglas J Lanska, and SE Thurston. 1988. Frequency and velocity of rotational head perturbations during locomotion. Exp. Brain Res. 70, 3 (1988), 470–6.Google ScholarCross Ref
19. Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Trans. Graph. (Proc. SIGGRAPH) 31, 6 (2012), 164. Google ScholarDigital Library
20. Yong He, Yan Gu, and Kayvon Fatahalian. 2014. Extending the graphics pipeline with adaptive, multi-rate shading. ACM Trans. Graph (Proc. SIGGRAPH) 33, 4 (2014), 142. Google ScholarDigital Library
21. Samuli Laine, Timo Aila, Tero Karras, and Jaakko Lehtinen. 2011. Clipless dual-space bounds for faster stochastic rasterization. ACM Trans. Graph (Proc. SIGGRAPH) 30, 4 (2011), 106. Google ScholarDigital Library
22. Baoquan Liu, Li-Yi Wei, Xu Yang, Chongyang Ma, Ying-Qing Xu, Baining Guo, and Enhua Wu. 2011. Non-Linear Beam Tracing on a GPU. Comp. Graph. Forum 30, 8 (2011), 2156–69.Google ScholarCross Ref
23. William R Mark, Leonard McMillan, and Gary Bishop. 1997. Post-rendering 3D warping. In Proc. i3D. 7–ff. Google ScholarDigital Library
24. Morgan McGuire, Eric Enderton, Peter Shirley, and David Luebke. 2010. Real-time stochastic rasterization on conventional GPU architectures. In Proc. HPG. 173–82. Google ScholarDigital Library
25. Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney. 2018. Kernel Foveated Rendering. Proc. ACM Comput. Graph. Interact. Tech. 1, 1 (2018), 5:1–5:20. Google ScholarDigital Library
26. Nvidia. 2017. NV_clip_space_w_scaling OpenGL extension.Google Scholar
27. Oculus. 2018. https://developer.oculus.com/documentation/mobilesdk/0.4/concepts/mobile-timewarp-overview/.Google Scholar
28. Oculus VR. 2017. Asynchronous TimeWarp.Google Scholar
29. Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Trans. Graph. (Proc. SIGGRAPH) 35, 6 (2016), 179. Google ScholarDigital Library
30. Juan Pineda. 1988. A parallel algorithm for polygon rasterization. ACM SIGGRAPH Computer Graphics 22, 4 (1988), 17–20. Google ScholarDigital Library
31. Voicu Popescu, Paul Rosen, and Nicoletta Adamo-Villani. 2009. The graph camera. ACM Trans. Graph. 28, 5 (2009), 158. Google ScholarDigital Library
32. Matthew Regan and Ronald Pose. 1994. Priority rendering with a virtual reality address recalculation pipeline. In Proc. SIGGRAPH. 155–162. Google ScholarDigital Library
33. Bernhard Reinert, Johannes Kopf, Tobias Ritschel, Eduardo Cuervo, David Chu, and Hans-Peter Seidel. 2016. Proxy-guided Image-based Rendering for Mobile Devices. Comp. Graph. Forum (Proc. Pacific Graphics) 35, 7 (2016). Google ScholarDigital Library
34. Andre Schollmeyer, Simon Schneegans, Stephan Beck, Anthony Steed, and Bernd Froehlich. 2017. Efficient Hybrid Image Warping for High Frame-Rate Stereoscopic Rendering. IEEE Trans. Vis. and Comp. Graph. 23, 4 (2017), 1332–41. Google ScholarDigital Library
35. Mel Slater. 2002. Presence and The Sixth Sense. Presence 11, 4 (2002), 435–439. Google ScholarDigital Library
36. AAS Sluyterman. 2006. What is needed in LCD panels to achieve CRT-like motion portrayal? J SID 14, 8 (2006), 681–686.Google Scholar
37. Michael Stengel, Steve Grogorick, Martin Eisemann, Elmar Eisemann, and Marcus A Magnor. 2015. An affordable solution for binocular eye tracking and calibration in head-mounted displays. In Proc. ACM Multimedia. 15–24. Google ScholarDigital Library
38. Michael Stengel, Steve Grogorick, Martin Eisemann, and Marcus Magnor. 2016. Adaptive Image-Space Sampling for Gaze-Contingent Real-time Rendering. Comp. Graph. Forum 35, 4 (2016), 129–39. Google ScholarDigital Library
39. Qi Sun, Fu-Chung Huang, Joohwan Kim, Li-Yi Wei, David Luebke, and Arie Kaufman. 2017. Perceptually-guided foveation for light field displays. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 36, 6 (2017), 192. Google ScholarDigital Library
40. Robert Toth, Jim Nilsson, and Tomas Akenine-Möller. 2016. Comparison of projection methods for rendering virtual reality. In Proc. HPG. 163–71. Google ScholarDigital Library
41. Ingo Wald, Sven Woop, Carsten Benthin, Gregory S Johnson, and Manfred Ernst. 2014. Embree: a kernel framework for efficient CPU ray tracing. ACM Trans. Graph (Proc. SIGGRAPH) 33, 4 (2014), 143. Google ScholarDigital Library
42. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Proc. 13, 4 (2004), 600–12. Google ScholarDigital Library
43. Martin Weier, Thorsten Roth, Ernst Kruijff, André Hinkenjann, Arsène Pérard-Gayot, Philipp Slusallek, and Yongmin Li. 2016. Foveated Real-Time Ray Tracing for Head-Mounted Displays. Comp. Graph. Forum 35, 7 (2016), 89–298. Google ScholarDigital Library
44. Martin Weier, Michael Stengel, Thorsten Roth, Piotr Didyk, Elmar Eisemann, Martin Eisemann, Steve Grogorick, André Hinkenjann, Ernst Kruijff, Marcus Magnor, et al. 2017. Perception-driven Accelerated Rendering. Comp. Graph. Forum 36, 2 (2017), 611–43. Google ScholarDigital Library
45. P.H. Wicksteed and F.M. Cornford. 1929. Aristotle. Physics. W. Heinemann.Google Scholar
46. Lei Yang, Yu-Chiu Tse, Pedro V Sander, Jason Lawrence, Diego Nehab, Hugues Hoppe, and Clara L Wilkins. 2011. Image-based bidirectional scene reprojection. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 30, 6 (2011), 150. Google ScholarDigital Library
