“Imperceptible manipulation of lateral camera motion for improved virtual reality applications” by Serrano, Martin, Gutierrez, Myszkowski and Masia
Conference:
Type(s):
Title:
- Imperceptible manipulation of lateral camera motion for improved virtual reality applications
Session/Category Title: VR and Real-time Techniques
Presenter(s)/Author(s):
Abstract:
Virtual Reality (VR) systems increase immersion by reproducing users’ movements in the real world. However, several works have shown that this real-to-virtual mapping does not need to be precise in order to convey a realistic experience. Being able to alter this mapping has many potential applications, since achieving an accurate real-to-virtual mapping is not always possible due to limitations in the capture or display hardware, or in the physical space available. In this work, we measure detection thresholds for lateral translation gains of virtual camera motion in response to the corresponding head motion under natural viewing, and in the absence of locomotion, so that virtual camera movement can be either compressed or expanded while these manipulations remain undetected. Finally, we propose three applications for our method, addressing three key problems in VR: improving 6-DoF viewing for captured 360° footage, overcoming physical constraints, and reducing simulator sickness. We have further validated our thresholds and evaluated our applications by means of additional user studies confirming that our manipulations remain imperceptible, and showing that (i) compressing virtual camera motion reduces visible artifacts in 6-DoF, hence improving perceived quality, (ii) virtual expansion allows for completion of virtual tasks within a reduced physical space, and (iii) simulator sickness may be alleviated in simple scenarios when our compression method is applied.
References:
1. Robert Anderson, David Gallup, Jonathan T Barron, Janne Kontkanen, Noah Snavely, Carlos Hernández, Sameer Agarwal, and Steven M Seitz. 2016. Jump: virtual reality video. ACM Trans. Graph. 35, 6 (2016), 198.Google ScholarDigital Library
2. Luke Bölling, Niklas Stein, Frank Steinicke, and Markus Lappe. 2019. Shrinking Circles: Adaptation to Increased Curvature Gain in Redirected Walking. IEEE Trans. on Vis. and Comp. Graph. 25, 5 (2019), 2032–2039.Google ScholarCross Ref
3. Benjamin Bolte and Markus Lappe. 2015. Subliminal reorientation and repositioning in immersive virtual environments using saccadic suppression. IEEE Trans. on Vis. and Comp. Graph. 21, 4 (2015), 545–552.Google ScholarDigital Library
4. Claude Bonnet. 1982. Thresholds of motion perception. In Tut. on Motion Perception.Google Scholar
5. Th Brandt, Johannes Dichgans, and Ellen Koenig. 1973. Differential effects of central versus peripheral vision on egocentric and exocentric motion perception. Exp. brain research 16, 5 (1973), 476–491.Google Scholar
6. Gerd Bruder, Frank Steinicke, Phil Wieland, and Markus Lappe. 2011. Tuning self-motion perception in virtual reality with visual illusions. IEEE Trans. on Vis. and Comp. Graph. 18, 7 (2011), 1068–1078.Google ScholarDigital Library
7. Zekun Cao, Jason Jerald, and Regis Kopper. 2018. Visually-induced motion sickness reduction via static and dynamic rest frames. In Proc. of IEEE VR. 105–112.Google ScholarCross Ref
8. Herbert Aron David. 1963. The method of paired comparisons. Vol. 12. London.Google Scholar
9. Kalpana Dokka, Paul R MacNeilage, Gregory C DeAngelis, and Dora E Angelaki. 2011. Estimating distance during self-motion: A role for visual-vestibular interactions. Journal of Vision 11, 13 (2011), 2–2.Google ScholarCross Ref
10. David Engel, Cristóbal Curio, Lili Tcheang, Betty Mohler, and Heinrich H Bülthoff. 2008. A psychophysically calibrated controller for navigating through large environments in a limited free-walking space. In Proc. of ACM VRST. 157–164.Google ScholarDigital Library
11. Ajoy S Fernandes and Steven K Feiner. 2016. Combating VR sickness through subtle dynamic field-of-view modification. In Proc. of IEEE 3DUI. 201–210.Google ScholarCross Ref
12. David Finlay. 1982. Motion perception in the peripheral visual field. Perception 11, 4 (1982), 457–462.Google ScholarCross Ref
13. John M Foley, Nilton P Ribeiro-Filho, and José A Da Silva. 2004. Visual perception of extent and the geometry of visual space. Vision Research 44, 2 (2004), 147–156.Google ScholarCross Ref
14. Harald Frenz, Markus Lappe, Marina Kolesnik, and Thomas Bührmann. 2007. Estimation of travel distance from visual motion in virtual environments. ACM Trans. on Applied Perception 4, 1 (2007).Google ScholarDigital Library
15. Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. 2019. Digging into self-supervised monocular depth estimation. In Proc. of IEEE ICCV.Google ScholarCross Ref
16. Walter C Gogel. 1982. Analysis of the perception of motion concomitant with a lateral motion of the head. Perception & Psychophysics 32, 3 (1982), 241–250.Google ScholarCross Ref
17. Renato Grasso, Stefan Glasauer, Yasuhiko Takei, and Alain Berthoz. 1996. The predictive brain: anticipatory control of head direction for the steering of locomotion. Neuroreport 7, 6 (1996), 1170–1174.Google ScholarCross Ref
18. Timofey Grechkin, Jerald Thomas, Mahdi Azmandian, Mark Bolas, and Evan Suma. 2016. Revisiting detection thresholds for redirected walking: Combining translation and curvature gains. In Proc. of the ACM Symp. on Applied Perception. 113–120.Google ScholarDigital Library
19. Leah A Groves, Patrick Carnahan, Daniel R Allen, Rankin Adam, Terry M Peters, and Elvis CS Chen. 2019. Accuracy assessment for the co-registration between optical and VIVE head-mounted display tracking. Intern. journal of computer assisted radiology and surgery 14, 7 (2019), 1207–1215.Google Scholar
20. Peter Hedman and Johannes Kopf. 2018. Instant 3D photography. ACM Trans. Graph. 37, 4 (2018), 101.Google ScholarDigital Library
21. Mark A Hollands, Nausica V Ziavra, and Adolfo M Bronstein. 2004. A new paradigm to investigate the roles of head and eye movements in the coordination of whole-body movements. Experimental brain research 154, 2 (2004), 261–266.Google Scholar
22. Ping Hu, Qi Sun, Piotr Didyk, Li-Yi Wei, and Arie E Kaufman. 2019. Reducing simulator sickness with perceptual camera control. ACM Trans. Graph. 38, 6 (2019), 1–12.Google ScholarDigital Library
23. Victoria Interrante, Brian Ries, and Lee Anderson. 2007. Seven league boots: A new metaphor for augmented locomotion through moderately large scale immersive virtual environments. In Proc. of IEEE 3DUI.Google ScholarCross Ref
24. Philip M Jaekl, Robert S Allison, Laurence R Harris, Urszula T Jasiobedzka, Heather L Jenkin, Michael R Jenkin, James E Zacher, and Daniel C Zikovitz. 2002. Perceptual stability during head movement in virtual reality. In Proc. of IEEE VR. 149–155.Google ScholarCross Ref
25. Jason Jerald, Tabitha Peck, Frank Steinicke, and Mary Whitton. 2008. Sensitivity to scene motion for phases of head yaws. In Proc. of the Symp. on Applied Percep. in Graph. and Vis. 155–162.Google ScholarDigital Library
26. Petr Kellnhofer, Piotr Didyk, Tobias Ritschel, Belen Masia, Karol Myszkowski, and Hans-Peter Seidel. 2016. Motion parallax in stereo 3D: Model and applications. ACM Trans. Graph. 35, 6 (2016), 176.Google ScholarDigital Library
27. HyunGoo R Kim, Dora E Angelaki, and Gregory C DeAngelis. 2016. The neural basis of depth perception from motion parallax. Philosophical Trans. of the Royal Society B: Biological Sciences 371, 1697 (2016), 20150256.Google Scholar
28. George Koulieris, Kaan Akşit, Michael Stengel, Rafał K Mantiuk, Katerina Mania, and Christian Richardt. 2019. Near-Eye Display and Tracking Technologies for Virtual and Augmented Reality. In Computer Graphics Forum, Vol. 38. 493–519.Google ScholarCross Ref
29. George Koulieris, Bee Bui, Martin S Banks, and George Drettakis. 2017. Accommodation and comfort in head-mounted displays. ACM Trans. Graph. 36, 4 (2017).Google ScholarDigital Library
30. Oliver Kreylos. 2016. Analysis of Valve’s ‘Lighthouse’ Tracking System Reveals Accuracy. https://www.roadtovr.com/analysis-of-valves-lighthouse-tracking-system-reveals-accuracy/. Last accessed on 2020-04-25.Google Scholar
31. Eike Langbehn, Paul Lubos, Gerd Bruder, and Frank Steinicke. 2017. Bending the curve: Sensitivity to bending of curved paths and application in room-scale VR. IEEE Trans. on Vis. and Comp. Graph. 23, 4 (2017), 1389–1398.Google ScholarDigital Library
32. Eike Langbehn, Frank Steinicke, Markus Lappe, Gregory F Welch, and Gerd Bruder. 2018. In the blink of an eye: leveraging blink-induced suppression for imperceptible position and orientation redirection in virtual reality. ACM Trans. Graph. 37, 4 (2018), 66.Google ScholarDigital Library
33. Markus Lappe, Michael Jenkin, and Laurence R Harris. 2007. Travel distance estimation from visual motion by leaky path integration. Exp. Brain Research 180, 1 (2007).Google Scholar
34. Joseph J. LaViola. 2000. A Discussion of Cybersickness in Virtual Environments. ACM SIGCHI Bulletin 32, 1 (2000), 47–56.Google ScholarDigital Library
35. Zhi Li, John Phillips, and Frank H Durgin. 2011. The underestimation of egocentric distance: Evidence from frontal matching tasks. Attention, Perception, & Psychophysics 73, 7 (2011), 2205.Google ScholarCross Ref
36. Bruce D. Lucas and Takeo Kanade. 1981. An Iterative Image Registration Technique with an Application to Stereo Vision. In Proc. of the Intern. Joint Conf. on Artificial Intelligence (Vancouver, BC, Canada) (IJCAI’81). San Francisco, CA, USA, 674–679.Google Scholar
37. Keigo Matsumoto, Eike Langbehn, Takuji Narumi, and Frank Steinicke. 2020. Detection Thresholds for Vertical Gains in VR and Drone-based Telepresence Systems. In Proc. of IEEE VR. 101–107.Google ScholarCross Ref
38. Mark McGill, Alexander Ng, and Stephen Brewster. 2017. I am the passenger: how visual motion cues can influence sickness for in-car VR. In Proc. of the ACM CHI Conf. on Human Factors in Computing Sys. 5655–5668.Google ScholarDigital Library
39. Shojiro Nagata. 1989. How to reinforce perception of depth in single two-dimensional pictures. (1989).Google Scholar
40. Mark Nawrot, Michael Ratzlaff, Zachary Leonard, and Keith Stroyan. 2014. Modeling depth from motion parallax with the motion/pursuit ratio. Frontiers in Psychology 5 (2014), 1103.Google ScholarCross Ref
41. Mark Nawrot and Keith Stroyan. 2009. The motion/pursuit law for visual depth perception from motion parallax. Vision Research 49, 15 (2009), 1969–1978.Google ScholarCross Ref
42. Christian T Neth, Jan L Souman, David Engel, Uwe Kloos, Heinrich H Bulthoff, and Betty J Mohler. 2012. Velocity-dependent dynamic curvature gain for redirected walking. IEEE Trans. on Vis. and Comp. Graph. 18, 7 (2012), 1041–1052.Google ScholarDigital Library
43. Anh Nguyen, Yannick Rothacher, Bigna Lenggenhager, Peter Brugger, and Andreas Kunz. 2018. Individual differences and impact of gender on curvature redirection thresholds. In Proc. of the ACM Symp. on Applied Perception. 1–4.Google ScholarDigital Library
44. Carlo Nicolini, Carlo Fantoni, Giovanni Mancuso, Robert Volcic, and Fulvio Domini. 2014. A framework for the study of vision in active observers. In Human Vision and Electronic Imaging XIX, Vol. 9014. Intern. Society for Optics and Photonics, 901414.Google Scholar
45. Niels Christian Nilsson, Tabitha Peck, Gerd Bruder, Eri Hodgson, Stefania Serafin, Mary Whitton, Frank Steinicke, and Evan Suma Rosenberg. 2018. 15 years of research on redirected walking in immersive virtual environments. IEEE Computer Graph. and App. 38, 2 (2018), 44–56.Google ScholarDigital Library
46. Norbert Nitzsche, Uwe D Hanebeck, and Günther Schmidt. 2003. Motion compression for telepresent walking in large-scale remote environments. In Helmet-and Head-Mounted Displays VIII: Technologies and Applications, Vol. 5079. Intern. Society for Optics and Photonics, 265–276.Google ScholarCross Ref
47. Hiroshi Ono and Hiroyasu Ujike. 2005. Motion parallax driven by head movements: Conditions for visual stability, perceived depth, and perceived concomitant motion. Perception 34, 4 (2005), 477–490.Google ScholarCross Ref
48. Ryan S. Overbeck, Daniel Erickson, Daniel Evangelakos, Matt Pharr, and Paul Debevec. 2018. A System for Acquiring, Processing, and Rendering Panoramic Light Field Stills for Virtual Reality. ACM Trans. Graph. 37, 6, Article 197 (2018), 15 pages.Google ScholarDigital Library
49. Francesco Panerai, Valerie Cornilleau-Pérès, and Jacques Droulez. 2002. Contribution of extraretinal signals to the scaling of object distance during self-motion. Perception & Psychophysics 64, 5 (2002), 717–731.Google ScholarCross Ref
50. Anjul Patney, Marina Zannoli, Joohwan Kim, Robert Konrad, Frank Steinicke, and Martin S. Banks. 2018. Applications of Vision Science to Virtual and Augmented Reality. In ACM SIGGRAPH Courses (Vancouver, British Columbia, Canada) (SIGGRAPH ’18). Article 8, 28 pages.Google Scholar
51. Tabitha C Peck, Henry Fuchs, and Mary C Whitton. 2011. An evaluation of navigational ability comparing Redirected Free Exploration with Distractors to Walking-in-Place and joystick locomotio interfaces. In Proc. of IEEE VR. 55–62.Google ScholarDigital Library
52. T. C. Peck, L. E. Sockol, and S. M. Hancock. 2020. Mind the Gap: The Underrepresentation of Female Participants and Authors in Virtual Reality Research. IEEE Trans. on Vis. and Comp. Graph. 26, 5 (2020), 1945–1954.Google ScholarCross Ref
53. Albert Parra Pozo, Michael Toksvig, Terry Filiba Schrager, Joyce Hsu, Uday Mathur, Alexander Sorkine-Hornung, Rick Szeliski, and Brian Cabral. 2019. An integrated 6DoF video camera and system design. ACM Trans. Graph. 38, 6 (2019), 1–16.Google ScholarDigital Library
54. Eric D Ragan, Siroberto Scerbo, Felipe Bacim, and Doug A Bowman. 2016. Amplified head rotation in virtual reality and the effects on 3d search, training transfer, and spatial orientation. IEEE Trans. on Vis. and Comp. Graph. 23, 8 (2016), 1880–1895.Google ScholarDigital Library
55. Yashas Rai, Jesús Gutiérrez, and Patrick Le Callet. 2017a. A dataset of head and eye movements for 360 degree images. In Proc. of the ACM Multim. Sys. Conf. 205–210.Google ScholarDigital Library
56. Yashas Rai, Patrick Le Callet, and Philippe Guillotel. 2017b. Which saliency weighting for omni directional image quality assessment?. In Proc. of IEEE QoMEX. 1–6.Google ScholarCross Ref
57. Sharif Razzaque, David Swapp, Mel Slater, Mary C. Whitton, and Anthony Steed. 2002. Redirected Walking in Place. In Proc. of the Workshop on Virtual Envs. 2002 (Barcelona, Spain) (EGVE ’02). Eurographics Association, 123–130.Google ScholarDigital Library
58. Rebekka Renner, Boris Velichkovsky, and Jens R Helmert. 2013. The perception of egocentric distances in virtual environments-a review. Comput. Surveys 46, 2 (2013).Google Scholar
59. Christian Richardt, Peter Hedman, Ryan S Overbeck, Brian Cabral, Robert Konrad, and Steve Sullivan. 2019. Capture4VR: From VR Photography to VR Video. In ACM SIGGRAPH Courses. https://richardt.name/Capture4VR/Google Scholar
60. Michael Rubinstein, Diego Gutierrez, Olga Sorkine, and Ariel Shamir. 2010. A comparative study of image retargeting. In ACM Trans. Graph., Vol. 29. 160.Google ScholarDigital Library
61. Shyam Prathish Sargunam, Kasra Rahimi Moghadam, Mohamed Suhail, and Eric D Ragan. 2017. Guided head rotation and amplified head rotation: Evaluating seminatural travel and viewing techniques in virtual reality. In Proc. of IEEE VR. 19–28.Google ScholarCross Ref
62. Peter Scarfe and Andrew Glennerster. 2015. Using high-fidelity virtual reality to study perception in freely moving observers. Journal of Vision 15, 9 (2015), 3–3.Google ScholarCross Ref
63. Patric Schmitz, Julian Hildebrandt, André Calero Valdez, Leif Kobbelt, and Martina Ziefle. 2018. You spin my head right round: Threshold of limited immersion for rotation gains in redirected walking. IEEE Trans. on Vis. and Comp. Graph. 24, 4 (2018), 1623–1632.Google ScholarDigital Library
64. Ana Serrano, Incheol Kim, Zhili Chen, Stephen DiVerdi, Diego Gutierrez, Aaron Hertzmann, and Belen Masia. 2019. Motion parallax for 360° RGBD video. IEEE Trans. on Vis. and Comp. Graph. 25, 5 (2019).Google ScholarCross Ref
65. Iwan Setyawan and Reginald L Lagendijk. 2004. Human perception of geometric distortions in images. In Security, Steganography, and Watermarking of Multim. Contents VI, Vol. 5306. Intern. Society for Optics and Photonics, 256–267.Google Scholar
66. Takashi Shibata, Joohwan Kim, David M Hoffman, and Martin S Banks. 2011. The zone of comfort: Predicting visual discomfort with stereo displays. Journal of vision 11, 8 (2011), 11–11.Google ScholarCross Ref
67. Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, and Gordon Wetzstein. 2018. Saliency in VR: How do people explore virtual environments? IEEE Trans. on Vis. and Comp. Graph. 36, 4 (2018).Google Scholar
68. Mel Slater. 2009. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Trans. of the Royal Society B: Biological Sciences 364, 1535 (2009), 3549–3557.Google ScholarCross Ref
69. Martin J Steinbach, Hiroshi Ono, and Marla E Wolf. 1991. Motion parallax judgements of depth as a function of the direction and type of head movement. Canadian Journal of Psychology / Revue Canadienne de Psychologie 45, 1 (1991), 92.Google ScholarCross Ref
70. Frank Steinicke, Gerd Bruder, Jason Jerald, Harald Frenz, and Markus Lappe. 2010. Estimation of detection thresholds for redirected walking techniques. IEEE Trans. on Vis. and Comp. Graph. 16, 1 (2010), 17–27.Google ScholarDigital Library
71. Evan A Suma, Seth Clark, David Krum, Samantha Finkelstein, Mark Bolas, and Zachary Warte. 2011. Leveraging change blindness for redirection in virtual environments. In Proc. of IEEE VR. 159–166.Google ScholarCross Ref
72. Evan A Suma, Zachary Lipps, Samantha Finkelstein, David M Krum, and Mark Bolas. 2012. Impossible spaces: Maximizing natural walking in virtual environments with self-overlapping architecture. IEEE Trans. on Vis. and Comp. Graph. 18, 4 (2012).Google ScholarDigital Library
73. Qi Sun, Anjul Patney, Li-Yi Wei, Omer Shapira, Jingwan Lu, Paul Asente, Suwen Zhu, Morgan Mcguire, David Luebke, and Arie Kaufman. 2018. Towards virtual reality infinite walking: dynamic saccadic redirection. ACM Trans. Graph. 37, 4 (2018), 67.Google ScholarDigital Library
74. Qi Sun, Li-Yi Wei, and Arie Kaufman. 2016. Mapping virtual and physical reality. ACM Trans. Graph. 35, 4 (2016), 64.Google ScholarDigital Library
75. Michael T Swanston and Nicholas J Wade. 1988. The perception of visual motion during movements of the eyes and of the head. Perception & Psychophysics 43, 6 (1988).Google Scholar
76. Sebastian Sylwan, Michael Interbartolo, Liz Warren, and Felix Lajeunesse. 2019. “Space Explorers: Life in Orbit”: Filming VR in Microgravity. In ACM SIGGRAPH Production Sessions (Los Angeles, California) (SIGGRAPH ’19). Article 6, 1 pages.Google Scholar
77. Jayant Thatte, Jean-Baptiste Boin, Haricharan Lakshman, and Bernd Girod. 2016. Depth augmented stereo panorama for cinematic virtual reality with head-motion parallax. In Proc. of IEEE ICME. 1–6.Google ScholarCross Ref
78. Jayant Thatte and Bernd Girod. 2018. Towards perceptual evaluation of six degrees of freedom virtual reality rendering from stacked omnistereo representation. Electronic Imaging 2018, 5 (2018), 352–1.Google ScholarCross Ref
79. Hiroyasu Ujike and Hiroshi Ono. 2001. Depth thresholds of motion parallax as a function of head movement velocity. Vision Research 41, 22 (2001), 2835–2843.Google ScholarCross Ref
80. Mark Wexler, Francesco Panerai, Ivan Lamouret, and Jacques Droulez. 2001. Self-motion and the perception of stationary objects. Nature 409, 6816 (2001), 85–88.Google Scholar
81. Mark Wexler and Jeroen JA Van Boxtel. 2005. Depth perception by the active observer. Trends in Cognitive Sciences 9, 9 (2005), 431–438.Google ScholarCross Ref
82. Niall L Williams and Tabitha C Peck. 2019. Estimation of rotation gain thresholds considering fov, gender, and distractors. IEEE Trans. on Vis. and Comp. Graph. 25, 11 (2019), 3158–3168.Google ScholarCross Ref
83. Betsy Williams-Sanders, Tom Carr, Gayathri Narasimham, Tim McNamara, John Rieser, and Bobby Bodenheimer. 2019. Scaling Gain and Eyeheight While Locomoting in a Large VE. In Intern. Conf. on Human-Computer Interaction. 277–298.Google ScholarCross Ref
84. Ahmad Yoonessi and Curtis L Baker. 2011. Contribution of motion parallax to segmentation and depth perception. Journal of Vision 11, 9 (2011), 13–13.Google ScholarCross Ref
85. Jingxin Zhang, Eike Langbehn, Dennis Krupke, Nicholas Katzakis, and Frank Steinicke. 2018. Detection Thresholds for Rotation and Translation Gains in 360° Video-Based Telepresence Systems. IEEE Trans. on Vis. and Comp. Graph. 24, 4 (2018), 1671–1680.Google ScholarDigital Library
86. Ruimin Zhang and Scott A Kuhl. 2013. Human sensitivity to dynamic rotation gains in head-mounted displays. In Proc. of the ACM Symp. on Applied Perception. 71–74.Google ScholarDigital Library
87. Ruimin Zhang, Bochao Li, and Scott A Kuhl. 2014. Human sensitivity to dynamic translational gains in head-mounted displays. In Proc. of the ACM SUI. 62–65.Google ScholarDigital Library
