“Mapping virtual and physical reality”

  • ©Qi Sun, Li-Yi Wei, and Arie A. Kaufman




    Mapping virtual and physical reality


Session Title: CAMERA CONTROL & VR



    Real walking offers higher immersive presence for virtual reality (VR) applications than alternative locomotive means such as walking-in-place and external control gadgets, but needs to take into consideration different room sizes, wall shapes, and surrounding objects in the virtual and real worlds. Despite perceptual study of impossible spaces and redirected walking, there are no general methods to match a given pair of virtual and real scenes.We propose a system to match a given pair of virtual and physical worlds for immersive VR navigation. We first compute a planar map between the virtual and physical floor plans that minimizes angular and distal distortions while conforming to the virtual environment goals and physical environment constraints. Our key idea is to design maps that are globally surjective to allow proper folding of large virtual scenes into smaller real scenes but locally injective to avoid locomotion ambiguity and intersecting virtual objects. From these maps we derive altered rendering to guide user navigation within the physical environment while retaining visual fidelity to the virtual environment. Our key idea is to properly warp the virtual world appearance into real world geometry with sufficient quality and performance. We evaluate our method through a formative user study, and demonstrate applications in gaming, architecture walkthrough, and medical imaging.


    1. Bonnans, J. F., Gilbert, J. C., Lemaréchal, C., and Sagastizábal, C. A. 2006. Numerical Optimization: Theoretical and Practical Aspects (Universitext). Springer-Verlag. Google ScholarDigital Library
    2. Bouchard, S., Robillard, G., and Renaud, P. 2007. Revising the factor structure of the simulator sickness questionnaire. Annual Review of CyberTherapy and Telemedicine 5, 128–137.Google Scholar
    3. Bowman, D. A., Gabbard, J. L., and Hix, D. 2002. A survey of usability evaluation in virtual environments: Classification and comparison of methods. Presence: Teleoper. Virtual Environ. 11, 4 (Aug.), 404–424. Google ScholarDigital Library
    4. Bruder, G., Lubas, P., and Steinicke, F. 2015. Cognitive resource demands of redirected walking. IEEE Transactions on Visualization and Computer Graphics 21, 4 (April), 539–544.Google ScholarDigital Library
    5. Chen, R., and Weber, O. 2015. Bounded distortion harmonic mappings in the plane. ACM Trans. Graph. 34, 4 (July), 73:1–73:12. Google ScholarDigital Library
    6. Cheng, L.-P., Roumen, T., Rantzsch, H., Köhler, S., Schmidt, P., Kovacs, R., Jasper, J., Kemper, J., and Baudisch, P. 2015. Turkdeck: Physical virtual reality based on people. In UIST ’15, 417–426. Google ScholarDigital Library
    7. Choi, S., Zhou, Q.-Y., and Koltun, V. 2015. Robust reconstruction of indoor scenes. In CVPR ’15, 5556–5565.Google Scholar
    8. Crassin, C., McGuire, M., Fatahalian, K., and Lefohn, A. 2015. Aggregate G-buffer anti-aliasing. In I3D ’15, 109–119. Google ScholarDigital Library
    9. Cui, J., Rosen, P., Popescu, V., and Hoffmann, C. 2010. A curved ray camera for handling occlusions through continuous multiperspective visualization. IEEE Transactions on Visualization and Computer Graphics 16, 6 (Nov.), 1235–1242. Google ScholarDigital Library
    10. Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In SIGGRAPH ’96, 11–20. Google ScholarDigital Library
    11. Febretti, A., Nishimoto, A., Mateevitsi, V., Renambot, L., Johnson, A., and Leigh, J. 2014. Omegalib: A multi-view application framework for hybrid reality display environments. In Virtual Reality (VR), 2014 IEEE, 9–14.Google Scholar
    12. Fu, X.-M., Liu, Y., and Guo, B. 2015. Computing locally injective mappings by advanced mips. ACM Trans. Graph. 34, 4 (July), 71:1–71:12. Google ScholarDigital Library
    13. Gal, R., Sorkine, O., and Cohen-Or, D. 2006. Feature-aware texturing. In EGSR ’06, 297–303. Google ScholarDigital Library
    14. Gray, A. 1996. Modern Differential Geometry of Curves and Surfaces with Mathematica, 1st ed. CRC Press, Inc. Google ScholarDigital Library
    15. Hodgson, E., Bachmann, E., and Waller, D. 2008. Redirected walking to explore virtual environments: Assessing the potential for spatial interference. ACM Trans. Appl. Percept. 8, 4 (Dec.), 22:1–22:22. Google ScholarDigital Library
    16. Hong, L., Muraki, S., Kaufman, A., Bartz, D., and He, T. 1997. Virtual voyage: Interactive navigation in the human colon. In SIGGRAPH ’97, 27–34. Google ScholarDigital Library
    17. Huang, F.-C., Chen, K., and Wetzstein, G. 2015. The light field stereoscope: Immersive computer graphics via factored near-eye light field displays with focus cues. ACM Trans. Graph. 34, 4 (July), 60:1–60:12. Google ScholarDigital Library
    18. Iwata, H., Yano, H., and Tomioka, H. 2006. Powered shoes. In SIGGRAPH ’06 Emerging Technologies. Google ScholarDigital Library
    19. Jang, Y., Noh, S.-T., Chang, H. J., Kim, T.-K., and Woo, W. 2015. 3d finger cape: Clicking action and position estimation under self-occlusions in egocentric viewpoint. IEEE Transactions on Visualization and Computer Graphics 21, 4 (April), 501–510.Google ScholarDigital Library
    20. Kennedy, R. S., Lane, N. E., Berbaum, K. S., and Lilienthal, M. G. 1993. Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The International Journal of Aviation Psychology 3, 3, 203–220.Google ScholarCross Ref
    21. Lévy, B., Petitjean, S., Ray, N., and Maillot, J. 2002. Least squares conformal maps for automatic texture atlas generation. ACM Trans. Graph. 21, 3 (July), 362–371. Google ScholarDigital Library
    22. Li, H., Trutoiu, L., Olszewski, K., Wei, L., Trutna, T., Hsieh, P.-L., Nicholls, A., and Ma, C. 2015. Facial performance sensing head-mounted display. ACM Trans. Graph. 34, 4 (July), 47:1–47:9. Google ScholarDigital Library
    23. Maesen, S., Goorts, P., and Bekaert, P. 2013. Scalable optical tracking for navigating large virtual environments using spatially encoded markers. In VRST ’13, 101–110. Google ScholarDigital Library
    24. McMillan, Jr., L. 1997. An Image-based Approach to Three-dimensional Computer Graphics. PhD thesis. UMI Order No. GAX97-30561. Google ScholarDigital Library
    25. Nescher, T., Huang, Y.-Y., and Kunz, A. 2014. Planning redirection techniques for optimal free walking experience using model predictive control. In 3DUI ’14, 111–118.Google Scholar
    26. Nilsson, N., Serafin, S., and Nordahl, R. 2014. Establishing the range of perceptually natural visual walking speeds for virtual walking-in-place locomotion. IEEE Transactions on Visualization and Computer Graphics 20, 4 (April), 569–578. Google ScholarDigital Library
    27. Popescu, V., Rosen, P., and Adamo-Villani, N. 2009. The graph camera. ACM Trans. Graph. 28, 5 (Dec.), 158:1–158:8. Google ScholarDigital Library
    28. Poranne, R., and Lipman, Y. 2014. Provably good planar mappings. ACM Trans. Graph. 33, 4, 76:1–76:11. Google ScholarDigital Library
    29. Razzaque, S., Kohn, Z., and Whitton, M. C. 2001. Redirected Walking. In Eurographics 2001 – Short Presentations, Eurographics Association.Google Scholar
    30. Razzaque, S., Swapp, D., Slater, M., Whitton, M. C., and Steed, A. 2002. Redirected walking in place. In EGVE ’02, 123–130. Google ScholarDigital Library
    31. Rudin, W. 1976. Principles of mathematical analysis, third ed. McGraw-Hill Book Co., New York. International Series in Pure and Applied Mathematics.Google Scholar
    32. Schild, J., LaViola, J., and Masuch, M. 2012. Understanding user experience in stereoscopic 3d games. In CHI ’12, 89–98. Google ScholarDigital Library
    33. Schüller, C., Kavan, L., Panozzo, D., and Sorkine-Hornung, O. 2013. Locally injective mappings. In SGP ’13, 125–135. Google ScholarDigital Library
    34. Schwaiger, M., Thümmel, T., and Ulbrich, H. 2007. Cyberwalk: Implementation of a ball bearing platform for humans. In Human-Computer Interaction. Interaction Platforms and Techniques. Springer, 926–935. Google ScholarDigital Library
    35. Simeone, A. L., Velloso, E., and Gellersen, H. 2015. Substitutional reality: Using the physical environment to design virtual reality experiences. In CHI ’15, 3307–3316. Google ScholarDigital Library
    36. Souman, J. L., Giordano, P. R., Schwaiger, M., Frissen, I., Thümmel, T., Ulbrich, H., Luca, A. D., Bülthoff, H. H., and Ernst, M. O. 2008. Cyberwalk: Enabling unconstrained omnidirectional walking through virtual environments. ACM Trans. Appl. Percept. 8, 4 (Dec.), 25:1–25:22. Google ScholarDigital Library
    37. Steinicke, F., Bruder, G., Jerald, J., Frenz, H., and Lappe, M. 2008. Analyses of human sensitivity to redirected walking. In VRST ’08, 149–156. Google ScholarDigital Library
    38. Suma, E., Lipps, Z., Finkelstein, S., Krum, D., and Bolas, M. 2012. Impossible spaces: Maximizing natural walking in virtual environments with self-overlapping architecture. IEEE Transactions on Visualization and Computer Graphics 18, 4 (April), 555–564. Google ScholarDigital Library
    39. Turk, G., and O’Brien, J. F. 2005. Shape transformation using variational implicit functions. In SIGGRAPH ’05 Courses. Google ScholarDigital Library
    40. Usoh, M., Arthur, K., Whitton, M. C., Bastos, R., Steed, A., Slater, M., and Brooks, Jr., F. P. 1999. Walking > walking-in-place > flying, in virtual environments. In SIGGRAPH ’99, 359–364. Google ScholarDigital Library
    41. Vasylevska, K., Kaufmann, H., Bolas, M., and Suma, E. 2013. Flexible spaces: Dynamic layout generation for infinite walking in virtual environments. In 3DUI ’13, 39–42.Google Scholar
    42. Witmer, B. G., and Singer, M. J. 1998. Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoper. Virtual Environ. 7, 3 (June), 225–240. Google ScholarDigital Library
    43. Wong, R., 2015. Vr startups: Stop trying to make virtual reality treadmills a thing. http://mashable.com/2015/06/20/virtual-reality-treadmills/.Google Scholar
    44. Yang, L., Tse, Y.-C., Sander, P. V., Lawrence, J., Nehab, D., Hoppe, H., and Wilkins, C. L. 2011. Image-based bidirectional scene reprojection. ACM Trans. Graph. 30, 6 (Dec.), 150:1–150:10. Google ScholarDigital Library
    45. Zhang, R., and Kuhl, S. A. 2013. Human sensitivity to dynamic rotation gains in head-mounted displays. In SAP ’13, 71–74. Google ScholarDigital Library
    46. Zmuda, M., Wonser, J., Bachmann, E., and Hodgson, E. 2013. Optimizing constrained-environment redirected walking instructions using search techniques. IEEE Transactions on Visualization and Computer Graphics 19, 11 (Nov), 1872–1884. Google ScholarDigital Library

ACM Digital Library Publication: