“Headset Removal for Virtual and Mixed Reality” by Frueh, Sud and Kwatra
Conference:
Type:
Entry Number: 80
Title:
- Headset Removal for Virtual and Mixed Reality
Presenter(s)/Author(s):
Abstract:
Virtual Reality (VR) has advanced significantly in recent years and allows users to explore novel environments (both real and imaginary), play games, and engage with media in a way that is unprecedentedly immersive. However, compared to physical reality, sharing these experiences is difficult because the user’s virtual environment is not easily observable from the outside and the user’s face is partly occluded by the VR headset. Mixed Reality (MR) is a medium that alleviates some of this disconnect by sharing the virtual context of a VR user in a flat video format that can be consumed by an audience to get a feel for the user’s experience.
Even though MR allows audiences to connect actions of the VR user with their virtual environment, empathizing with them is difficult because their face is hidden by the headset. We present a solution to address this problem by virtually removing the headset and revealing the face underneath it using a combination of 3D vision, machine learning and graphics techniques. We have integrated our headset removal approach with Mixed Reality, and demonstrate results on several VR games and experiences.
References:
Xavier P. Burgos-Artizzu, Julien Fleureau, Olivier Dumas, Thierry Tapie, François LeClerc, and Nicolas Mollet. 2015. Real-time Expression-sensitive HMD Face Reconstruction. In SIGGRAPH Asia 2015 Technical Briefs. ACM. Google ScholarDigital Library
Kert Gartner. 2016. Making High Quality Mixed Reality VR Trailers and Videos. (2016). http://www.kertgartner.com/making-mixed-reality-vr-trailers-and-videosGoogle Scholar
Vivek Kwatra, Christian Frueh, and Avneesh Sud. 2017. Headset “Removal” for Virtual and Mixed Reality. (2017). https://research.googleblog.com/2017/02/headset-removal-for-virtual-and-mixed.htmlGoogle Scholar
Justus Thies, Michael Zollöfer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2016. FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality. arXiv preprint arXiv:1610.03151 (2016).
Keyword(s):
Acknowledgements:
We thank our collaborators in Daydream Labs, Google Tilt Brush,
YouTube Spaces, and in particular, Hayes Rae, Tom Small, Chris
Bregler and Sergey Ioe for their insightful suggestions and support.
