“AR-ia: Volumetric Opera for Mobile Augmented Reality” by Kelly, Cordingley, Nolan, Rhemann, Fanello, et al. …
Conference:
Experience Type(s):
Title:
- AR-ia: Volumetric Opera for Mobile Augmented Reality
Presenter(s):
- Sean Kelly
- Samantha Cordingley
- Patrick Nolan
- Christoph Rhemann
- Sean Ryan Fanello
- Danhang Tang
- Jude Osborn
- Jay Busch
- Philip L. Davidson
- Paul E. Debevec
- Peter Denny
- Graham Fyffe
- Kaiwen Guo
- Geoff Harvey
- Shahram Izadi
- Peter Lincoln
- Wan-Chun Alex Ma
- Jonathan Taylor
- Xueming Yu
- Matt Whalen
- Jason Dourgarian
- Genevieve Blanchett
- Narelle French
- Kirstin Sillitoe
- Tea Uglow
- Brenton Spiteri
- Emma Pearson
- Wade Kernot
- Jonathan Richards
Description:
Motivated by recent availability of augmented and virtual reality platforms, Project AR-ia tackles the challenging problem of immersive storytelling experiences on mobile devices. In particular, we show an end-to-end system to generate 3D assets that enable real-time rendering of an opera on high end mobile phones. We call our system AR-ia and in this paper we walk through the main components and technical challenges of such a system, showing how to deliver an immersive mixed reality experience in every user’s living room. The result is a demo app – suitable for event installations – in which a user can experience three standalone scenes from Mozart’s The Magic Flute, each featuring a combination of volumetrically captured performance and tech art. By situating dramatic action in the immediate surrounds of the user, it attempts to chart a new kind of immersive experience, and by creating a vision in which any user can experience world-class operatic performance in their own home, it seeks to make the creative arts more accessible to more people. In 7 short minutes, Project AR-ia provides a glimpse into what storytelling and entertainment could look like in five to ten years time, and beyond.
References:
[1] Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. 2015. High-quality Streamable Free-viewpoint Video. ACM TOG (2015).
[2] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the Reflectance Field of a Human Face. In SIGGRAPH.
[3] Mingsong Dou, Philip Davidson, Sean Ryan Fanello, Sameh Khamis, Adarsh Kowdle, Christoph Rhemann, Vladimir Tankovich, and Shahram Izadi. 2017. Motion2Fusion: Real-time Volumetric Performance Capture. SIGGRAPH Asia (2017).
[4] Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, Pushmeet Kohli, Vladimir Tankovich, and Shahram Izadi. 2016. Fusion4D: Real-time Performance Capture of Challenging Scenes. SIGGRAPH (2016).
[5] Sean Ryan Fanello, Julien Valentin, Adarsh Kowdle, Christoph Rhemann, Vladimir Tankovich, Carlo Ciliberto, Philip Davidson, and Shahram Izadi. 2017a. Low Compute and Fully Parallel Computer Vision with HashMatch. In ICCV.
[6] Sean Ryan Fanello, Julien Valentin, Christoph Rhemann, Adarsh Kowdle, Vladimir Tankovich, Philip Davidson, and Shahram Izadi. 2017b. UltraStereo: Efficient Learning-based Matching for Active Stereo Systems. In CVPR.
[7] Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Escolano, Rohit Pandey, Jason Dourgarian, Danhang Tang, Anastasia Tkach, Adarsh Kowdle, Emily Cooper, Mingsong Dou, Sean Fanello, Graham Fyffe, Christoph Rhemann, Jonathan Taylor, Paul Debevec, and Shahram Izadi. 2019. The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting. In ACM TOG.
[8] Adarsh Kowdle, Christoph Rhemann, Sean Fanello, Andrea Tagliasacchi, Jon Taylor, Philip Davidson, Mingsong Dou, Kaiwen Guo, Cem Keskin, Sameh Khamis, David Kim, Danhang Tang, Vladimir Tankovich, Julien Valentin, and Shahram Izadi. 2018. The Need 4 Speed in Real-Time Dense Visual Tracking. SIGGRAPH Asia (2018).
[9] Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L. Davidson, Sameh Khamis, Mingsong Dou, Vladimir Tankovich, Charles Loop, Qin Cai, Philip A. Chou, Sarah Mennicken, Julien Valentin, Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli, Yuliya Lutchyn, Cem Keskin, and Shahram Izadi. 2016. Holoportation: Virtual 3D Teleportation in Real-time. In UIST.
[10] Kun Zhou, John Synder, Baining Guo, and Heung-Yeung Shum. 2004. Iso-charts: Stretch-driven Mesh Parameterization Using Spectral Analysis. In Eurographics.
