“Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax” by Luo, Xu, Richardt and Yong
Conference:
Type(s):
Title:
- Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax
Session/Category Title: IEEE TVCG Session on Virtual and Augmented Reality
Presenter(s)/Author(s):
Abstract:
We propose a novel 360° scene representation for converting real scenes into stereoscopic 3D virtual reality content with head-motion parallax. Our image-based scene representation enables efficient synthesis of novel views with six degrees-of-freedom (6-DoF) by fusing motion fields at two scales: (1) disparity motion fields carry implicit depth information and are robustly estimated from multiple laterally displaced auxiliary viewpoints, and (2) pairwise motion fields enable real-time flow-based blending, which improves the visual fidelity of results by minimizing ghosting and view transition artifacts. Based on our scene representation, we present an end-to-end system that captures real scenes with a robotic camera arm, processes the recorded data, and finally renders the scene in a head-mounted display in real time (more than 40 Hz). Our approach is the first to support head-motion parallax when viewing real 360° scenes. We demonstrate compelling results that illustrate the enhanced visual experience — and hence sense of immersion-achieved with our approach compared to widely-used stereoscopic panoramas.
References:
[1]
R. Anderson, D. Gallup, J.T. Barron, J. Kontkanen, N. Snavely, C. Hernandez, S. Agarwal, and S.M. Seitz. Jump: virtual reality video. ACM Transactions on Graphics, Volume 35 Issue 6: p. pp.198, 2016.
[2]
M. Arikan, R. Preiner, and M. Wimmer. Multi-depth-map raytracing for efficient large-scene reconstruction. IEEE Transactions on Visualization and Computer Graphics, Volume 22 Issue 2: pp. 1127–1137, 2016.
[3]
C. Birklbauer and O. Bimber. Panorama light-field imaging. Computer Graphics Forum, Volume 33 Issue 2: pp. 43–52, 2014.
[4]
M. Brown and D.G. Lowe. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision, Volume 74 Issue 1: pp. 59–73, 2007.
[5]
C. Buehler, M. Bosse, L. McMillan, S. Gortler, and M. Cohen. Unstructured lumigraph rendering. In SIGGRAPH, pp. 425–432, 2001.
[6]
G. Chaurasia, S. Duchene, O. Sorkine-Hornung, and G. Drettakis. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics, Volume 32 Issue 3: p. pp.30, 2013.
[7]
S.E. Chen and L. Williams. View interpolation for image synthesis. In SIGGRAPH, pp. 279–288, 1993.
[8]
A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, and S. Sullivan. High-quality streamable free-viewpoint video. ACM Transactions on Graphics, Volume 34 Issue 4: p. pp.69, 2015.
[9]
M. Dou, S. Khamis, Y. Degtyarev, P. Davidson, S.R. Fanello, A. Kowdle, S.O. Escolano, C. Rhemann, D. Kim, J. Taylor, P. Kohli, V. Tankovich, and S. Izadi. Fusion4D: real-time performance capture of challenging scenes. ACM Transactions on Graphics, Volume 35 Issue 4: p. pp.114, 2016.
[10]
M. Eisemann, B. De Decker, M. Magnor, P. Bekaert, E. De Aguiar, N. Ahmed, C. Theobalt, and A. Sellent. Floating textures. Computer Graphics Forum, Volume 27 Issue 2: pp. 409–418, 2008.
[11]
Y. Furukawa and J. Ponce. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 32 Issue 8: pp. 1362–1376, 2010.
[12]
M. Goesele, J. Ackermann, S. Fuhrmann, C. Haubold, R. Klowsky, D. Steedly, and R. Szeliski. Ambient point clouds for view interpolation. ACM Transactions on Graphics, Volume 29 Issue 4: p. pp.95, 2010.
[13]
S.J. Gortler, R. Grzeszczuk, R. Szeliski, and M.F. Cohen. The lumigraph. In SIGGRAPH, pp. 43–54, 1996.
[14]
P. Hedman, S. Alsisan, R. Szeliski, and J. Kopf. Casual 3D photography. ACM Transactions on Graphics, Volume 36 Issue 6: pp. 234:1–15, 2017.
[15]
P. Hedman, T. Ritschel, G. Drettakis, and G. Brostow. Scalable inside-out image-based rendering. ACM Transactions on Graphics, Volume 35 Issue 6: pp. 231:1–11, 2016.
[16]
J. Huang, Z. Chen, D. Ceylan, and H. Jin. 6-DOF VR videos with a single 360-camera. In Proceedings of IEEE Virtual Reality (VR), pp. 37–44, 2017.
[17]
E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. FlowNet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2017.
[18]
M. Innmann, M. Zollhöfer, M. Nießner, C. Theobalt, and M. Stamminger. VolumeDeform: Real-time volumetric non-rigid reconstruction. In ECCV, 2016.
[19]
S.B. Kang, R. Szeliski, and M. Uyttendaele. Seamless stitching using multi-perspective plane sweep . Technical Report, Microsoft Research, 2004.
[20]
J. Kopf, D. Lischinski, O. Deussen, D. Cohen-Or, and M. Cohen. Locally adapted projections to reduce panorama distortions. Computer Graphics Forum, Volume 28 Issue 4: pp. 1083–1089, 2009.
[21]
B. Krolla, M. Diebold, B. Goldlücke, and D. Stricker. Spherical light fields. In BMVC, 2014.
[22]
J. Lee, B. Kim, K. Kim, Y. Kim, and J. Noh. Rich360: optimized spherical representation from structured panoramic camera arrays. ACM Transactions on Graphics, Volume 35 Issue 4: p. pp.63, 2016.
[23]
A. Levin and F. Durand. Linear view synthesis using a dimensionality gap light field prior. In CVPR, 2010.
[24]
M. Levoy and P. Hanrahan. Light field rendering. In SIGGRAPH, pp. 31–42, 1996.
[25]
K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar. Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Transactions on Graphics, Volume 32 Issue 4: p. pp.46, 2013.
[26]
K. Matzen, M.F. Cohen, B. Evans, J. Kopf, and R. Szeliski. Low-cost 360 stereo photography and video capture. ACM Transactions on Graphics, Volume 36 Issue 4: p. pp.148, 2017.
[27]
K. Mitra and A. Veeraraghavan. Light field denoising, light field super-resolution and stereo camera based refocussing using a GMM light field patch prior. In CVPR Workshops, 2012.
[28]
R.A. Newcombe, D. Fox, and S.M. Seitz. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In CVPR, 2015.
[29]
R.A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A.J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon. KinectFusion: Real-time dense surface mapping and tracking. In ISMAR, 2011.
[30]
M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics, Volume 32 Issue 6: pp. 169:1–11, 2013.
[31]
S. Peleg, M. Ben-Ezra, and Y. Pritch. Omnistereo: Panoramic stereo imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 23 Issue 3: pp. 279–290, 2001.
[32]
F. Perazzi, A. Sorkine-Hornung, H. Zimmer, P. Kaufmann, O. Wang, S. Watson, and M. Gross. Panoramic video from unstructured camera arrays. Computer Graphics Forum, Volume 34 Issue 2: pp. 57–68, 2015.
[33]
S. Philip, B. Summa, J. Tierny, P.-T. Bremer, and V. Pascucci. Distributed seams for gigapixel panoramas. IEEE Transactions on Visualization and Computer Graphics, Volume 21 Issue 3: pp. 350–362, 2015.
[34]
L. Piegl and W. Tiller. The NURBS book . Springer, 2012.
[35]
C. Richardt, Y. Pritch, H. Zimmer, and A. Sorkine-Hornung. Megastereo: Constructing high-resolution stereo panoramas. In CVPR, 2013.
[36]
L. Shi, H. Hassanieh, A. Davis, D. Katabi, and F. Durand. Light field reconstruction using sparsity in the continuous Fourier domain. ACM Transactions on Graphics, Volume 34 Issue 1: p. pp.12, 2014.
[37]
H.-Y. Shum, S.-C. Chan, and S.B. Kang. Image-Based Rendering . Springer, 2007.
[38]
H.-Y. Shum and L.-W. He. Rendering with concentric mosaics. In SIGGRAPH, pp. 299–306, 1999.
[39]
H.-Y. Shum and R. Szeliski. Stereo reconstruction from multiperspective panoramas. In ICCV, 1999.
[40]
H.-Y. Shum and R. Szeliski. Systems and experiment paper: Construction of panoramic image mosaics with global and local alignment. International Journal of Computer Vision, Volume 36 Issue 2: pp. 101–130, 2000.
[41]
R. Szeliski. Image alignment and stitching: A tutorial. Foundations and Trends® in Computer Graphics and Vision, Volume 2 Issue 1: pp. 1–104, 2006.
[42]
S. Wanner and B. Goldluecke. Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 36 Issue 3: pp. 606–619, 2014.
[43]
K. Xu, H. Huang, Y. Shi, H. Li, P. Long, J. Caichen, W. Sun, and B. Chen. Autoscanning for coupled scene reconstruction and proactive object analysis. ACM Transactions on Graphics, Volume 34 Issue 6: p. pp.177, 2015.
[44]
L. Xu, J. Jia, and Y. Matsushita. Motion detail preserving optical flow estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 34 Issue 9: pp. 1744–1757, 2012.
[45]
L. Zelnik-Manor, G. Peters, and P. Perona. Squaring the circle in panoramas. In ICCV, 2005.
[46]
Y. Zhang, W. Xu, Y. Tong, and K. Zhou. Online structure analysis for real-time indoor scene reconstruction. ACM Transactions on Graphics, Volume 34 Issue 5: p. pp.159, 2015.
[47]
Z. Zhang, Y. Liu, and Q. Dai. Light field from micro-baseline image pair. In CVPR, 2015.
