“Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax” by Luo, Xu, Richardt and Yong – ACM SIGGRAPH HISTORY ARCHIVES

“Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax” by Luo, Xu, Richardt and Yong

  • ©

Conference:


Type(s):


Title:

    Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax

Session/Category Title:   IEEE TVCG Session on Virtual and Augmented Reality


Presenter(s)/Author(s):



Abstract:


    We propose a novel 360° scene representation for converting real scenes into stereoscopic 3D virtual reality content with head-motion parallax. Our image-based scene representation enables efficient synthesis of novel views with six degrees-of-freedom (6-DoF) by fusing motion fields at two scales: (1) disparity motion fields carry implicit depth information and are robustly estimated from multiple laterally displaced auxiliary viewpoints, and (2) pairwise motion fields enable real-time flow-based blending, which improves the visual fidelity of results by minimizing ghosting and view transition artifacts. Based on our scene representation, we present an end-to-end system that captures real scenes with a robotic camera arm, processes the recorded data, and finally renders the scene in a head-mounted display in real time (more than 40 Hz). Our approach is the first to support head-motion parallax when viewing real 360° scenes. We demonstrate compelling results that illustrate the enhanced visual experience — and hence sense of immersion-achieved with our approach compared to widely-used stereoscopic panoramas.

References:


    [1]
    R. Anderson, D. Gallup, J.T. Barron, J. Kontkanen, N. Snavely, C. Hernandez, S. Agarwal, and S.M. Seitz. Jump: virtual reality video. ACM Transactions on Graphics, Volume 35 Issue 6: p. pp.198, 2016.

    [2]
    M. Arikan, R. Preiner, and M. Wimmer. Multi-depth-map raytracing for efficient large-scene reconstruction. IEEE Transactions on Visualization and Computer Graphics, Volume 22 Issue 2: pp. 1127–1137, 2016.

    [3]
    C. Birklbauer and O. Bimber. Panorama light-field imaging. Computer Graphics Forum, Volume 33 Issue 2: pp. 43–52, 2014.

    [4]
    M. Brown and D.G. Lowe. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision, Volume 74 Issue 1: pp. 59–73, 2007.

    [5]
    C. Buehler, M. Bosse, L. McMillan, S. Gortler, and M. Cohen. Unstructured lumigraph rendering. In SIGGRAPH, pp. 425–432, 2001.

    [6]
    G. Chaurasia, S. Duchene, O. Sorkine-Hornung, and G. Drettakis. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics, Volume 32 Issue 3: p. pp.30, 2013.

    [7]
    S.E. Chen and L. Williams. View interpolation for image synthesis. In SIGGRAPH, pp. 279–288, 1993.

    [8]
    A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, and S. Sullivan. High-quality streamable free-viewpoint video. ACM Transactions on Graphics, Volume 34 Issue 4: p. pp.69, 2015.

    [9]
    M. Dou, S. Khamis, Y. Degtyarev, P. Davidson, S.R. Fanello, A. Kowdle, S.O. Escolano, C. Rhemann, D. Kim, J. Taylor, P. Kohli, V. Tankovich, and S. Izadi. Fusion4D: real-time performance capture of challenging scenes. ACM Transactions on Graphics, Volume 35 Issue 4: p. pp.114, 2016.

    [10]
    M. Eisemann, B. De Decker, M. Magnor, P. Bekaert, E. De Aguiar, N. Ahmed, C. Theobalt, and A. Sellent. Floating textures. Computer Graphics Forum, Volume 27 Issue 2: pp. 409–418, 2008.

    [11]
    Y. Furukawa and J. Ponce. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 32 Issue 8: pp. 1362–1376, 2010.

    [12]
    M. Goesele, J. Ackermann, S. Fuhrmann, C. Haubold, R. Klowsky, D. Steedly, and R. Szeliski. Ambient point clouds for view interpolation. ACM Transactions on Graphics, Volume 29 Issue 4: p. pp.95, 2010.

    [13]
    S.J. Gortler, R. Grzeszczuk, R. Szeliski, and M.F. Cohen. The lumigraph. In SIGGRAPH, pp. 43–54, 1996.

    [14]
    P. Hedman, S. Alsisan, R. Szeliski, and J. Kopf. Casual 3D photography. ACM Transactions on Graphics, Volume 36 Issue 6: pp. 234:1–15, 2017.

    [15]
    P. Hedman, T. Ritschel, G. Drettakis, and G. Brostow. Scalable inside-out image-based rendering. ACM Transactions on Graphics, Volume 35 Issue 6: pp. 231:1–11, 2016.

    [16]
    J. Huang, Z. Chen, D. Ceylan, and H. Jin. 6-DOF VR videos with a single 360-camera. In Proceedings of IEEE Virtual Reality (VR), pp. 37–44, 2017.

    [17]
    E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. FlowNet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2017.

    [18]
    M. Innmann, M. Zollhöfer, M. Nießner, C. Theobalt, and M. Stamminger. VolumeDeform: Real-time volumetric non-rigid reconstruction. In ECCV, 2016.

    [19]
    S.B. Kang, R. Szeliski, and M. Uyttendaele. Seamless stitching using multi-perspective plane sweep . Technical Report, Microsoft Research, 2004.

    [20]
    J. Kopf, D. Lischinski, O. Deussen, D. Cohen-Or, and M. Cohen. Locally adapted projections to reduce panorama distortions. Computer Graphics Forum, Volume 28 Issue 4: pp. 1083–1089, 2009.

    [21]
    B. Krolla, M. Diebold, B. Goldlücke, and D. Stricker. Spherical light fields. In BMVC, 2014.

    [22]
    J. Lee, B. Kim, K. Kim, Y. Kim, and J. Noh. Rich360: optimized spherical representation from structured panoramic camera arrays. ACM Transactions on Graphics, Volume 35 Issue 4: p. pp.63, 2016.

    [23]
    A. Levin and F. Durand. Linear view synthesis using a dimensionality gap light field prior. In CVPR, 2010.

    [24]
    M. Levoy and P. Hanrahan. Light field rendering. In SIGGRAPH, pp. 31–42, 1996.

    [25]
    K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar. Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Transactions on Graphics, Volume 32 Issue 4: p. pp.46, 2013.

    [26]
    K. Matzen, M.F. Cohen, B. Evans, J. Kopf, and R. Szeliski. Low-cost 360 stereo photography and video capture. ACM Transactions on Graphics, Volume 36 Issue 4: p. pp.148, 2017.

    [27]
    K. Mitra and A. Veeraraghavan. Light field denoising, light field super-resolution and stereo camera based refocussing using a GMM light field patch prior. In CVPR Workshops, 2012.

    [28]
    R.A. Newcombe, D. Fox, and S.M. Seitz. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In CVPR, 2015.

    [29]
    R.A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A.J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon. KinectFusion: Real-time dense surface mapping and tracking. In ISMAR, 2011.

    [30]
    M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics, Volume 32 Issue 6: pp. 169:1–11, 2013.

    [31]
    S. Peleg, M. Ben-Ezra, and Y. Pritch. Omnistereo: Panoramic stereo imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 23 Issue 3: pp. 279–290, 2001.

    [32]
    F. Perazzi, A. Sorkine-Hornung, H. Zimmer, P. Kaufmann, O. Wang, S. Watson, and M. Gross. Panoramic video from unstructured camera arrays. Computer Graphics Forum, Volume 34 Issue 2: pp. 57–68, 2015.

    [33]
    S. Philip, B. Summa, J. Tierny, P.-T. Bremer, and V. Pascucci. Distributed seams for gigapixel panoramas. IEEE Transactions on Visualization and Computer Graphics, Volume 21 Issue 3: pp. 350–362, 2015.

    [34]
    L. Piegl and W. Tiller. The NURBS book . Springer, 2012.

    [35]
    C. Richardt, Y. Pritch, H. Zimmer, and A. Sorkine-Hornung. Megastereo: Constructing high-resolution stereo panoramas. In CVPR, 2013.

    [36]
    L. Shi, H. Hassanieh, A. Davis, D. Katabi, and F. Durand. Light field reconstruction using sparsity in the continuous Fourier domain. ACM Transactions on Graphics, Volume 34 Issue 1: p. pp.12, 2014.

    [37]
    H.-Y. Shum, S.-C. Chan, and S.B. Kang. Image-Based Rendering . Springer, 2007.

    [38]
    H.-Y. Shum and L.-W. He. Rendering with concentric mosaics. In SIGGRAPH, pp. 299–306, 1999.

    [39]
    H.-Y. Shum and R. Szeliski. Stereo reconstruction from multiperspective panoramas. In ICCV, 1999.

    [40]
    H.-Y. Shum and R. Szeliski. Systems and experiment paper: Construction of panoramic image mosaics with global and local alignment. International Journal of Computer Vision, Volume 36 Issue 2: pp. 101–130, 2000.

    [41]
    R. Szeliski. Image alignment and stitching: A tutorial. Foundations and Trends® in Computer Graphics and Vision, Volume 2 Issue 1: pp. 1–104, 2006.

    [42]
    S. Wanner and B. Goldluecke. Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 36 Issue 3: pp. 606–619, 2014.

    [43]
    K. Xu, H. Huang, Y. Shi, H. Li, P. Long, J. Caichen, W. Sun, and B. Chen. Autoscanning for coupled scene reconstruction and proactive object analysis. ACM Transactions on Graphics, Volume 34 Issue 6: p. pp.177, 2015.

    [44]
    L. Xu, J. Jia, and Y. Matsushita. Motion detail preserving optical flow estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 34 Issue 9: pp. 1744–1757, 2012.

    [45]
    L. Zelnik-Manor, G. Peters, and P. Perona. Squaring the circle in panoramas. In ICCV, 2005.

    [46]
    Y. Zhang, W. Xu, Y. Tong, and K. Zhou. Online structure analysis for real-time indoor scene reconstruction. ACM Transactions on Graphics, Volume 34 Issue 5: p. pp.159, 2015.

    [47]
    Z. Zhang, Y. Liu, and Q. Dai. Light field from micro-baseline image pair. In CVPR, 2015.


ACM Digital Library Publication:



Overview Page:



Submit a story:

If you would like to submit a story about this presentation, please contact us: historyarchives@siggraph.org