“Rich360: optimized spherical representation from structured panoramic camera arrays”
Conference:
Type:
Title:
- Rich360: optimized spherical representation from structured panoramic camera arrays
Session/Category Title: CAMERA CONTROL & VR
Presenter(s)/Author(s):
Moderator(s):
Abstract:
This paper presents Rich360, a novel system for creating and viewing a 360° panoramic video obtained from multiple cameras placed on a structured rig. Rich360 provides an as-rich-as-possible 360° viewing experience by effectively resolving two issues that occur in the existing pipeline. First, a deformable spherical projection surface is utilized to minimize the parallax from multiple cameras. The surface is deformed spatio-temporally according to the depth constraints estimated from the overlapping video regions. This enables fast and efficient parallax-free stitching independent of the number of views. Next, a non-uniform spherical ray sampling is performed. The density of the sampling varies depending on the importance of the image region. Finally, for interactive viewing, the non-uniformly sampled video is mapped onto a uniform viewing sphere using a UV map. This approach can preserve the richness of the input videos when the resolution of the final 360° panoramic video is smaller than the overall resolution of the input videos, which is the case for most 360° panoramic videos. We show various results from Rich360 to demonstrate the richness of the output video and the advancement in the stitching results.
References:
1. Birklbauer, C., and Bimber, O. 2014. Panorama light-field imaging. Computer Graphics Forum 33, 2, 43–52. Google ScholarDigital Library
2. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Computer Vision-ECCV 2004. 25–36.Google Scholar
3. Carroll, R., Agrawala, M., and Agarwala, A. 2009. Optimizing content-preserving projections for wide-angle images. ACM Transactions on Graphics (TOG) 28, 3, 43. Google ScholarDigital Library
4. Carroll, R., Agarwala, A., and Agrawala, M. 2010. Image warps for artistic perspective manipulation. ACM Transactions on Graphics (TOG) 29, 4, 127. Google ScholarDigital Library
5. Chang, C.-H., Hu, M.-C., Cheng, W.-H., and Chuang, Y.-Y. 2013. Rectangling stereographic projection for wide-angle image visualization. In Proceedings of the IEEE International Conference on Computer Vision, 2824–2831. Google ScholarDigital Library
6. Chang, C.-H., Sato, Y., and Chuang, Y.-Y. 2014. Shape-preserving half-projective warps for image stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3254–3261. Google ScholarDigital Library
7. Chaurasia, G., Duchene, S., Sorkine-Hornung, O., and Drettakis, G. 2013. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics (TOG) 32, 3, 30. Google ScholarDigital Library
8. Collet, A., Chuang, M., Sweeney, P., Gillett, D., Evseev, D., Calabrese, D., Hoppe, H., Kirk, A., and Sullivan, S. 2015. High-quality streamable free-viewpoint video. ACM Transactions on Graphics (TOG) 34, 4 (July), 69:1–69:13. Google ScholarDigital Library
9. Hartley, R., and Zisserman, A. 2003. Multiple view geometry in computer vision. Cambridge university press. Google ScholarDigital Library
10. He, K., Chang, H., and Sun, J. 2013. Rectangling panoramic images via warping. ACM Transactions on Graphics (TOG) 32, 4, 79. Google ScholarDigital Library
11. Igarashi, T., Moscovich, T., and Hughes, J. F. 2005. As-rigid-as-possible shape manipulation. ACM transactions on Graphics (TOG) 24, 3, 1134–1141. Google ScholarDigital Library
12. Jiang, W., and Gu, J. 2015. Video stitching with spatial-temporal content-preserving warping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 42–48.Google Scholar
13. Kopf, J., Lischinski, D., Deussen, O., Cohen-Or, D., and Cohen, M. 2009. Locally adapted projections to reduce panorama distortions. Computer Graphics Forum 28, 4, 1083–1089. Google ScholarDigital Library
14. Krähenbühl, P., Lang, M., Hornung, A., and Gross, M. 2009. A system for retargeting of streaming video. ACM Transactions on Graphics (TOG) 28, 5, 126. Google ScholarDigital Library
15. Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., and Gross, M. 2010. Nonlinear disparity mapping for stereoscopic 3d. ACM Transactions on Graphics (TOG) 29, 4, 75. Google ScholarDigital Library
16. Li, B., Heng, L., Koser, K., and Pollefeys, M. 2013. A multiple-camera system calibration toolbox using a feature descriptor-based calibration pattern. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 1301–1307.Google Scholar
17. Li, S., Yuan, L., Sun, J., and Quan, L. 2015. Dual-feature warping-based motion model estimation. In Proceedings of the IEEE International Conference on Computer Vision, 4283–4291. Google ScholarDigital Library
18. Lin, W.-Y., Liu, S., Matsushita, Y., Ng, T.-T., and Cheong, L.-F. 2011. Smoothly varying affine stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 345–352. Google ScholarDigital Library
19. Lin, C.-C., Pankanti, S. U., Ramamurthy, K. N., and Ar-avkin, A. Y. 2015. Adaptive as-natural-as-possible image stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1155–1163.Google Scholar
20. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3d video stabilization. ACM Transactions on Graphics (TOG) 28, 3, 44. Google ScholarDigital Library
21. Liu, S., Yuan, L., Tan, P., and Sun, J. 2013. Bundled camera paths for video stabilization. ACM Transactions on Graphics (TOG) 32, 4, 78. Google ScholarDigital Library
22. Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision 60, 2, 91–110. Google ScholarDigital Library
23. Mei, C., and Rives, P. 2007. Single view point omnidirectional camera calibration from planar grids. In IEEE International Conference on Robotics and Automation, 3945–3950.Google Scholar
24. Panozzo, D., Weber, O., and Sorkine, O. 2012. Robust image retargeting via axis-aligned deformation. Computer Graphics Forum 31, 2pt1, 229–236. Google ScholarDigital Library
25. Perazzi, F., Sorkine-Hornung, A., Zimmer, H., Kaufmann, P., Wang, O., Watson, S., and Gross, M. 2015. Panoramic video from unstructured camera arrays. Computer Graphics Forum 34, 2, 57–68. Google ScholarDigital Library
26. Richardt, C., Pritch, Y., Zimmer, H., and Sorkine-Hornung, A. 2013. Megastereo: Constructing high-resolution stereo panoramas. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1256–1263. Google ScholarDigital Library
27. Shum, H.-Y., Chan, S.-C., and Kang, S. B. 2008. Image-based rendering. Springer Science & Business Media. Google ScholarDigital Library
28. Szeliski, R. 2006. Image alignment and stitching: A tutorial. Foundations and Trends® in Computer Graphics and Vision 2, 1, 1–104. Google ScholarDigital Library
29. Uyttendaele, M., Criminisi, A., Kang, S. B., Winder, S., Szeliski, R., and Hartley, R. 2004. Image-based interactive exploration of real-world environments. IEEE Computer Graphics and Applications 24, 3, 52–63. Google ScholarDigital Library
30. Viola, P., and Jones, M. 2001. Rapid object detection using a boosted cascade of simple features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, I–511.Google Scholar
31. Vollmer, J., Mencl, R., and Mueller, H. 1999. Improved laplacian smoothing of noisy surface meshes. Computer Graphics Forum 18, 3, 131–138.Google ScholarCross Ref
32. von Gioi, R. G., Jakubowicz, J., Morel, J.-M., and Randall, G. 2008. Lsd: A fast line segment detector with a false detection control. IEEE Transactions on Pattern Analysis & Machine Intelligence, 4, 722–732. Google ScholarDigital Library
33. Wang, Y.-S., Fu, H., Sorkine, O., Lee, T.-Y., and Seidel, H.-P. 2009. Motion-aware temporal coherence for video resizing. ACM Transactions on Graphics (TOG) 28, 5, 127. Google ScholarDigital Library
34. Wang, Y.-S., Lin, H.-C., Sorkine, O., and Lee, T.-Y. 2010. Motion-based video retargeting with optimized crop-and-warp. ACM Transactions on Graphics (TOG) 29, 4, 90. Google ScholarDigital Library
35. Wang, Y.-S., Liu, F., Hsu, P.-S., and Lee, T.-Y. 2013. Spatially and temporally optimized video stabilization. IEEE Transactions on Visualization and Computer Graphics 19, 8, 1354–1361. Google ScholarDigital Library
36. Xu, W., and Mulligan, J. 2013. Panoramic video stitching from commodity hdtv cameras. Multimedia systems 19, 5, 407–426. Google ScholarDigital Library
37. Yildirim, G., and Susstrunk, S. 2015. Fasa: fast, accurate, and size-aware salient object detection. In Computer Vision-ACCV2014. 514–528.Google Scholar
38. Zaragoza, J., Chin, T.-J., Tran, Q.-H., Brown, M. S., and Suter, D. 2014. As-projective-as-possible image stitching with moving dlt. IEEE Transactions on Pattern Analysis and Machine Intelligence 36,7, 1285–1298. Google ScholarDigital Library
39. Zelnik-Manor, L., Peters, G., and Perona, P. 2005. Squaring the circle in panoramas. In Proceedings of the IEEE International Conference on Computer Vision, vol. 2, 1292–1299. Google ScholarDigital Library
40. Zhang, F., and Liu, F. 2014. Parallax-tolerant image stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3262–3269. Google ScholarDigital Library
41. Zhi, Q., and Cooperstock, J. R. 2012. Toward dynamic image mosaic generation with robustness to parallax. IEEE Transactions on Image Processing 21, 1, 366–378. Google ScholarDigital Library
