“360° video stabilization” by Kopf
Conference:
Type(s):
Title:
- 360° video stabilization
Session/Category Title: Video
Presenter(s)/Author(s):
Abstract:
We present a hybrid 3D-2D algorithm for stabilizing 360° video using a deformable rotation motion model. Our algorithm uses 3D analysis to estimate the rotation between key frames that are appropriately spaced such that the right amount of motion has occurred to make that operation reliable. For the remaining frames, it uses 2D optimization to maximize the visual smoothness of feature point trajectories. A new low-dimensional flexible deformed rotation motion model enables handling small translational jitter, parallax, lens deformation, and rolling shutter wobble. Our 3D-2D architecture achieves better robustness, speed, and smoothing ability than either pure 2D or 3D methods can provide. Stabilizing a video with our method takes less time than playing it at normal speed. The results are sufficiently smooth to be played back at high speed-up factors; for this purpose we present a simple 360° hyperlapse algorithm that remaps the video frame time stamps to balance the apparent camera velocity.
References:
1. Agarwal, S., Mierle, K., and Others, 2015. Ceres Solver. http://ceres-solver.org.
2. Ayache, N. 2007. Vision Stéréoscopique et Perception Multisensorielle.
3. Baker, S., Bennett, E. P., Kang, S. B., and Szeliski, R. 2010. Removing rolling shutter wobble. Computer Vision and Pattern Recognition, 2392–2399.
4. Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M., Curless, B., and Kang, S. B. 2007. Using Photographs to Enhance Videos of a Static Scene. Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR’07), 327–338.
5. Bouguet, J. 2000. Pyramidal implementation of the Lucas Kanade feature tracker. Intel Corporation, Microprocessor Research Labs.
6. Buehler, C., Bosse, M., and McMillan, L. 2001. Non-Metric Image-Based Rendering for Video Stabilization. 2014 IEEE Conference on Computer Vision and Pattern Recognition 2, 609.
7. Fischler, M. A., and Bolles, R. C. 1981. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Commun. ACM 24, 6, 381–395.
8. Goldstein, A., and Fattal, R. 2012. Video Stabilization Using Epipolar Geometry. ACM Trans. Graph. 31, 5, article no. 126.
9. Grundmann, M., Kwatra, V., and Essa, I. 2011. Auto-Directed Video Stabilization with Robust L1 Optimal Camera Paths. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
10. Kamali, M., Banno, A., Bazin, J.-C., Kweon, I., and Ikeuchi, K. 2011. Stabilizing Omnidirectional Videos Using 3D Structure and Spherical Image Warping. Proceedings of the 12th IAPR Conference on Machine Vision Applications (MVA).
11. Kasahara, S., Nagai, S., and Rekimoto, J. 2015. First Person Omnidirectional Video: System Design and Implications for Immersive Experience. Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video, 33–42.
12. Kennedy, R. S., Drexler, J., and Kennedy, R. C. 2010. Research in visually induced motion sickness. Applied Ergonomics 41, 4, 494–503. Cross Ref
13. Kneip, L., and Furgale, P. T. 2014. OpenGV: A Unified and Generalized Approach to Real-time Calibrated Geometric Vision. IEEE International Conference on Robotics and Automation (ICRA 2014), 1–8. Cross Ref
14. Kneip, L., Roland, S., and Marc, P. 2012. Finding the exact rotation between two images independently of the translation. Proceedings of the European Conference on Computer Vision (ECCV).
15. Kopf, J., Cohen, M. F., and Szeliski, R. 2014. First-person Hyper-lapse Videos. ACM Transactions on Graphics 33, 4, article no. 78.
16. Langer, T., Belyaev, A., and Seidel, H.-P. 2006. Spherical Barycentric Coordinates. Proceedings of the Fourth Eurographics Symposium on Geometry Processing (SGP ’06), 81–88.
17. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving Warps for 3D Video Stabilization. ACM Trans. Graph. 28, 3, article no. 44.
18. Liu, F., Gleicher, M., Wang, J., Jin, H., and Agarwala, A. 2011. Subspace Video Stabilization. ACM Trans. Graph. 30, 1, article no. 4.
19. Liu, S., Yuan, L., Tan, P., and Sun, J. 2013. Bundled camera Paths for Video Stabilization. ACM Transactions on Graphics (TOG) (Proceedings of SIGGRAPH 2013) 32, 4.
20. Liu, S., Yuan, L., Tan, P., and Sun, J. 2014. SteadyFlow: Spatially Smooth Optical Flow for Video Stabilization. Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, 4209–4216.
21. Morimoto, C., and Chellappa, R. 1998. Evaluation of image stabilization algorithms. Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’98), 2789–2792.
22. Nistér, D. 2004. An Efficient Solution to the Five-Point Relative Pose Problem. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 6, 756–777.
23. Shi, J., and Tomasi, C. 1994. Good Features to Track. IEEE Conference on Computer Vision and Pattern Recognition (CVPR’94), 593–600.
24. SymPy Development Team. 2016. SymPy: Python library for symbolic mathematics.
25. Yang, L., Tse, Y.-C., Sander, P. V., Lawrence, J., Nehab, D., Hoppe, H., and Wilkins, C. L. 2011. Image-based Bidirectional Scene Reprojection. ACM Trans. Graph. 30, 6, article no. 150.
