“Subspace Video Stabilization” by Liu, Gleicher, Wang, Jin and Agarwala

We present a robust and efficient approach to video stabilization that achieves high-quality camera motion for a wide range of videos. In this article, we focus on the problem of transforming a set of input 2D motion trajectories so that they are both smooth and resemble visually plausible views of the imaged scene; our key insight is that we can achieve this goal by enforcing subspace constraints on feature trajectories while smoothing them. Our approach assembles tracked features in the video into a trajectory matrix, factors it into two low-rank matrices, and performs filtering or curve fitting in a low-dimensional linear space. In order to process long videos, we propose a moving factorization that is both efficient and streamable. Our experiments confirm that our approach can efficiently provide stabilization results comparable with prior 3D methods in cases where those methods succeed, but also provides smooth camera motions in cases where such approaches often fail, such as videos that lack parallax. The presented approach offers the first method that both achieves high-quality video stabilization and is practical enough for consumer applications.

1. Baker, S., Bennett, E., Kang, S. B., and Szeliski, R. 2010. Removing rolling shutter wobble. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2392–2399.
2. Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M., Curless, B., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Proceedings of the 18th Eurographics Workshop on Rendering. 327–338.
3. Brand, M. 2002. Incremental singular value decomposition of uncertain data with missing values. In Proceedings of the European Conference on Computer Vision. 707–720.
4. Buchanan, A. M. and Fitzgibbon, A. 2005. Damped Newton algorithms for matrix factorization with missing data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 316–322.
5. Buehler, C., Bosse, M., and McMillan, L. 2001. Non-Metric image-based rendering for video stabilization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 609–614.
6. Chen, B.-Y., Lee, K.-Y., Huang, W.-T., and Lin, J.-S. 2008. Capturing intention-based full-frame video stabilization. Comput. Graph. Forum 27, 7, 1805–1814.
7. Chen, P. 2008. Optimization algorithms on subspaces: Revisiting missing data problem in low-rank matrix. Int. J. Comput. Vis. 80, 1, 125–142.
8. Davison, A. J., Reid, I. D., Molton, N. D., and Stasse, O. 2007. MonoSLAM: Real-time single camera SLAM. IEEE Trans. Patt. Anal. Mach. Intell. 26, 6, 1052–1067.
9. Fischler, M. A. and Bolles, R. C. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commu. ACM 24, 6, 381–395.
10. Fitzgibbon, A., Wexler, Y., and Zisserman, A. 2005. Image-Based rendering using image-based priors. Int. J. Comput. Vis. 63, 2, 141–151.
11. Forssén, P.-E. and Ringaby, E. 2010. Rectifying rolling shutter video from hand-held devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 507–514.
12. Gleicher, M. L. and Liu, F. 2008. Re-cinematography: Improving the camerawork of casual video. ACM Trans. Multimedia Comput. Comm. Appl. 5, 1, 1–28.
13. Goh, A. and Vidal, R. 2007. Segmenting motions of different types by unsupervised manifold clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1–6.
14. Golub, G. H. and Van Loan, C. F. 1996. Matrix Computations 3rd Ed. Johns Hopkins University Press.
15. Hartley, R. I. and Zisserman, A. 2000. Multiple View Geometry in Computer Vision. Cambridge University Press.
16. Irani, M. 2002. Multi-frame correspondence estimation using subspace constraints. Int. J. Comput. Vis. 48, 1, 39–51.
17. Lee, J. and Shin, S. Y. 2002. General construction of time-domain filters for orientation data. IEEE Trans. Vis. Comput. Graph. 8, 2, 119–128.
18. Lee, K.-Y., Chuang, Y.-Y., Chen, B.-Y., and Ouhyoung, M. 2009. Video stabilization using robust feature trajectories. In Proceedings of the IEEE International Conference on Computer Vision. 1397–1404.
19. Liang, C. K., Chang, L. W., and Chen, H. H. 2008. Analysis and compensation of rolling shutter effect. IEEE Trans. Image Process. 17, 8, 1323–1330.
20. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3D video stabilization. ACM Trans. Graph. 28, 3, Article No. 44.
21. Matsushita, Y., Ofek, E., Ge, W., Tang, X., and Shum, H.-Y. 2006. Full-frame video stabilization with motion inpainting. IEEE Trans. Patt. Anal. Mach. Intell. 28, 7, 1150–1163.
22. Meingast, M., Geyer, C., and Sastry, S. 2005. Geometric models of rolling-shutter cameras. In Proceedings of the 6th International Workshop on Omnidirectional Vision, Camera Networks, and Non-Classical Cameras. 12–19.
23. Morimoto, C. and Chellappa, R. 1997. Evaluation of image stabilization algorithms. In Proceedings of the DARPA Image Understanding Workshop. 295–302.
24. Nister, D., Naroditsky, O., and Bergen, J. 2004. Visual odometry. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 652–659.
25. Shi, J. and Tomasi, C. 1994. Good features to track. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 593–600.
26. Sinha, S., Frahm, J.-M., Pollefeys, M., and Genc, Y. 2006. GPU-based video feature tracking and matching. In Proceedings of the Workshop on Edge Computing Using New Commodity Architectures.
27. Tomasi, C. and Kanade, T. 1992. Shape and motion from image streams under orthography: a factorization method. Int. J. Comput. Vis. 9, 2, 137–154.
28. Torr, P. H. S., Fitzgibbon, A. W., and Zisserman, A. 1999. The problem of degeneracy in structure and motion recovery from uncalibrated image sequences. Int. J. Comput. Vis. 32, 1, 27–44.
29. Vidal, R., Tron, R., and Hartley, R. 2008. Multiframe motion segmentation with missing data using PowerFactorization and GPCA. Int. J. Comput. Vis. 79, 1, 85–105.
30. Zhang, G., Hua, W., Qin, X., Shao, Y., and Bao, H. 2009. Video stabilization based on a 3D perspective camera model. Vis. Comput. 25, 11, 997–1008.