“Selectively de-animating video” by Bai, Agarwala, Agrawala and Ramamoorthi

We present a semi-automated technique for selectively deanimating video to remove the large-scale motions of one or more objects so that other motions are easier to see. The user draws strokes to indicate the regions of the video that should be immobilized, and our algorithm warps the video to remove the large-scale motion of these regions while leaving finer-scale, relative motions intact. However, such warps may introduce unnatural motions in previously motionless areas, such as background regions. We therefore use a graph-cut-based optimization to composite the warped video regions with still frames from the input video; we also optionally loop the output in a seamless manner. Our technique enables a number of applications such as clearer motion visualization, simpler creation of artistic cinemagraphs (photos that include looping motions in some regions), and new ways to edit appearance and complicated motion paths in video by manipulating a de-animated representation. We demonstrate the success of our technique with a number of motion visualizations, cinemagraphs and video editing examples created from a variety of short input videos, as well as visual and numerical comparison to previous techniques.

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Transactions on Graphics 23, 3 (Aug.), 294–302. Google ScholarDigital Library
2. Agarwala, A., Zheng, K. C., Pal, C., Agrawala, M., Cohen, M., Curless, B., Salesin, D. H., and Szeliski, R. 2005. Panoramic video textures. ACM Transactions on Graphics 24, 3 (Aug.), 821–827. Google ScholarDigital Library
3. Assa, J., Caspi, Y., and Cohen-Or, D. 2005. Action synopsis: pose selection and illustration. ACM Transactions on Graphics 24, 3 (Aug.), 667–676. Google ScholarDigital Library
4. Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video snapcut: Robust video object cutout using localized classifiers. ACM Transactions on Graphics 28, 3 (July), 70:1–70:11. Google ScholarDigital Library
5. Barnes, C., Goldman, D. B., Shechtman, E., and Finkelstein, A. 2010. Video tapestries with continuous temporal zoom. ACM Transactions on Graphics 29, 4 (July), 89:1–89:9. Google ScholarDigital Library
6. Bennett, E. P., and McMillan, L. 2007. Computational time-lapse video. ACM Transactions on Graphics 26, 3 (July), 102:1–102:6. Google ScholarDigital Library
7. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11 (Nov.), 1222–1239. Google ScholarDigital Library
8. Caspi, Y., Axelrod, A., Matsushita, Y., and Gamliel, A. 2006. Dynamic stills and clip trailers. The Visual Computer 22, 9–10, 642–652. Google ScholarDigital Library
9. Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Transactions on Graphics 21, 3 (July), 243–248. Google ScholarDigital Library
10. Chuang, Y.-Y., Goldman, D. B., Zheng, K. C., Curless, B., Salesin, D. H., and Szeliski, R. 2005. Animating pictures with stochastic motion textures. ACM Transactions on Graphics 24, 3 (Aug.), 853–860. Google ScholarDigital Library
11. Cohen, M. F., and Szeliski, R. 2006. The moment camera. Computer 39, 8, 40–45. Google ScholarDigital Library
12. Correa, C. D., and Ma, K.-L. 2010. Dynamic video narratives. ACM Transactions on Graphics 29, 4 (July), 88:1–88:9. Google ScholarDigital Library
13. Goldman, D. B., Curless, B., Salesin, D., and Seitz, S. M. 2006. Schematic storyboarding for video visualization and editing. ACM Transactions on Graphics 25, 3 (July), 862–871. Google ScholarDigital Library
14. Kim, B., and Essa, I. 2005. Video-based nonphotorealistic and expressive illustration of motion. In Computer Graphics International 2005. Google ScholarDigital Library
15. Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics 22, 3 (July), 277–286. Google ScholarDigital Library
16. Liu, C., Torralba, A., Freeman, W. T., Durand, F., and Adelson, E. H. 2005. Motion magnification. ACM Transactions on Graphics 24, 3 (Aug.), 519–526. Google ScholarDigital Library
17. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3d video stabilization. ACM Transactions on Graphics 28, 3 (July), 44:1–44:9. Google ScholarDigital Library
18. Liu, F., Gleicher, M., Wang, J., Jin, H., and Agarwala, A. 2011. Subspace video stabilization. ACM Transactions on Graphics 30, 1 (Jan.), 4:1–4:10. Google ScholarDigital Library
19. Lucas, B. D., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. International Joint Conference on Artificial Intelligence. Google ScholarDigital Library
20. Pritch, Y., Rav-Acha, A., and Peleg, S. 2008. Nonchronological video synopsis and indexing. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 11 (Nov.), 1971–1984. Google ScholarDigital Library
21. Rav-Acha, A., Kohli, P., Rother, C., and Fitzgibbon, A. 2008. Unwrap mosaics: A new representation for video editing. ACM Transactions on Graphics 27, 3 (Aug.), 17:1–17:11. Google ScholarDigital Library
22. Rubinstein, M., Liu, C., Sand, P., Durand, F., and Freeman, W. T. 2011. Motion denoising with application to time-lapse photography. IEEE Computer Vision and Pattern Recognition (CVPR) (June), 313–320. Google ScholarDigital Library
23. Schödl, A., Szeliski, R., Salesin, D. H., and Essa, I. 2000. Video textures. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, 489–498. Google ScholarDigital Library
24. Shi, J., and Tomasi, C. 1994. Good features to track. In Computer Vision and Pattern Recognition, 593–600.Google Scholar
25. Sutton, G. P., and Burrows, M. 2011. Biomechanics of jumping in the flea. J Exp Biol 214, 5 (Mar.), 836–847.Google ScholarCross Ref
26. Tompkin, J., Pece, F., Subr, K., and Kautz, J. 2011. Towards moment imagery: Automatic cinemagraphs. Visual Media Production, Conference for 0, 87–93. Google ScholarDigital Library
27. Truong, B. T., and Venkatesh, S. 2007. Video abstraction: A systematic review and classification. ACM Trans. Multimedia Comput. Commun. Appl. 3, 1 (Feb.). Google ScholarDigital Library