“Video object cut and paste” by Li, Sun and Shum

In this paper, we present a system for cutting a moving object out from a video clip. The cutout object sequence can be pasted onto another video or a background image. To achieve this, we first apply a new 3D graph cut based segmentation approach on the spatial-temporal video volume. Our algorithm partitions watershed presegmentation regions into foreground and background while preserving temporal coherence. Then, the initial segmentation result is refined locally. Given two frames in the video sequence, we specify two respective windows of interest which are then tracked using a bi-directional feature tracking algorithm. For each frame in between these two given frames, the segmentation in each tracked window is refined using a 2D graph cut that utilizes a local color model. Moreover, we provide brush tools for the user to control the object boundary precisely wherever needed. Based on the accurate binary segmentation result, we apply coherent matting to extract the alpha mattes and foreground colors of the object.

1. Agarwala, A., Hertzmann, A., Seitz, S., and Salesin, D. H. 2004. Keyframe-based tracking for rotoscoping and animation. In Proceedings of ACM SIGGRAPH 2004, 584–591. Google ScholarDigital Library
2. Apostoloff, N. E., and Fitzgibbon, A. W. 2004. Bayesian video matting using learnt image priors. In Proceedings of CVPR 2004, I: 407–414.Google Scholar
3. Bellman, R. E. 1957. Dynamic Programming. Princeton University Press. Princeton. NJ. Google ScholarDigital Library
4. Black, M. J., and Ananda, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. In Computer Vision and Image Understanding, vol. 63, 75–104. Google ScholarDigital Library
5. Blake, A., and Isard. M. 1998. Active contours. In Springer Verlag, London.Google Scholar
6. Blake, A., Rother, C., Brown, M., P. Perez, and P. Torr. 2004. Interactive image segmentation using an adaptive gmmrf model. In Proceedings of ECCV. I: 428–441.Google Scholar
7. Boykov, Y., and Jolly, M. P. 2001. Interactive graph cuts for optimal boundary & region segmentation of objects in n-d images. In Proceedings of ICCV 2001. I: 105–112.Google Scholar
8. Chuang, Y.-Y. Curless, B., Salesin, D. H., and Szeliski. R. 2001. A bayesian approach to digital matting. In Proceedings of CVPR 2001, II: 264–271.Google Scholar
9. Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski. R. 2002. Video matting of complex scenes. In Proceedings of ACM SIGGRAPH 2002, 243–248. Google ScholarDigital Library
10. Drori, I., Leyvand, T., Cohen-Or, D., and Yeshurun, H. 2004. Interactive object segmentation in video by fitting splines to graph cuts. In ACM SIGGRAPH 2004 Posters Session. Google ScholarDigital Library
11. Kass, M., Witkin, A., and Terzopoulos, D. 1987. Snakes: Active contour models. International Journal on Computer Vision 1, 4, 321–331.Google ScholarCross Ref
12. Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. In Proceedings of ACM SIGGRAPH 2003, 277–286. Google ScholarDigital Library
13. Li, Y., Sun, J., Tang, C. K., and Shum, H. Y. 2004. Lazy snapping. In Proceedings of ACM SIGGRAPH 2004, 303–308. Google ScholarDigital Library
14. Mitsunaga, T., Yokoyama, T., and Totsuka, T. 1995. Autokey: Human assisted key extraction. In Proceedings of ACM SIGGRAPH’95, 265–272. Google ScholarDigital Library
15. Mortensen, E. N., and Barrett, W. A. 1999. Toboggan-based intelligent scissors with a four parameter edge model. In Proceedings of CVPR 1999, II: 452–458.Google Scholar
16. Rother, C., Blake, A., and Kolmogorov, V. 2004. Grabcut – interactive fore-ground extraction using iterated graph cuts. In Proceedings of ACM SIGGRAPH 2004, 309–314. Google ScholarDigital Library
17. Shum, H., Sun, J., Yamazaki, S., Li, Y., and Tang, C. 2004. Pop-up light field: An interactive image-based modeling and rendering system. ACM Transaction of Graphics 23, 2, 143–162. Google ScholarDigital Library
18. Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In Proceedings of ACM SIGGRAPH 1996, 259–268. Google ScholarDigital Library
19. Vincent, L., and Soille, P. 1991. Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Tran. on PAMI 13, 6, 583–598. Google ScholarDigital Library