“Keyframe-based tracking for rotoscoping and animation” by Agarwala, Hertzmann, Salesin and Seitz

We describe a new approach to rotoscoping — the process of tracking contours in a video sequence — that combines computer vision with user interaction. In order to track contours in video, the user specifies curves in two or more frames; these curves are used as keyframes by a computer-vision-based tracking algorithm. The user may interactively refine the curves and then restart the tracking algorithm. Combining computer vision with user interaction allows our system to track any sequence with significantly less effort than interpolation-based systems — and with better reliability than “pure” computer vision systems. Our tracking algorithm is cast as a spacetime optimization problem that solves for time-varying curve shapes based on an input video sequence and user-specified constraints. We demonstrate our system with several rotoscoped examples. Additionally, we show how these rotoscoped contours can be used to help create cartoon animation by attaching user-drawn strokes to the tracked contours.

1. AGARWALA, A. 2002. SnakeToonz: A Semi-Automatic Approach to Creating Cel Animation from Video. In NPAR 2002: Second International Symposium on Non Photorealistic Rendering, 139–146. Google ScholarDigital Library
2. BEIER, T., AND NEELY, S. 1992. Feature-based image metamorphosis. In Computer Graphics (Proceedings of SIGGRAPH 92), vol. 26, 35–42. Google ScholarDigital Library
3. BERGEN, J. R., ANANDAN, P., HANNA, K. J., AND HINGORANI, R. 1992. Hierarchical model-based motion estimation. In European Conference on Computer Vision, 237–252. Google ScholarDigital Library
4. BLACK, M. J., AND ANANDAN, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding 63, 1, 75–104. Google ScholarDigital Library
5. BLAKE, A., AND ISARD, M. 1998. Active Contours, Springer-Verlag.Google Scholar
6. BURTNYK, N., AND WEIN, M. 1976. Interactive skeleton techniques for enhancing motion dynamics in key frame animation. CACM 19 (Oct.), 564–569. Google ScholarDigital Library
7. CHUANG, Y.-Y., AGARWALA, A., CURLESS, B., SALESIN, D. H., AND SZELISKI, R. 2002. Video matting of complex scenes. ACM Transactions on Graphics 21, 3, 243–248. Google ScholarDigital Library
8. COHEN, M. F. 1992. Interactive spacetime control for animation. In Computer Graphics (Proceedings of SIGGRAPH 92), vol. 26, 293–302. Google ScholarDigital Library
9. FLEISCHER, M., 1917. Method of Producing Moving Picture Cartoons. US Patent no. 1,242,674.Google Scholar
10. GEIGER, D., GUPTA, A., COSTA, L., AND VLONTZOS, J. 1995. Dynamic programming for detecting, tracking and matching deformable contours. IEEE Transactions On Pattern Analysis and Machine Intelligence 17, 3, 294–302. Google ScholarDigital Library
11. GOLDMAN, D., 2003. Computer graphics supervisor, Industrial Light & Magic, personal communication.Google Scholar
12. HALL, J., GREENHILL, D., AND JONES, G. 1997. Segmenting film sequences using active surfaces. In International Conference on Image Processing (ICIP), 751–754. Google ScholarDigital Library
13. HOCH, M., AND LITWINOWICZ, P. C. 1996. A semi-automatic system for edge tracking with snakes. The Visual Computer 12, 2, 75–83.Google ScholarCross Ref
14. HSU, S. C., AND LEE, I. H. H. 1994. Drawing and animation using skeletal strokes. In Proceedings of SIGGRAPH 94, 109–118. Google ScholarDigital Library
15. IRANI, M. 2002. Multi-Frame Correspondence Estimation Using Subspace Constraints. International Journal of Computer Vision 48, 3, 173–194. Google ScholarDigital Library
16. KALNINS, R. D., MARKOSIAN, L., MEIER, B. J., KOWALSKI, M. A., LEE, J. C., DAVIDSON, P. L., WEBB, M., HUGHES, J. F., AND FINKELSTEIN, A. 2002. WYSIWYG NPR: Drawing Strokes Directly on 3D Models. ACM Transactions on Graphics 21, 3, 755–762. Google ScholarDigital Library
17. KASS, M., WITKIN, A., AND TERZOPOULOS, D. 1987. Snakes: Active contour models. International Journal of Computer Vision 1, 4, 321–331.Google ScholarCross Ref
18. KORT, A. 2002. Computer aided inbetweening. In NPAR 2002: Second International Symposium on Non Photorealistic Rendering, 125–132. Google ScholarDigital Library
19. LINKLATER, R., 2001. Waking Life. 20th Century Fox.Google Scholar
20. LITWINOWICZ, P., AND WILLIAMS, L. 1994. Animating images with drawings. In Proceedings of SIGGRAPH 94, 409–412. Google ScholarDigital Library
21. LITWINOWICZ, P. 1997. Processing images and video for an impressionist effect. In Proceedings of SIGGRAPH 97, 407–414. Google ScholarDigital Library
22. LUCAS, B. D., AND KANADE, T. 1981. An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI’81), 674–679.Google Scholar
23. LUO, H., AND ELEFTHERIADIS, A. 1999. Spatial temporal active contour interpolation for semi-automatic video object generation. In International Conference on Image Processing (ICIP), 944–948.Google ScholarCross Ref
24. MEIER, B. J. 1996. Painterly rendering for animation. In Proceedings of SIGGRAPH 96, 477–484. Google ScholarDigital Library
25. MITSUNAGA, T., YOKOYAMA, T., AND TOTSUKA, T. 1995. Autokey: Human assisted key extraction. In Proceedings of SIGGRAPH 95, 265–272. Google ScholarDigital Library
26. MORTENSEN, E. N., AND BARRETT, W. A. 1995. Intelligent scissors for image composition. In Proceedings of SIGGRAPH 95, 191–198. Google ScholarDigital Library
27. MORTENSEN, E. N., 1999. Vision-assisted image editing. Computer Graphics 33, 4 (Nov.), 55–57. Google ScholarDigital Library
28. NOCEDAL, J., AND WRIGHT, S. J. 1999. Numerical Optimization. Springer.Google Scholar
29. SEDERBERG, T. W., AND GREENWOOD, E. 1992. A physically based approach to 2d shape blending. In Computer Graphics (Proceedings of SIGGRAPH 92), vol. 26, 25–34. Google ScholarDigital Library
30. SEDERBERG, T. W., GAO, P., WANG, G., AND MU, H. 1993. 2d shape blending: An intrinsic solution to the vertex path problem. In Proceedings of SIGGRAPH 93, 15–18. Google ScholarDigital Library
31. SMITH, A. R., AND BLINN, J. F. 1996. Blue screen matting. In Proceedings of SIGGRAPH 96, 259–268. Google ScholarDigital Library
32. STEIHAUG, T. 1983. The conjugate gradient method and trust regions in large scale optimization. SIAM Journal on Numerical Analysis 20, 3, 626–637.Google ScholarCross Ref
33. STEWART, S., 2003. Confessions of a roto artist: Three rules for better mattes. http://www.pinnaclesys.com/SupportFiles/Rotoscoping.pdf.Google Scholar
34. SZELISKI, R. 1990. Fast surface interpolation using hierarchical basis functions. IEEE Transactions On Pattern Analysis and Machine Intelligence 12, 6, 513–528. Google ScholarDigital Library
35. TORRESANI, L., AND BREGLER, C. 2002. Space-time tracking. In European Conference on Computer Vision, 801–802. Google ScholarDigital Library
36. WITKIN, A., AND KASS, M. 1988. Spacetime constraints. In Computer Graphics (Proceedings of SIGGRAPH 88), vol. 22, 159–168. Google ScholarDigital Library