“Keyframe-based tracking for rotoscoping and animation” by Agarwala, Hertzmann, Salesin and Seitz

  • ©Aseem Agarwala, Aaron Hertzmann, David H. Salesin, and Steven Seitz




    Keyframe-based tracking for rotoscoping and animation



    We describe a new approach to rotoscoping — the process of tracking contours in a video sequence — that combines computer vision with user interaction. In order to track contours in video, the user specifies curves in two or more frames; these curves are used as keyframes by a computer-vision-based tracking algorithm. The user may interactively refine the curves and then restart the tracking algorithm. Combining computer vision with user interaction allows our system to track any sequence with significantly less effort than interpolation-based systems — and with better reliability than “pure” computer vision systems. Our tracking algorithm is cast as a spacetime optimization problem that solves for time-varying curve shapes based on an input video sequence and user-specified constraints. We demonstrate our system with several rotoscoped examples. Additionally, we show how these rotoscoped contours can be used to help create cartoon animation by attaching user-drawn strokes to the tracked contours.


    1. AGARWALA, A. 2002. SnakeToonz: A Semi-Automatic Approach to Creating Cel Animation from Video. In NPAR 2002: Second International Symposium on Non Photorealistic Rendering, 139–146. Google ScholarDigital Library
    2. BEIER, T., AND NEELY, S. 1992. Feature-based image metamorphosis. In Computer Graphics (Proceedings of SIGGRAPH 92), vol. 26, 35–42. Google ScholarDigital Library
    3. BERGEN, J. R., ANANDAN, P., HANNA, K. J., AND HINGORANI, R. 1992. Hierarchical model-based motion estimation. In European Conference on Computer Vision, 237–252. Google ScholarDigital Library
    4. BLACK, M. J., AND ANANDAN, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding 63, 1, 75–104. Google ScholarDigital Library
    5. BLAKE, A., AND ISARD, M. 1998. Active Contours, Springer-Verlag.Google Scholar
    6. BURTNYK, N., AND WEIN, M. 1976. Interactive skeleton techniques for enhancing motion dynamics in key frame animation. CACM 19 (Oct.), 564–569. Google ScholarDigital Library
    7. CHUANG, Y.-Y., AGARWALA, A., CURLESS, B., SALESIN, D. H., AND SZELISKI, R. 2002. Video matting of complex scenes. ACM Transactions on Graphics 21, 3, 243–248. Google ScholarDigital Library
    8. COHEN, M. F. 1992. Interactive spacetime control for animation. In Computer Graphics (Proceedings of SIGGRAPH 92), vol. 26, 293–302. Google ScholarDigital Library
    9. FLEISCHER, M., 1917. Method of Producing Moving Picture Cartoons. US Patent no. 1,242,674.Google Scholar
    10. GEIGER, D., GUPTA, A., COSTA, L., AND VLONTZOS, J. 1995. Dynamic programming for detecting, tracking and matching deformable contours. IEEE Transactions On Pattern Analysis and Machine Intelligence 17, 3, 294–302. Google ScholarDigital Library
    11. GOLDMAN, D., 2003. Computer graphics supervisor, Industrial Light & Magic, personal communication.Google Scholar
    12. HALL, J., GREENHILL, D., AND JONES, G. 1997. Segmenting film sequences using active surfaces. In International Conference on Image Processing (ICIP), 751–754. Google ScholarDigital Library
    13. HOCH, M., AND LITWINOWICZ, P. C. 1996. A semi-automatic system for edge tracking with snakes. The Visual Computer 12, 2, 75–83.Google ScholarCross Ref
    14. HSU, S. C., AND LEE, I. H. H. 1994. Drawing and animation using skeletal strokes. In Proceedings of SIGGRAPH 94, 109–118. Google ScholarDigital Library
    15. IRANI, M. 2002. Multi-Frame Correspondence Estimation Using Subspace Constraints. International Journal of Computer Vision 48, 3, 173–194. Google ScholarDigital Library
    16. KALNINS, R. D., MARKOSIAN, L., MEIER, B. J., KOWALSKI, M. A., LEE, J. C., DAVIDSON, P. L., WEBB, M., HUGHES, J. F., AND FINKELSTEIN, A. 2002. WYSIWYG NPR: Drawing Strokes Directly on 3D Models. ACM Transactions on Graphics 21, 3, 755–762. Google ScholarDigital Library
    17. KASS, M., WITKIN, A., AND TERZOPOULOS, D. 1987. Snakes: Active contour models. International Journal of Computer Vision 1, 4, 321–331.Google ScholarCross Ref
    18. KORT, A. 2002. Computer aided inbetweening. In NPAR 2002: Second International Symposium on Non Photorealistic Rendering, 125–132. Google ScholarDigital Library
    19. LINKLATER, R., 2001. Waking Life. 20th Century Fox.Google Scholar
    20. LITWINOWICZ, P., AND WILLIAMS, L. 1994. Animating images with drawings. In Proceedings of SIGGRAPH 94, 409–412. Google ScholarDigital Library
    21. LITWINOWICZ, P. 1997. Processing images and video for an impressionist effect. In Proceedings of SIGGRAPH 97, 407–414. Google ScholarDigital Library
    22. LUCAS, B. D., AND KANADE, T. 1981. An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI’81), 674–679.Google Scholar
    23. LUO, H., AND ELEFTHERIADIS, A. 1999. Spatial temporal active contour interpolation for semi-automatic video object generation. In International Conference on Image Processing (ICIP), 944–948.Google ScholarCross Ref
    24. MEIER, B. J. 1996. Painterly rendering for animation. In Proceedings of SIGGRAPH 96, 477–484. Google ScholarDigital Library
    25. MITSUNAGA, T., YOKOYAMA, T., AND TOTSUKA, T. 1995. Autokey: Human assisted key extraction. In Proceedings of SIGGRAPH 95, 265–272. Google ScholarDigital Library
    26. MORTENSEN, E. N., AND BARRETT, W. A. 1995. Intelligent scissors for image composition. In Proceedings of SIGGRAPH 95, 191–198. Google ScholarDigital Library
    27. MORTENSEN, E. N., 1999. Vision-assisted image editing. Computer Graphics 33, 4 (Nov.), 55–57. Google ScholarDigital Library
    28. NOCEDAL, J., AND WRIGHT, S. J. 1999. Numerical Optimization. Springer.Google Scholar
    29. SEDERBERG, T. W., AND GREENWOOD, E. 1992. A physically based approach to 2d shape blending. In Computer Graphics (Proceedings of SIGGRAPH 92), vol. 26, 25–34. Google ScholarDigital Library
    30. SEDERBERG, T. W., GAO, P., WANG, G., AND MU, H. 1993. 2d shape blending: An intrinsic solution to the vertex path problem. In Proceedings of SIGGRAPH 93, 15–18. Google ScholarDigital Library
    31. SMITH, A. R., AND BLINN, J. F. 1996. Blue screen matting. In Proceedings of SIGGRAPH 96, 259–268. Google ScholarDigital Library
    32. STEIHAUG, T. 1983. The conjugate gradient method and trust regions in large scale optimization. SIAM Journal on Numerical Analysis 20, 3, 626–637.Google ScholarCross Ref
    33. STEWART, S., 2003. Confessions of a roto artist: Three rules for better mattes. http://www.pinnaclesys.com/SupportFiles/Rotoscoping.pdf.Google Scholar
    34. SZELISKI, R. 1990. Fast surface interpolation using hierarchical basis functions. IEEE Transactions On Pattern Analysis and Machine Intelligence 12, 6, 513–528. Google ScholarDigital Library
    35. TORRESANI, L., AND BREGLER, C. 2002. Space-time tracking. In European Conference on Computer Vision, 801–802. Google ScholarDigital Library
    36. WITKIN, A., AND KASS, M. 1988. Spacetime constraints. In Computer Graphics (Proceedings of SIGGRAPH 88), vol. 22, 159–168. Google ScholarDigital Library

ACM Digital Library Publication:

Overview Page: