“Spacetime Expression Cloning for Blendshapes” by Seol, Lewis, Seo, Choi, Anjyo, et al. …

  • ©Yeongho Seol, John-Peter Lewis, Jaewoo Seo, Byungkuk Choi, Ken-ichi (Ken) Anjyo, and Junyong Noh




    Spacetime Expression Cloning for Blendshapes



    The goal of a practical facial animation retargeting system is to reproduce the character of a source animation on a target face while providing room for additional creative control by the animator. This article presents a novel spacetime facial animation retargeting method for blendshape face models. Our approach starts from the basic principle that the source and target movements should be similar. By interpreting movement as the derivative of position with time, and adding suitable boundary conditions, we formulate the retargeting problem as a Poisson equation. Specified (e.g., neutral) expressions at the beginning and end of the animation as well as any user-specified constraints in the middle of the animation serve as boundary conditions. In addition, a model-specific prior is constructed to represent the plausible expression space of the target face during retargeting. A Bayesian formulation is then employed to produce target animation that is consistent with the source movements while satisfying the prior constraints. Since the preservation of temporal derivatives is the primary goal of the optimization, the retargeted motion preserves the rhythm and character of the source movement and is free of temporal jitter. More importantly, our approach provides spacetime editing for the popular blendshape representation of facial models, exhibiting smooth and controlled propagation of user edits across surrounding frames.


    Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30, 75:1–75:10. Google ScholarDigital Library
    Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-Scale capture of facial geometry and motion. ACM Trans. Graph. 26, 3, 33. Google ScholarDigital Library
    Blanz, V. and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive techniques. ACM Press/Addison-Wesley Publishing Co., New York, 187–194. Google ScholarDigital Library
    Borshukov, G., Piponi, D., Larsen, O., Lewis, J., and Tempelaar-Lietz, C. 2003. Universal capture: Image-based facial animation for “The Matrix Reloaded.” In ACM SIGGRAPH 2003 Sketches & Applications. ACM, New York, 1–1. Google ScholarDigital Library
    Bregler, C., Loeb, L., Chuang, E., and Deshpande, H. 2002. Turning to the masters: Motion capturing cartoons. ACM Trans. Graph. 21, 3, 399–407. Google ScholarDigital Library
    Cao, Y., Faloutsos, P., and Pighin, F. 2003. Unsupervised learning for speech motion editing. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 225–231. Google ScholarDigital Library
    Choe, B., Lee, H., and seok Ko, H. 2001. Performance-driven muscle-based facial animation. J. Vis. Comput. Anim. 12, 67–79.Google ScholarCross Ref
    Chuang, E. and Bregler, C. 2002. Performance driven facial animation using blendshape interpolation. Tech. rep., Department of Computer Science, Stanford University.Google Scholar
    Chuang, E. and Bregler, C. 2005. Mood swings: Expressive speech animation. ACM Trans. Graph. 24, 2, 331–347. Google ScholarDigital Library
    Deng, Z., Chiang, P.-Y., Fox, P., and Neumann, U. 2006. Animating blendshape faces by cross-mapping motion capture data. In Proceedings of the Symposium on Interactive 3D Graphics and Games. ACM, New York, 43–48. Google ScholarDigital Library
    Deng, Z. and Noh, J. 2007. Computer Facial Animation: A Survey. Springer, London.Google Scholar
    Ekman, P. and Friesen, W. 1977. Manual for the Facial Action Coding System. Consulting Psychologists Press, Palo Alto, CA.Google Scholar
    Havaldar, P. 2006. Performance driven facial animation. In ACM SIGGRAPH ’06 Course #30 Notes. Google ScholarDigital Library
    Hertzmann, A. 2004. Introduction to Bayesian learning. In ACM SIGGRAPH’04 Course Notes. ACM, New York, 22. Google ScholarDigital Library
    Joshi, P., Tien, W. C., Desbrun, M., and Pighin, F. 2003. Learning controls for blend shape based realistic facial animation. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 187–192. Google ScholarDigital Library
    Kovar, L. and Gleicher, M. 2003. Flexible automatic motion blending with registration curves. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 214–224. Google ScholarDigital Library
    Kraevoy, V., Sheffer, A., and Gotsman, C. 2003. Matchmaker: constructing constrained texture maps. ACM SIGGRAPH’03 Papers. ACM, 326–333. Google ScholarDigital Library
    Lau, M., Chai, J., Xu, Y.-Q., and Shum, H.-Y. 2009. Face poser: Interactive modeling of 3d facial expressions using facial priors. ACM Trans. Graph. 29, 1, 1–17. Google ScholarDigital Library
    Lee, J., Chai, J., Reitsma, P. S. A., Hodgins, J. K., and Pollard, N. S. 2002. Interactive control of avatars animated with human motion data. ACM Trans. Graph. 21, 3, 491–500. Google ScholarDigital Library
    Levy, B., Petitjean, S., Ray, N., and Maillo t, J. 2002. Least squares conformal maps for automatic texture atlas generation. In ACM SIGGRAPH Conference Proceedings. Google ScholarDigital Library
    Lewis, J. and Anjyo, K. 2010. Direct manipulation blendshapes. IEEE Comput. Graph. Appl. 30, 4, 42–50. Google ScholarDigital Library
    Li, H., Sumner, R. W., and Pauly, M. 2008. Global correspondence optimization for non-rigid registration of depth scans. Comput. Graph. Forum 27, 5. Google ScholarDigital Library
    Li, H., Weise, T., and Pauly, M. 2010. Example-based facial rigging. ACM Trans. Graph. 29, 4, 1–6. Google ScholarDigital Library
    Li, Q. and Deng, Z. 2008. Orthogonal blendshape based editing system for facial motion capture data. IEEE Comput. Graph. Appl., 76–82. Google ScholarDigital Library
    Luamanuvae, J. 2010. Personal communication (Weta Digital).Google Scholar
    Ma, W.-C., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., and Debevec, P. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graph. 27, 5, 1–10. Google ScholarDigital Library
    Ma, X., Le, B. H., and Deng, Z. 2009. Style learning and transferring for facial animation editing. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, New York, 123–132. Google ScholarDigital Library
    Noh, J. and Neumann, U. 2001. Expression cloning. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. ACM. 277–288. Google ScholarDigital Library
    Orvalho, V. C., Zacur, E., and Susin, A. 2008. Transferring the rig and animations from a character to different face models. Comput. Graph. Forum 27, 8, 1997–2012.Google ScholarCross Ref
    Parke, F. I. 1972. Computer generated animation of faces. In Proceedings of the ACM Annual Conference. ACM, New York, 451–457. Google ScholarDigital Library
    Parke, F. I. and Waters, K. 1996. Computer Facial Animation. A. K. Peters. Google ScholarDigital Library
    Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3, 313–318. Google ScholarDigital Library
    Pighin, F., Szeliski, R., and Salesin, D. H. 2002. Modeling and animating realistic faces from images. Int. J. Comput. Vis. 50, 2, 143–169. Google ScholarDigital Library
    Pyun, H., Kim, Y., Chae, W., Kang, H. W., and Shin, S. Y. 2003. An example-based approach for facial expression cloning. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 167–176. Google ScholarDigital Library
    Reveret, L. and Essa, I. 2001. Visual coding and tracking of speech related facial motion. Tech. rep., IEEE CVPR International Workshop on Cues in Communication.Google Scholar
    Sagar, M. and Grossman, R. 2006. Facial performance capture and expressive translation for King Kong. In SIGGRAPH Sketches. Google ScholarDigital Library
    Sumner, R. W. and Popovic, J. 2004. Deformation transfer for triangle meshes. ACM SIGGRAPH Papers. ACM. 399–405. Google ScholarDigital Library
    Sumner, R. W., Zwicker, M., Gotsman, C., and Popović, J. 2005. Mesh-based inverse kinematics. In ACM SIGGRAPH Papers. ACM, New York, 488–495. Google ScholarDigital Library
    Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Trans. Graph. 24, 3, 426–433. Google ScholarDigital Library
    Wahba, G. 1990. Spline Models for Observational Data. SIAM.Google Scholar
    Weise, T., Li, H., Van Gool, L., and Pauly, M. 2009. Face/off: Live facial puppetry. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, New York, 7–16. Google ScholarDigital Library
    Williams, L. 1990. Performance-driven facial animation. In Proceedings of the 17th Annual Conference on Computer Graphics and Interactive Techniques. ACM, New York, 235–242. Google ScholarDigital Library
    Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High resolution capture for modeling and animation. ACM SIGGRAPH Papers. ACM, New York, 548–558. Google ScholarDigital Library

ACM Digital Library Publication: