“Controllable high-fidelity facial performance transfer” by Xu, Liu, Chai and Tong

  • ©Feng Xu, Yilong Liu, Jinxiang Chai, and Xin Tong




    Controllable high-fidelity facial performance transfer

Session/Category Title: Faces




    Recent technological advances in facial capture have made it possible to acquire high-fidelity 3D facial performance data with stunningly high spatial-temporal resolution. Current methods for facial expression transfer, however, are often limited to large-scale facial deformation. This paper introduces a novel facial expression transfer and editing technique for high-fidelity facial performance data. The key idea of our approach is to decompose high-fidelity facial performances into high-level facial feature lines, large-scale facial deformation and fine-scale motion details and transfer them appropriately to reconstruct the retargeted facial animation in an efficient optimization framework. The system also allows the user to quickly modify and control the retargeted facial sequences in the spatial-temporal domain. We demonstrate the power of our approach by transferring and editing high-fidelity facial animation data from high-resolution source models to a wide range of target models, including both human faces and non-human faces such as “monster” and “dog”.


    1. Akhter, I., Simon, T., Khan, S., Matthews, I., and Sheikh, Y. 2012. Bilinear spatiotemporal basis models. ACM Trans. Graph. 31, 2 (Apr.), 17:1–17:12. Google ScholarDigital Library
    2. Baran, I., Vlasic, D., Grinspun, E., and Popović, J. 2009. Semantic deformation transfer. ACM Trans. Graph. 28, 3 (July), 36:1–36:6. Google ScholarDigital Library
    3. Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30, 4, 75:1–75:10. Google ScholarDigital Library
    4. Bermano, A. H., Bradley, D., Beeler, T., Zund, F., Nowrouzezahrai, D., Baran, I., Sorkine-Hornung, O., Pfister, H., Sumner, R. W., Bickel, B., and Gross, M. 2014. Facial performance enhancement using dynamic shape space analysis. ACM Trans. Graph. 33, 2 (Apr.), 13:1–13:12. Google ScholarDigital Library
    5. Bhat, K. S., Goldenthal, R., Ye, Y., Mallet, R., and Koperwas, M. 2013. High fidelity facial animation capture and retargeting with contours. In Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation, ACM, New York, NY, USA, SCA ’13, 7–14. Google ScholarDigital Library
    6. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graph. 26, 3 (July). Google ScholarDigital Library
    7. Bickel, B., Lang, M., Botsch, M., Otaduy, M. A., and Gross, M. 2008. Pose-space animation and transfer of facial details. In Proceedings of the 2008 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SCA ’08, 57–66. Google ScholarDigital Library
    8. Bouaziz, S., Wang, Y., and Pauly, M. 2013. Online modeling for realtime facial animation. ACM Trans. Graph. 32, 4 (July), 40:1–40:10. Google ScholarDigital Library
    9. Buck, I., Finkelstein, A., Jacobs, C., Klein, A., Salesin, D., Seims, J., Szeliski, R., and Toyama, K. 2000. Performance-driven hand-drawn animation. In Proceedings of NPAR 2000. Google ScholarDigital Library
    10. Cao, C., Weng, Y., Lin, S., and Zhou, K. 2013. 3d shape regression for real-time facial animation. ACM Trans. Graph. 32, 4 (July), 41:1–41:10. Google ScholarDigital Library
    11. Chai, J., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3D facial animation. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 193–206. Google ScholarDigital Library
    12. Chuang, E., 2004. Analysis, synthesis, and retargeting of facial expressions. PhD thesis, Stanford University. Google ScholarDigital Library
    13. Havaldar, P. 2006. Performance driven facial animation. ACM SIGGRAPH Course Notes.Google Scholar
    14. Huang, H., Chai, J., Tong, X., and Wu, H.-T. 2011. Leveraging motion capture and 3d scanning for high-fidelity facial performance acquisition. ACM Trans. Graph. 30, 4, 74:1–74:10. Google ScholarDigital Library
    15. Image-Metrics. Live driver (face tracking). http://www.image-metrics.com.Google Scholar
    16. James, D. L., and Twigg, C. D. 2005. Skinning mesh animations. ACM Transactions on Graphics 24, 3, 399–407. Google ScholarDigital Library
    17. Joshi, P., Tien, W. C., Desbrun, M., and Pighin, F. 2003. Learning controls for blendshape based realistic facial animation. In Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Aimation. 187–192. Google ScholarDigital Library
    18. Kholgade, N., Matthews, I., and Sheikh, Y. 2011. Content retargeting using parameter-parallel facial layers. In Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, ACM, New York, NY, USA, SCA ’11, 195–204. Google ScholarDigital Library
    19. Lau, M., Chai, J., Xu, Y.-Q., and Shum, H.-Y. 2007. Face poser: interactive facial modeling using model priors. In Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Computer animation. Google ScholarDigital Library
    20. Lau, M., Chai, J., Xu, Y.-Q., and Shum, H. 2009. Face poser: Interactive modeling of 3d facial expressions using facial priors. ACM Transactions on Graphics 29, 1, 3:1–3:17. Google ScholarDigital Library
    21. Lewis, J., and Anjyo, K. 2010. Direct manipulation blendshapes. IEEE Computer Graphics and Applications 30, 4, 42–50. Google ScholarDigital Library
    22. Li, Q., and Deng, Z. 2008. Orthogonal-blendshape-based editing system for facial motion capture data. IEEE Comput. Graph. Appl. 28, 6 (Nov.), 76–82. Google ScholarDigital Library
    23. Li, H., Weise, T., and Pauly, M. 2010. Example-based facial rigging. ACM Transactions on Graphics 29, 4, 32:1–32:6. Google ScholarDigital Library
    24. Li, H., Yu, J., Ye, Y., and Bregler, C. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. 32, 4 (July), 42:1–42:10. Google ScholarDigital Library
    25. Meyer, M., and Anderson, J. 2007. Key point subspace acceleration and soft caching. In ACM Transactions on Graphics. 26(3): Article No. 74. Google ScholarDigital Library
    26. Noh, J., and Neumann, U. 2001. Expression cloning. In Proceedings of ACM SIGGRAPH 2001. 277–288. Google ScholarDigital Library
    27. Pighin, F., and Lewis, J. 2005. Digital face cloning. ACM SIGGRAPH Course Notes. Google ScholarDigital Library
    28. Seol, Y., Seo, J., Kim, P. H., Lewis, J. P., and Noh, J. 2011. Artist friendly facial animation retargeting. In Proceedings of the 2011 SIGGRAPH Asia Conference, ACM, New York, NY, USA, SA ’11, 162:1–162:10. Google ScholarDigital Library
    29. Seol, Y., Lewis, J., Seo, J., Choi, B., Anjyo, K., and Noh, J. 2012. Spacetime expression cloning for blendshapes. ACM Trans. Graph. 31, 2 (Apr.), 14:1–14:12. Google ScholarDigital Library
    30. Sorkine, O., Cohen-Or, D., Lipman, Y., Alexa, M., Rössl, C., and Seidel, H. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, ACM, 175–184. Google ScholarDigital Library
    31. Sumner, R. W., and Popović, J. 2004. Deformation transfer for triangle meshes. ACM Trans. Graph. 23, 3 (Aug.), 399–405. Google ScholarDigital Library
    32. Taubin, G. 1995. A signal processing approach to fair surface design. In Proceedings of the 22Nd Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH ’95, 351–358. Google ScholarDigital Library
    33. Tena, J. R., De la Torre, F., and Matthews, I. 2011. Interactive region-based linear 3d face models. ACM Trans. Graph. 30, 4 (July), 76:1–76:10. Google ScholarDigital Library
    34. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Transactions on Graphics 24, 3, 426–433. Google ScholarDigital Library
    35. Weise, T., Li, H., Van Gool, L., and Pauly, M. 2009. Face/off: live facial puppetry. In Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, ACM, New York, NY, USA, SCA ’09, 7–16. Google ScholarDigital Library
    36. Zhang, L., Snavely, N., Curless, B., and Seitz, S. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3, 548–558. Google ScholarDigital Library

ACM Digital Library Publication:

Overview Page: