“Realtime facial animation with on-the-fly correctives” by Li, Yu, Ye and Bregler

  • ©Hao Li, Jihun Yu, Yuting Ye, and Christoph (Chris) Bregler




    Realtime facial animation with on-the-fly correctives


Session Title: Faces & Hands



    We introduce a real-time and calibration-free facial performance capture framework based on a sensor with video and depth input. In this framework, we develop an adaptive PCA model using shape correctives that adjust on-the-fly to the actor’s expressions through incremental PCA-based learning. Since the fitting of the adaptive model progressively improves during the performance, we do not require an extra capture or training session to build this model. As a result, the system is highly deployable and easy to use: it can faithfully track any individual, starting from just a single face scan of the subject in a neutral pose. Like many real-time methods, we use a linear subspace to cope with incomplete input data and fast motion. To boost the training of our tracking model with reliable samples, we use a well-trained 2D facial feature tracker on the input video and an efficient mesh deformation algorithm to snap the result of the previous step to high frequency details in visible depth map regions. We show that the combination of dense depth maps and texture features around eyes and lips is essential in capturing natural dialogues and nuanced actor-specific emotions. We demonstrate that using an adaptive PCA model not only improves the fitting accuracy for tracking but also increases the expressiveness of the retargeted character.


    1. Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital Emily project: photo-real facial modeling and animation. In ACM SIGGRAPH 2009 Courses, 12:1–12:15. Google ScholarDigital Library
    2. Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30 (August), 75:1–75:10. Google ScholarDigital Library
    3. Bickel, B., Lang, M., Botsch, M., Otaduy, M. A., and Gross, M. 2008. Pose-space animation and transfer of facial details. In Proceedings of the 2008 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Google ScholarDigital Library
    4. Black, M. J., and Yacoob, Y. 1995. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In Proceedings of the Fifth International Conference on Computer Vision, IEEE Computer Society, Washington, DC, USA, ICCV ’95, 374–. Google ScholarDigital Library
    5. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of ACM Siggraph 99, ACM Press/Addison-Wesley Publishing Co., 187–194. Google ScholarDigital Library
    6. Borshukov, G., Piponi, D., Larsen, O., Lewis, J. P., and Tempelaar-Lietz, C. 2005. Universal capture – image-based facial animation for “the matrix reloaded”. In ACM SIGGRAPH 2005 Courses. Google ScholarDigital Library
    7. Botsch, M., and Sorkine, O. 2008. On linear variational surface deformation methods. IEEE Transactions on Visualization and Computer Graphics 14, 1 (Jan.), 213–230. Google ScholarDigital Library
    8. Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29 (July), 41:1–41:10. Google ScholarDigital Library
    9. Bregler, C., and Omohundro, S. 1994. Surface learning with applications to lipreading. Advances in neural information processing systems, 43–43.Google Scholar
    10. Bregler, C., Covell, M., and Slaney, M. 1997. Video rewrite: Driving visual speech with audio. In Proceedings of Computer graphics and interactive techniques. Google ScholarDigital Library
    11. Chai, J.-x., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3d facial animation. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SCA ’03, 193–206. Google ScholarDigital Library
    12. Chandrasekaran, S., Manjunath, B. S., Wang, Y.-F., Winkeler, J., and Zhang, H. 1997. An eigenspace update algorithm for image analysis. CVGIP: Graphical Model and Image Processing, 5, 321–332. Google ScholarDigital Library
    13. Chuang, E., and Bregler, C. 2002. Performance driven facial animation using blendshape interpolation. Tech. rep., Stanford University.Google Scholar
    14. Collins, R., Liu, Y., and Leordeanu, M. 2005. Online selection of discriminative tracking features. Pattern Analysis and Machine Intelligence, IEEE Transactions on 27, 10, 1631–1643. Google ScholarDigital Library
    15. Cootes, T. F., Edwards, G. J., and Taylor, C. J. 1998. Active appearance models. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Springer, 484–498. Google ScholarDigital Library
    16. Covell, M., and Bregler, C. 1996. Eigen-points. In Image Processing, 1996. Proceedings., International Conference on, vol. 3, IEEE, 471–474.Google Scholar
    17. Decarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. J. Comput. Vision 38, 2 (July), 99–127. Google ScholarDigital Library
    18. Dempster, A., Laird, N., and Rubin, D. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 1–38.Google ScholarCross Ref
    19. Ekman, P., and Friesen, W. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto.Google Scholar
    20. Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads using input from video. In Proceedings of the Computer Animation, IEEE Computer Society, CA ’96, 68–. Google ScholarDigital Library
    21. Furukawa, Y., and Ponce, J. 2009. Dense 3d motion capture for human faces. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20–25 June 2009, Miami, Florida, USA, IEEE, 1674–1681.Google Scholar
    22. Fyffe, G., Hawkins, T., Watts, C., Ma, W.-C., and Debevec, P. 2011. Comprehensive facial performance capture. In Eurographics 2011.Google Scholar
    23. Grabner, H., Leistner, C., and Bischof, H. 2008. Semi-supervised on-line boosting for robust tracking. In Proceedings of the 10th European Conference on Computer Vision: Part I, Springer-Verlag, Berlin, Heidelberg, ECCV ’08, 234–247. Google ScholarDigital Library
    24. Gu, M., and Eisenstat, S. C. 1993. A Stable and Fast Algorithm for Updating the Singular Value Decomposition. Tech. Rep. YALEU/DCS/RR-966, Yale University, New Haven, CT.Google Scholar
    25. Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Proceedings of SIGGRAPH ’98, ACM, 55–66. Google ScholarDigital Library
    26. ImageMetrics. 2012. Image metrics live driver SDK http://www.image-metrics.com/livedriver/.Google Scholar
    27. Kalal, Z., Matas, J., and Mikolajczyk, K. 2009. Online learning of robust object detectors during unstable tracking. In In International Conference on Computer Vision.Google Scholar
    28. Kirby, M., and Sirovich, L. 1990. Application of the karhunen-loeve procedure for the characterization of human faces. Pattern Analysis and Machine Intelligence, IEEE Transactions on 12, 1, 103–108. Google ScholarDigital Library
    29. Li, H., Roivainen, P., and Forcheimer, R. 1993. 3-d motion estimation in model-based facial image coding. IEEE Transactions on PAMI 15, 6, 545–555. Google ScholarDigital Library
    30. Li, H., Adams, B., Guibas, L. J., and Pauly, M. 2009. Robust single-view geometry and motion reconstruction. ACM Transactions on Graphics (Proceedings SIGGRAPH Asia 2009) 28, 5. Google ScholarDigital Library
    31. Li, H., Weise, T., and Pauly, M. 2010. Example-based facial rigging. ACM Transactions on Graphics (Proceedings SIGGRAPH 2010) 29, 3 (July). Google ScholarDigital Library
    32. Paysan, P., Knothe, R., Amberg, B., Romdhani, S., and Vetter, T. 2009. A 3d face model for pose and illumination invariant face recognition.Google Scholar
    33. Pearson, K. 1901. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2, 11, 559–572.Google ScholarCross Ref
    34. Pighin, F., and Lewis, J. P. 2006. Performance-driven facial animation. In ACM SIGGRAPH 2006 Courses, ACM, New York, NY, USA, SIGGRAPH ’06.Google Scholar
    35. Pighin, F. H., Szeliski, R., and Salesin, D. 1999. Resynthesizing Facial Animation through 3D Model-based Tracking. In Proc. 7th International Conference on Computer Vision, Kerkyra, Greece, 143–150.Google Scholar
    36. Roweis, S. 1998. EM algorithms for pca and spca. In in Advances in Neural Information Processing Systems, MIT Press, 626–632. Google ScholarDigital Library
    37. Rusinkiewicz, S., and Levoy, M. 2001. Efficient variants of the icp algorithm. In International Conference on 3-D Digital Imaging and Modeling.Google Scholar
    38. Rusinkiewicz, S., Hall-Holt, O., and Levoy, M. 2002. Real-time 3D model acquisition. ACM Transactions on Graphics (Proc. SIGGRAPH) 21, 3 (July), 438–446. Google ScholarDigital Library
    39. Saragih, J. M., Lucey, S., and Cohn, J. F. 2011. Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vision 91, 2 (Jan.), 200–215. Google ScholarDigital Library
    40. Skocaj, D., and Leonardis, A. 2003. Weighted and robust incremental method for subspace learning. In Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on, IEEE, 1494–1501. Google ScholarDigital Library
    41. Sugimoto, T., Fukushima, M., and Ibaraki, T. 1995. A parallel relaxation method for quadratic programming problems with interval constraints. Journal of Computational and Applied Mathematics 60, 12, 219–236. Google ScholarDigital Library
    42. Sumner, R. W., and Popović, J. 2004. Deformation transfer for triangle meshes. ACM Trans. Graph. 23, 3 (Aug.), 399–405. Google ScholarDigital Library
    43. Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. In ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2012), vol. 31. Google ScholarDigital Library
    44. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. In ACM SIGGRAPH 2005 Papers, ACM, New York, NY, USA, SIGGRAPH ’05, 426–433. Google ScholarDigital Library
    45. Weise, T., Li, H., Gool, L. V., and Pauly, M. 2009. Face/off: Live facial puppetry. In Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer animation. Google ScholarDigital Library
    46. Weise, T., Bouaziz, S., Li, H., and Pauly, M. 2011. Real-time performance-based facial animation. ACM Transactions on Graphics (Proceedings SIGGRAPH 2011) 30, 4 (July). Google ScholarDigital Library
    47. Welch, G., and Bishop, G. 1995. An introduction to the kalman filter. Tech. rep., Chapel Hill, NC, USA. Google ScholarDigital Library
    48. Williams, L. 1990. Performance-driven facial animation. SIGGRAPH Comput. Graph. 24, 4 (Sept.), 235–242. Google ScholarDigital Library
    49. Zhang, S., and Huang, P. 2004. High-resolution, real-time 3d shape acquisition. In Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04) Volume 3 – Volume 03, IEEE Computer Society, Washington, DC, USA, CVPRW ’04, 28–. Google ScholarDigital Library
    50. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High-resolution capture for modeling and animation. In ACM Annual Conference on Computer Graphics, 548–558. Google ScholarDigital Library

ACM Digital Library Publication: