“Real-time facial animation with image-based dynamic avatars” by Cao, Wu, Weng, Shao and Zhou

  • ©Chen Cao, Hongzhi Wu, Yanlin Weng, Tianjia Shao, and Kun Zhou




    Real-time facial animation with image-based dynamic avatars

Session/Category Title: FACES & PORTRAITS




    We present a novel image-based representation for dynamic 3D avatars, which allows effective handling of various hairstyles and headwear, and can generate expressive facial animations with fine-scale details in real-time. We develop algorithms for creating an image-based avatar from a set of sparsely captured images of a user, using an off-the-shelf web camera at home. An optimization method is proposed to construct a topologically consistent morphable model that approximates the dynamic hair geometry in the captured images. We also design a real-time algorithm for synthesizing novel views of an image-based avatar, so that the avatar follows the facial motions of an arbitrary actor. Compelling results from our pipeline are demonstrated on a variety of cases.


    1. Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital Emily project: photoreal facial modeling and animation. In ACM SIGGRAPH 2009 Courses, 12:1–12:15. Google ScholarDigital Library
    2. Alexander, O., Fyffe, G., Busch, J., Yu, X., Ichikari, R., Jones, A., Debevec, P., Jimenez, J., Danvoye, E., Antionazzi, B., Eheler, M., Kysela, Z., and von der Pahlen, J. 2013. Digital Ira: creating a real-time photoreal digital actor. In ACM SIGGRAPH 2013 Posters. Google ScholarDigital Library
    3. Amberg, B., Blake, A., Fitzgibbon, A., Romdhani, S., and Vetter, T. 2007. Reconstructing high quality face-surfaces using model based stereo. In Proceedings of ICCV, 1–8.Google Scholar
    4. Baltrušaitis, T., Robinson, P., and Morency, L.-P. 2012. 3D constrained local model for rigid and non-rigid facial tracking. In Proceedings of IEEE CVPR, 2610–2617. Google ScholarDigital Library
    5. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3 (July), 24:1–24:11. Google ScholarDigital Library
    6. Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30, 4, 75:1–75:10. Google ScholarDigital Library
    7. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of SIGGRAPH, 187–194. Google ScholarDigital Library
    8. Bouaziz, S., Wang, Y., and Pauly, M. 2013. Online modeling for realtime facial animation. ACM Trans. Graph. 32, 4 (July), 40:1–40:10. Google ScholarDigital Library
    9. Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 4, 41:1–41:10. Google ScholarDigital Library
    10. Cao, C., Weng, Y., Lin, S., and Zhou, K. 2013. 3d shape regression for real-time facial animation. ACM Trans. Graph. 32, 4 (July), 41:1–41:10. Google ScholarDigital Library
    11. Cao, C., Hou, Q., and Zhou, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans. Graph. 33, 4 (July), 43:1–43:10. Google ScholarDigital Library
    12. Cao, C., Weng, Y., Zhou, S., Tong, Y., and Zhou, K. 2014. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics 20, 3 (Mar.), 413–425. Google ScholarDigital Library
    13. Cao, C., Bradley, D., Zhou, K., and Beeler, T. 2015. Real-time high-fidelity facial performance capture. ACM Trans. Graph. 34, 4 (July), 46:1–46:9. Google ScholarDigital Library
    14. Casas, D., Alexander, O., Feng, A. W., Fyffe, G., Ichikari, R., Debevec, P., Wang, R., Suma, E., and Shapiro, A. 2015. Rapid photorealistic blendshapes from commodity rgb-d sensors. In Proceedings of the 19th Symposium on Interactive 3D Graphics and Games, i3D ’15, 134–134. Google ScholarDigital Library
    15. Chai, J.-X., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3d facial animation. In Symp. Comp. Anim., 193–206. Google ScholarDigital Library
    16. Chai, M., Wang, L., Weng, Y., Yu, Y., Guo, B., and Zhou, K. 2012. Single-view hair modeling for portrait manipulation. ACM Trans. Graph. 31, 4 (July), 116:1–116:8. Google ScholarDigital Library
    17. Chai, M., Wang, L., Weng, Y., Jin, X., and Zhou, K. 2013. Dynamic hair manipulation in images and videos. ACM Trans. Graph. 32, 4 (July), 75:1–75:8. Google ScholarDigital Library
    18. Chai, M., Zheng, C., and Zhou, K. 2014. A reduced model for interactive hairs. ACM Trans. Graph. 33, 4 (July), 1–11. Google ScholarDigital Library
    19. Chai, M., Luo, L., Sunkavalli, K., Carr, N., Hadap, S., and Zhou, K. 2015. High-quality hair modeling from a single portrait photo. ACM Trans. Graph. 34, 6, 204. Google ScholarDigital Library
    20. DeCarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. Journal of Computer Vision 38, 2, 99–127. Google ScholarDigital Library
    21. Desbrun, M., Meyer, M., Schroder, P., and Barr, A. H. 1999. Implicit fairing of irregular meshes using diffusion and curvature flow. In Proceedings of ACM SIGGRAPH, 317–324. Google ScholarDigital Library
    22. Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads: using input from video. In Computer Animation, 68–79. Google ScholarDigital Library
    23. Garrido, P., Valgaerts, L., Wu, C., and Theobalt, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. 32, 6, 158. Google ScholarDigital Library
    24. Garrido, P., Zollhofer, M., Casas, D., Valgaerts, L., Varanasi, K., Perez, P., and Theobalt, C. 2016. Reconstruction of personalized 3d face rigs from monocular video. ACM Trans. Graph. to appear. Google ScholarDigital Library
    25. Hu, L., Ma, C., Luo, L., and Li, H. 2014. Robust hair capture using simulated examples. ACM Trans. Graph. 33, 4 (July), 126:1–126:10. Google ScholarDigital Library
    26. Hu, L., Ma, C., Luo, L., Wei, L.-Y., and Li, H. 2014. Capturing braided hairstyles. ACM Trans. Graph. 33, 6 (Nov.), 225:1–225:9. Google ScholarDigital Library
    27. Hu, L., Ma, C., Luo, L., and Li, H. 2015. Single-view hair modeling using a hairstyle database. ACM Trans. Graph. 34, 4 (July), 125:1–125:9. Google ScholarDigital Library
    28. Huang, J., Shi, X., Liu, X., Zhou, K., Wei, L.-Y., Teng, S.-H., Bao, H., Guo, B., and Shum, H.-Y. 2006. Subspace gradient domain mesh deformation. ACM Trans. Graph. 25, 3 (July), 1126–1134. Google ScholarDigital Library
    29. Huang, H., Chai, J., Tong, X., and Wu, H.-T. 2011. Leveraging motion capture and 3d scanning for high-fidelity facial performance acquisition. ACM Trans. Graph. 30, 4, 74:1–74:10. Google ScholarDigital Library
    30. Ichim, A. E., Bouaziz, S., and Pauly, M. 2015. Dynamic 3d avatar creation from hand-held video input. ACM Trans. Graph. 34, 4 (July), 45:1–45:14. Google ScholarDigital Library
    31. Jancosek, M., and Pajdla, T. 2011. Multi-view reconstruction preserving weakly-supported surfaces. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, 3121–3128. Google ScholarDigital Library
    32. Jimenez, J., Scully, T., Barbosa, N., Donner, C., Alvarez, X., Vieira, T., Matts, P., Orvalho, V., Gutierrez, D., and Weyrich, T. 2010. A practical appearance model for dynamic facial color. ACM Trans. Graph. 29, 6 (Dec.), 141:1–141:10. Google ScholarDigital Library
    33. Jimenez, J., Echevarria, J. I., Oat, C., and Gutierrez, D. 2011. GPU Pro 2. AK Peters Ltd., ch. Practical and Realistic Facial Wrinkles Animation.Google Scholar
    34. Kazhdan, M., Bolitho, M., and Hoppe, H. 2006. Poisson surface reconstruction. In Proceedings of the Fourth Eurographics Symposium on Geometry Processing, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SGP ’06, 61–70. Google ScholarDigital Library
    35. Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (Feb), 228–242. Google ScholarDigital Library
    36. Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. ACM Trans. Graph. 23, 3 (Aug.), 303–308. Google ScholarDigital Library
    37. Li, H., Weise, T., and Pauly, M. 2010. Example-based facial rigging. ACM Trans. Graph. 29, 4 (July), 32:1–32:6. Google ScholarDigital Library
    38. Li, H., Yu, J., Ye, Y., and Bregler, C. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. 32, 4 (July), 42:1–42:10. Google ScholarDigital Library
    39. Li, H., Trutoiu, L., Olszewski, K., Wei, L., Trutna, T., Hsieh, P.-L., Nicholls, A., and Ma, C. 2015. Facial performance sensing head-mounted display. ACM Trans. Graph. 34, 4, 47. Google ScholarDigital Library
    40. Liu, Y., Xu, F., Chai, J., Tong, X., Wang, L., and Huo, Q. 2015. Video-audio driven real-time facial animationy. ACM Trans. Graph. 34, 6, 182. Google ScholarDigital Library
    41. Luo, L., Li, H., and Rusinkiewicz, S. 2013. Structure-aware hair capture. ACM Trans. Graph. 32, 4, 76. Google ScholarDigital Library
    42. McMillan, L. 1997. An Image-Based Approach to Three-Dimensional Computer Graphics. PhD thesis, University of North Carolina at Chapel Hill. Google ScholarDigital Library
    43. Nagano, K., Fyffe, G., Alexander, O., Barbiç, J., Li, H., Ghosh, A., and Debevec, P. 2015. Skin microstructure deformation with displacement map convolution. ACM Trans. Graph. 34, 4 (July), 109:1–109:10. Google ScholarDigital Library
    44. Paris, S., Chang, W., Kozhushnyan, O. I., Jarosz, W., Matusik, W., Zwicker, M., and Durand, F. 2008. Hair photobooth: Geometric and photometric acquisition of real hairstyles. ACM Trans. Graph. 27, 3 (Aug.), 30:1–30:9. Google ScholarDigital Library
    45. Pighin, F., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3d model-based tracking. In Int. Conf. Computer Vision, 143–150.Google Scholar
    46. Saragih, J., Lucey, S., and Cohn, J. 2011. Real-time avatar animation from a single image. In AFGR, 213–220.Google Scholar
    47. Seitz, S. M., and Dyer, C. R. 1996. View morphing. In Proceedings of ACM SIGGRAPH, ACM, New York, NY, USA, SIGGRAPH ’96, 21–30. Google ScholarDigital Library
    48. Shi, F., Wu, H.-T., Tong, X., and Chai, J. 2014. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Trans. Graph. 33, 6, 222. Google ScholarDigital Library
    49. Shum, H.-Y., Chan, S.-C., and Kang, S. B. 2007. Image-Based Rendering. Springer. Google ScholarDigital Library
    50. Stich, T., Linz, C., Albuquerque, G., and Magnor, M. 2008. View and time interpolation in image space. In Computer Graphics Forum, vol. 27, Wiley Online Library, 1781–1787.Google Scholar
    51. Thies, J., Zollhöfer, M., Niessner, M., Valgaerts, L., Stamminger, M., and Theobalt, C. 2015. Real-time expression transfer for facial reenactment. ACM Trans. Graph. 34, 6, 183. Google ScholarDigital Library
    52. Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graph. 31, 6, 187. Google ScholarDigital Library
    53. Venkataramana, K., Lodhaa, S., and Raghavan, R. 2005. A kinematic-variational model for animating skin with wrinkles. Computer & Graphics 29, 5 (Oct), 756–770. Google ScholarDigital Library
    54. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Trans. Graph. 24, 3 (July), 426–433. Google ScholarDigital Library
    55. Wei, Y., Ofek, E., Quan, L., and Shum, H.-Y. 2005. Modeling hair from multiple views. ACM Trans. Graph. 24, 3 (July), 816–820. Google ScholarDigital Library
    56. Weise, T., Li, H., Gool, L. V., and Pauly, M. 2009. Face/off: Live facial puppetry. In Symp. Computer Animation, 7–16. Google ScholarDigital Library
    57. Weise, T., Bouaziz, S., Li, H., and Pauly, M. 2011. Realtime performance-based facial animation. ACM Trans. Graph. 30, 4 (July), 77:1–77:10. Google ScholarDigital Library
    58. Xu, F., Liu, Y., Stoll, C., Tompkin, J., Bharaj, G., Dai, Q., Seidel, H.-P., Kautz, J., and Theobalt, C. 2011. Video-based characters: Creating new human performances from a multi-view video database. In ACM SIGGRAPH 2011 Papers, SIGGRAPH ’11, 32:1–32:10. Google ScholarDigital Library
    59. Xu, Z., Wu, H.-T., Wang, L., Zheng, C., Tong, X., and Qi, Y. 2014. Dynamic hair capture using spacetime optimization. ACM Trans. Graph. 33, 6 (Nov.), 224:1–224:11. Google ScholarDigital Library
    60. Yang, F., Shechtman, E., Wang, J., Bourdev, L., and Metaxas, D. 2012. Face morphing using 3d-aware appearance optimization. In Proceedings of Graphics Interface 2012, GI ’12, 93–99. Google ScholarDigital Library
    61. Zanella, V., Vargas, H., and Rosas, L. V. 2007. Active shape models and evolution strategies to automatic face morphing. In Proceedings of the 8th International Conference on Adaptive and Natural Computing Algorithms, Part II, Springer-Verlag, Berlin, Heidelberg, ICANNGA ’07, 564–571. Google ScholarDigital Library
    62. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Trans. Graph. 23, 3, 548–558. Google ScholarDigital Library

ACM Digital Library Publication: