“Real-time facial animation with image-based dynamic avatars” by Cao, Wu, Weng, Shao and Zhou
Conference:
Type:
Title:
- Real-time facial animation with image-based dynamic avatars
Session/Category Title: FACES & PORTRAITS
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We present a novel image-based representation for dynamic 3D avatars, which allows effective handling of various hairstyles and headwear, and can generate expressive facial animations with fine-scale details in real-time. We develop algorithms for creating an image-based avatar from a set of sparsely captured images of a user, using an off-the-shelf web camera at home. An optimization method is proposed to construct a topologically consistent morphable model that approximates the dynamic hair geometry in the captured images. We also design a real-time algorithm for synthesizing novel views of an image-based avatar, so that the avatar follows the facial motions of an arbitrary actor. Compelling results from our pipeline are demonstrated on a variety of cases.
References:
1. Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital Emily project: photoreal facial modeling and animation. In ACM SIGGRAPH 2009 Courses, 12:1–12:15. Google ScholarDigital Library
2. Alexander, O., Fyffe, G., Busch, J., Yu, X., Ichikari, R., Jones, A., Debevec, P., Jimenez, J., Danvoye, E., Antionazzi, B., Eheler, M., Kysela, Z., and von der Pahlen, J. 2013. Digital Ira: creating a real-time photoreal digital actor. In ACM SIGGRAPH 2013 Posters. Google ScholarDigital Library
3. Amberg, B., Blake, A., Fitzgibbon, A., Romdhani, S., and Vetter, T. 2007. Reconstructing high quality face-surfaces using model based stereo. In Proceedings of ICCV, 1–8.Google Scholar
4. Baltrušaitis, T., Robinson, P., and Morency, L.-P. 2012. 3D constrained local model for rigid and non-rigid facial tracking. In Proceedings of IEEE CVPR, 2610–2617. Google ScholarDigital Library
5. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3 (July), 24:1–24:11. Google ScholarDigital Library
6. Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30, 4, 75:1–75:10. Google ScholarDigital Library
7. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of SIGGRAPH, 187–194. Google ScholarDigital Library
8. Bouaziz, S., Wang, Y., and Pauly, M. 2013. Online modeling for realtime facial animation. ACM Trans. Graph. 32, 4 (July), 40:1–40:10. Google ScholarDigital Library
9. Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 4, 41:1–41:10. Google ScholarDigital Library
10. Cao, C., Weng, Y., Lin, S., and Zhou, K. 2013. 3d shape regression for real-time facial animation. ACM Trans. Graph. 32, 4 (July), 41:1–41:10. Google ScholarDigital Library
11. Cao, C., Hou, Q., and Zhou, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans. Graph. 33, 4 (July), 43:1–43:10. Google ScholarDigital Library
12. Cao, C., Weng, Y., Zhou, S., Tong, Y., and Zhou, K. 2014. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics 20, 3 (Mar.), 413–425. Google ScholarDigital Library
13. Cao, C., Bradley, D., Zhou, K., and Beeler, T. 2015. Real-time high-fidelity facial performance capture. ACM Trans. Graph. 34, 4 (July), 46:1–46:9. Google ScholarDigital Library
14. Casas, D., Alexander, O., Feng, A. W., Fyffe, G., Ichikari, R., Debevec, P., Wang, R., Suma, E., and Shapiro, A. 2015. Rapid photorealistic blendshapes from commodity rgb-d sensors. In Proceedings of the 19th Symposium on Interactive 3D Graphics and Games, i3D ’15, 134–134. Google ScholarDigital Library
15. Chai, J.-X., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3d facial animation. In Symp. Comp. Anim., 193–206. Google ScholarDigital Library
16. Chai, M., Wang, L., Weng, Y., Yu, Y., Guo, B., and Zhou, K. 2012. Single-view hair modeling for portrait manipulation. ACM Trans. Graph. 31, 4 (July), 116:1–116:8. Google ScholarDigital Library
17. Chai, M., Wang, L., Weng, Y., Jin, X., and Zhou, K. 2013. Dynamic hair manipulation in images and videos. ACM Trans. Graph. 32, 4 (July), 75:1–75:8. Google ScholarDigital Library
18. Chai, M., Zheng, C., and Zhou, K. 2014. A reduced model for interactive hairs. ACM Trans. Graph. 33, 4 (July), 1–11. Google ScholarDigital Library
19. Chai, M., Luo, L., Sunkavalli, K., Carr, N., Hadap, S., and Zhou, K. 2015. High-quality hair modeling from a single portrait photo. ACM Trans. Graph. 34, 6, 204. Google ScholarDigital Library
20. DeCarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. Journal of Computer Vision 38, 2, 99–127. Google ScholarDigital Library
21. Desbrun, M., Meyer, M., Schroder, P., and Barr, A. H. 1999. Implicit fairing of irregular meshes using diffusion and curvature flow. In Proceedings of ACM SIGGRAPH, 317–324. Google ScholarDigital Library
22. Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads: using input from video. In Computer Animation, 68–79. Google ScholarDigital Library
23. Garrido, P., Valgaerts, L., Wu, C., and Theobalt, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. 32, 6, 158. Google ScholarDigital Library
24. Garrido, P., Zollhofer, M., Casas, D., Valgaerts, L., Varanasi, K., Perez, P., and Theobalt, C. 2016. Reconstruction of personalized 3d face rigs from monocular video. ACM Trans. Graph. to appear. Google ScholarDigital Library
25. Hu, L., Ma, C., Luo, L., and Li, H. 2014. Robust hair capture using simulated examples. ACM Trans. Graph. 33, 4 (July), 126:1–126:10. Google ScholarDigital Library
26. Hu, L., Ma, C., Luo, L., Wei, L.-Y., and Li, H. 2014. Capturing braided hairstyles. ACM Trans. Graph. 33, 6 (Nov.), 225:1–225:9. Google ScholarDigital Library
27. Hu, L., Ma, C., Luo, L., and Li, H. 2015. Single-view hair modeling using a hairstyle database. ACM Trans. Graph. 34, 4 (July), 125:1–125:9. Google ScholarDigital Library
28. Huang, J., Shi, X., Liu, X., Zhou, K., Wei, L.-Y., Teng, S.-H., Bao, H., Guo, B., and Shum, H.-Y. 2006. Subspace gradient domain mesh deformation. ACM Trans. Graph. 25, 3 (July), 1126–1134. Google ScholarDigital Library
29. Huang, H., Chai, J., Tong, X., and Wu, H.-T. 2011. Leveraging motion capture and 3d scanning for high-fidelity facial performance acquisition. ACM Trans. Graph. 30, 4, 74:1–74:10. Google ScholarDigital Library
30. Ichim, A. E., Bouaziz, S., and Pauly, M. 2015. Dynamic 3d avatar creation from hand-held video input. ACM Trans. Graph. 34, 4 (July), 45:1–45:14. Google ScholarDigital Library
31. Jancosek, M., and Pajdla, T. 2011. Multi-view reconstruction preserving weakly-supported surfaces. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, 3121–3128. Google ScholarDigital Library
32. Jimenez, J., Scully, T., Barbosa, N., Donner, C., Alvarez, X., Vieira, T., Matts, P., Orvalho, V., Gutierrez, D., and Weyrich, T. 2010. A practical appearance model for dynamic facial color. ACM Trans. Graph. 29, 6 (Dec.), 141:1–141:10. Google ScholarDigital Library
33. Jimenez, J., Echevarria, J. I., Oat, C., and Gutierrez, D. 2011. GPU Pro 2. AK Peters Ltd., ch. Practical and Realistic Facial Wrinkles Animation.Google Scholar
34. Kazhdan, M., Bolitho, M., and Hoppe, H. 2006. Poisson surface reconstruction. In Proceedings of the Fourth Eurographics Symposium on Geometry Processing, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SGP ’06, 61–70. Google ScholarDigital Library
35. Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (Feb), 228–242. Google ScholarDigital Library
36. Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. ACM Trans. Graph. 23, 3 (Aug.), 303–308. Google ScholarDigital Library
37. Li, H., Weise, T., and Pauly, M. 2010. Example-based facial rigging. ACM Trans. Graph. 29, 4 (July), 32:1–32:6. Google ScholarDigital Library
38. Li, H., Yu, J., Ye, Y., and Bregler, C. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. 32, 4 (July), 42:1–42:10. Google ScholarDigital Library
39. Li, H., Trutoiu, L., Olszewski, K., Wei, L., Trutna, T., Hsieh, P.-L., Nicholls, A., and Ma, C. 2015. Facial performance sensing head-mounted display. ACM Trans. Graph. 34, 4, 47. Google ScholarDigital Library
40. Liu, Y., Xu, F., Chai, J., Tong, X., Wang, L., and Huo, Q. 2015. Video-audio driven real-time facial animationy. ACM Trans. Graph. 34, 6, 182. Google ScholarDigital Library
41. Luo, L., Li, H., and Rusinkiewicz, S. 2013. Structure-aware hair capture. ACM Trans. Graph. 32, 4, 76. Google ScholarDigital Library
42. McMillan, L. 1997. An Image-Based Approach to Three-Dimensional Computer Graphics. PhD thesis, University of North Carolina at Chapel Hill. Google ScholarDigital Library
43. Nagano, K., Fyffe, G., Alexander, O., Barbiç, J., Li, H., Ghosh, A., and Debevec, P. 2015. Skin microstructure deformation with displacement map convolution. ACM Trans. Graph. 34, 4 (July), 109:1–109:10. Google ScholarDigital Library
44. Paris, S., Chang, W., Kozhushnyan, O. I., Jarosz, W., Matusik, W., Zwicker, M., and Durand, F. 2008. Hair photobooth: Geometric and photometric acquisition of real hairstyles. ACM Trans. Graph. 27, 3 (Aug.), 30:1–30:9. Google ScholarDigital Library
45. Pighin, F., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3d model-based tracking. In Int. Conf. Computer Vision, 143–150.Google Scholar
46. Saragih, J., Lucey, S., and Cohn, J. 2011. Real-time avatar animation from a single image. In AFGR, 213–220.Google Scholar
47. Seitz, S. M., and Dyer, C. R. 1996. View morphing. In Proceedings of ACM SIGGRAPH, ACM, New York, NY, USA, SIGGRAPH ’96, 21–30. Google ScholarDigital Library
48. Shi, F., Wu, H.-T., Tong, X., and Chai, J. 2014. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Trans. Graph. 33, 6, 222. Google ScholarDigital Library
49. Shum, H.-Y., Chan, S.-C., and Kang, S. B. 2007. Image-Based Rendering. Springer. Google ScholarDigital Library
50. Stich, T., Linz, C., Albuquerque, G., and Magnor, M. 2008. View and time interpolation in image space. In Computer Graphics Forum, vol. 27, Wiley Online Library, 1781–1787.Google Scholar
51. Thies, J., Zollhöfer, M., Niessner, M., Valgaerts, L., Stamminger, M., and Theobalt, C. 2015. Real-time expression transfer for facial reenactment. ACM Trans. Graph. 34, 6, 183. Google ScholarDigital Library
52. Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graph. 31, 6, 187. Google ScholarDigital Library
53. Venkataramana, K., Lodhaa, S., and Raghavan, R. 2005. A kinematic-variational model for animating skin with wrinkles. Computer & Graphics 29, 5 (Oct), 756–770. Google ScholarDigital Library
54. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Trans. Graph. 24, 3 (July), 426–433. Google ScholarDigital Library
55. Wei, Y., Ofek, E., Quan, L., and Shum, H.-Y. 2005. Modeling hair from multiple views. ACM Trans. Graph. 24, 3 (July), 816–820. Google ScholarDigital Library
56. Weise, T., Li, H., Gool, L. V., and Pauly, M. 2009. Face/off: Live facial puppetry. In Symp. Computer Animation, 7–16. Google ScholarDigital Library
57. Weise, T., Bouaziz, S., Li, H., and Pauly, M. 2011. Realtime performance-based facial animation. ACM Trans. Graph. 30, 4 (July), 77:1–77:10. Google ScholarDigital Library
58. Xu, F., Liu, Y., Stoll, C., Tompkin, J., Bharaj, G., Dai, Q., Seidel, H.-P., Kautz, J., and Theobalt, C. 2011. Video-based characters: Creating new human performances from a multi-view video database. In ACM SIGGRAPH 2011 Papers, SIGGRAPH ’11, 32:1–32:10. Google ScholarDigital Library
59. Xu, Z., Wu, H.-T., Wang, L., Zheng, C., Tong, X., and Qi, Y. 2014. Dynamic hair capture using spacetime optimization. ACM Trans. Graph. 33, 6 (Nov.), 224:1–224:11. Google ScholarDigital Library
60. Yang, F., Shechtman, E., Wang, J., Bourdev, L., and Metaxas, D. 2012. Face morphing using 3d-aware appearance optimization. In Proceedings of Graphics Interface 2012, GI ’12, 93–99. Google ScholarDigital Library
61. Zanella, V., Vargas, H., and Rosas, L. V. 2007. Active shape models and evolution strategies to automatic face morphing. In Proceedings of the 8th International Conference on Adaptive and Natural Computing Algorithms, Part II, Springer-Verlag, Berlin, Heidelberg, ICANNGA ’07, 564–571. Google ScholarDigital Library
62. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Trans. Graph. 23, 3, 548–558. Google ScholarDigital Library
