“Facial performance synthesis using deformation-driven polynomial displacement maps”
Conference:
Type(s):
Title:
- Facial performance synthesis using deformation-driven polynomial displacement maps
Session/Category Title: Character animation II
Presenter(s)/Author(s):
Abstract:
We present a novel method for acquisition, modeling, compression, and synthesis of realistic facial deformations using polynomial displacement maps. Our method consists of an analysis phase where the relationship between motion capture markers and detailed facial geometry is inferred, and a synthesis phase where novel detailed animated facial geometry is driven solely by a sparse set of motion capture markers. For analysis, we record the actor wearing facial markers while performing a set of training expression clips. We capture real-time high-resolution facial deformations, including dynamic wrinkle and pore detail, using interleaved structured light 3D scanning and photometric stereo. Next, we compute displacements between a neutral mesh driven by the motion capture markers and the high-resolution captured expressions. These geometric displacements are stored in a polynomial displacement map which is parameterized according to the local deformations of the motion capture dots. For synthesis, we drive the polynomial displacement map with new motion capture data. This allows the recreation of large-scale muscle deformation, medium and fine wrinkles, and dynamic skin pore detail. Applications include the compression of existing performance data and the synthesis of new performances. Our technique is independent of the underlying geometry capture system and can be used to automatically generate high-frequency wrinkle and pore details on top of many existing facial animation systems.
References:
1. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Transactions on Graphics 26, 3 (July), 33: 1–33: 10. Google ScholarDigital Library
2. Bickel, B., Lang, M., Botsch, M., Otaduy, M., and Gross, M. 2008. Pose-space animation and transfer of facial details. In 2008 ACM SIGGRAPH / Eurographics Symposium on Computer Animation, ACM, New York, NY, USA, 57–66. Google ScholarDigital Library
3. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proc. of the 8th European Conference on Computer Vision, Springer-Verlag, Prague, Czech Republic.Google Scholar
4. Cerda, E., and Mahadevan, L. 2003. Geometry and physics of wrinkling. Physical Review Letters 90, 7 (February), 074302+.Google ScholarCross Ref
5. Davis, J., Nehab, D., Ramamoorthi, R., and Rusinkiewicz, S. 2005. Spacetime stereo: A unifying framework for depth from triangulation. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 2, 296–302. Google ScholarDigital Library
6. Golovinskiy, A., Matusik, W., Pfister, H., Rusinkiewicz, S., and Funkhouser, T. 2006. A statistical model for synthesis of detailed facial geometry. ACM Transactions on Graphics 25, 3 (July), 1025–1034. Google ScholarDigital Library
7. Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, 55–66. Google Scholar
8. Haro, A., Guenter, B., and Essa, I. 2001. Real-time, photorealistic, physically based rendering of fine scale human skin structure. In Rendering Techniques 2001: 12th Eurographics Workshop on Rendering, 53–62. Google Scholar
9. Hawkins, T., Wenger, A., Tchou, C., Gardner, A., Göransson, F., and Debevec, P. 2004. Animatable facial reflectance fields. In Rendering Techniques 2004: 15th Eurographics Workshop on Rendering, 309–320. Google Scholar
10. Jones, A., Gardner, A., Bolas, M., McDowall, I., and Debevec, P. 2006. Performance geometry capture for spatially varying relighting. In CVMP 2006.Google Scholar
11. Joshi, P., Tien, W. C., Desbrun, M., and Pighin, F. 2003. Learning controls for blend shape based realistic facial animation. In 2003 ACM SIGGRAPH / Eurographics Symposium on Computer Animation, 187–192. Google ScholarDigital Library
12. Lee, Y., Terzopoulos, D., and Waters, K. 1995. Realistic modeling for facial animation. Proceedings of SIGGRAPH 95, 55–62. Google Scholar
13. Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Rendering Techniques, 183–194. Google Scholar
14. Malzbender, T., Gelb, D., and Wolters, H. 2001. Polynomial texture maps. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 519–528. Google Scholar
15. Malzbender, T., Wilburn, B., Gelb, D., and Ambrisco, B. 2006. Surface enhancement using real-time photometric stereo and reflectance transformation. In Rendering Techniques 2006: 17th Eurographics Workshop on Rendering, 245–250. Google Scholar
16. Nehab, D., Rusinkiewicz, S., Davis, J., and Ramamoorthi, R. 2005. Efficiently combining positions and normals for precise 3d geometry. ACM Transactions on Graphics 24, 3 (Aug.), 536–543. Google ScholarDigital Library
17. Oat, C. 2007. Animated wrinkle maps. In SIGGRAPH 2007: ACM SIGGRAPH 2007 courses, ACM, New York, NY, USA, 33–37. Google Scholar
18. Parke, F. I. 1972. Computer generated animation of faces. In ACM’72: Proceedings of the ACM annual conference, ACM, New York, NY, USA, 451–457. Google Scholar
19. Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. H. 1998. Synthesizing realistic facial expressions from photographs. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, 75–84. Google Scholar
20. Rusinkiewicz, S., Hall-Holt, O., and Levoy, M. 2002. Real-time 3d model acquisition. ACM Transactions on Graphics 21, 3 (July), 438–446. Google ScholarDigital Library
21. Sánchez, M. A. 2006. Techniques for performance-based, realtime facial animation. PhD thesis, University of Sheffield.Google Scholar
22. Sifakis, E., Neverov, I., and Fedkiw, R. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics 24, 3 (Aug.), 417–425. Google ScholarDigital Library
23. Terzopoulos, D., and Waters, K. 1990. Physically-based facial modelling, analysis, and animation. Journal of Visualization and Computer Animation 1, 2, 73–80.Google ScholarCross Ref
24. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Transactions on Graphics 24, 3 (Aug.), 426–433. Google ScholarDigital Library
25. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., and Debevec, P. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics 24, 3 (Aug.), 756–764. Google ScholarDigital Library
26. Williams, L. 1990. Performance-driven facial animation. In Computer Graphics (Proceedings of SIGGRAPH 90), 235–242. Google Scholar
27. Zhang, S., and Huang, P. 2006. High-resolution, real-time three-dimensional shape measurement. Optical Engineering 45, 12.Google Scholar
28. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3 (Aug.), 548–558. Google ScholarDigital Library
