“Automatic determination of facial muscle activations from sparse motion capture marker data” by Sifakis, Neverov and Fedkiw

We built an anatomically accurate model of facial musculature, passive tissue and underlying skeletal structure using volumetric data acquired from a living male subject. The tissues are endowed with a highly nonlinear constitutive model including controllable anisotropic muscle activations based on fiber directions. Detailed models of this sort can be difficult to animate requiring complex coordinated stimulation of the underlying musculature. We propose a solution to this problem automatically determining muscle activations that track a sparse set of surface landmarks, e.g. acquired from motion capture marker data. Since the resulting animation is obtained via a three dimensional nonlinear finite element method, we obtain visually plausible and anatomically correct deformations with spatial and temporal coherence that provides robustness against outliers in the motion capture data. Moreover, the obtained muscle activations can be used in a robust simulation framework including contact and collision of the face with external objects.

1. Basu, S., Oliver, N., and Pentland, A. 1998. 3D lip shapes from video: a combined physical-statistical model. Speech Communication 26, 131–148. Google ScholarDigital Library
2. Basu, S., Oliver, N. and Pentland, A. 1998. 3D modeling and tracking of human lip motions. IEEE Computer Society, 337–343. Google ScholarDigital Library
3. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3D faces. In Proc. of ACM SIGGRAPH, ACM Press, 187–194. Google ScholarDigital Library
4. Blanz, V., and Vetter, T. 2003. Face recognition based on fitting a 3D morphable model. IEEE Trans. on Pattern Analysis and Machine Intelligence 29, 9, 1063–1074. Google ScholarDigital Library
5. Blanz, V., Basso, C., Poggio, T., and Vetter, T. 2003. Reanimating faces in images and video. In Proc. of Eurographics, vol. 22.Google Scholar
6. Blanz, V., Scherbaum, K., Vetter, T., and Seidel, H. P. 2004. Exchanging faces in images. In Proc. of Eurographics, vol. 23.Google Scholar
7. Borshukov, G., Piponi, D., Larsen, O., Lewis, J. P., and Tempelaar-Lietz, C. 2003. Universal Capture – image-based facial animation for “The Matrix Reloaded”. In ACM SIGGRAPH 2003 Sketches & Applications, ACM Press, 1–1. Google ScholarDigital Library
8. Brand, M. 1999. Voice puppetry. In Proc. of ACM SIGGRAPH, 21–28. Google ScholarDigital Library
9. Bregler, C., Covell, M., and Slaney, M. 1997. Video Rewrite: driving visual speech with audio. In Proc. of ACM SIGGRAPH, 353–360. Google ScholarDigital Library
10. Byun, M., and Badler, N. I. 2002. FacEMOTE: Qualitative parametric modifiers for facial animations. In Proc. of ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., ACM Press, 65–71. Google ScholarDigital Library
11. Cao, Y., Faloutsos, P., and Pighin, F. 2003. Unsupervised learning for speech motion editing. In Proc. of the ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 225–231. Google ScholarDigital Library
12. Cao, Y., Faloutsos, P., Kohler, E., and Pighin, F. 2004. Real-time speech motion synthesis from recorded motions. In Proc. of 2003 ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 347–355. Google ScholarDigital Library
13. Cassell, J., Pelachaud, C., Badler, N., Steedman, M., Achorn, B., Becket, T., Doubille, B., Prevost, S., and Stone, M. 1994. Animated conversation: Rule-based generation of facial expression, gesture and spoken intonation for multiple conversational agents. In Proc. of ACM SIGGRAPH, ACM Press, 413–420. Google ScholarDigital Library
14. Cassell, J., Vilhjálmsson, H. H., and Bickmore, T. 2001. BEAT: the Behavior Expression Animation Toolkit. In Proc. of ACM SIGGRAPH, 477–486. Google ScholarDigital Library
15. Chai, J., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3D facial animation. In Proc. of ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 193–206. Google ScholarDigital Library
16. Choe, B., and Ko, H.-S. 2001. Analysis and synthesis of facial expressions with hand-generated muscle actuation basis. In Proc. of Comput. Anim., 12–19. Google ScholarDigital Library
17. Choe, B., Lee, H., and Ko, H.-S. 2001. Performance-driven muscle-based facial animation. J. Vis. and Comput. Anim. 12, 67–79.Google ScholarCross Ref
18. DeCarlo, D., Metaxas, D., and Stone, M. 1998. An anthropometric face model using variational techniques. In Proc. of ACM SIGGRAPH, ACM Press, 67–74. Google ScholarDigital Library
19. Ekman, P., and Friesen, W. V. 1978. Facial Action Coding System. Consulting Psychologist Press, Palo Alto.Google Scholar
20. Essa, I., and Pentland, A. 1997. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. on Pattern Analysis and Machine Intelligence 19, 7, 757-763. Google ScholarDigital Library
21. Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads using input from video. In Proc. of Computer Animation, IEEE Computer Society, 68–79. Google ScholarDigital Library
22. Ezzat, T., Geiger, G., and Poggio, T. 2002. Trainable videorealistic speech animation. In ACM Transactions on Graphics, ACM Press, vol. 21, 388–398. Google ScholarDigital Library
23. Gill, P. E., Murray, W., and Wright, M. H. 1981. Practical Optimization. Academic Press, San Diego, USA.Google Scholar
24. Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Proc. ACM SIGGRAPH. ACM Press, 55–66. Google ScholarDigital Library
25. Heidelberger, B., Teschner, M., Keiser, R., Müller, M., and Gross, M. 2004. Consistent penetration depth estimation for deformable collision response. In Proc. of Vision, Modeling, Visualization (VMV), 339–346.Google Scholar
26. Joshi, P., Tien, W. C., Desbrun, M., and Pighin, F. 2003. Learning controls for blend shape based realistic facial animation. In Proc. ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 365–373. Google ScholarDigital Library
27. Kahler, K., Haber, J., and Seidel, H.-P. 2001. Geometry-based muscle modeling for facial animation. In Proc. of Graphics Interface, 37–46. Google ScholarDigital Library
28. Kahler, K., Haber, J., Yamauchi, H., and Seidel, H.-P. 2002. Head shop: Generating animated head models with anatomical structure. In Proc. of ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 55–63. Google ScholarDigital Library
29. Kahler, K., Haber, J., and Seidel, H.-P. 2003. Reanimating the dead: Reconstruction of expressive faces from skull data. In ACM Trans. on Graphics, vol. 22, 554–561. Google ScholarDigital Library
30. Kalra, P., Mangili, A., Magnetat-Thalmann, N., and Thalmann, D. 1992. Simulation of facial muscle actions based on rational free form deformations. In Proc. of Eurographics, 59–69.Google Scholar
31. Keeve, E., Girod, S., Pfeifle, P., and Girod, B. 1996. Anatomy-based facial tissue modeling using the finite element method. In Proc. of Visualization, 21–28. Google ScholarDigital Library
32. Koch, R. M., Gross, M. H., Carls, F. R., von Buren, D. F., Fankhauser, G., and Parish, Y. I. H. 1996. Simulating facial surgery using finite element models. Computer Graphics 30, Annual Conference Series, 421–428. Google ScholarDigital Library
33. Koch, R., Gross, M., and Bosshard, A. 1998. Emotion editing using finite elements. Proceedings of Eurographics 1998 17, 3.Google Scholar
34. Kshirsagar, S., and Magnenat-Thalmann, N. 2003. Visyllable based speech animation. In Proc. of Eurographics, vol. 22.Google Scholar
35. Lee, Y., Terzopoulos, D., and Waters, K. 1995. Realistic modeling for facial animation. Comput. Graph. (SIGGRAPH Proc.), 55–62. Google ScholarDigital Library
36. Lee, S. P., Badler, J. B., and Badler, N. I. 2002. Eyes alive. In Proc. of ACM SIGGRAPH, ACM Press, 637–644. Google ScholarDigital Library
37. Magnenat-Thalmann, N., Primeau, E., and Thalmann, D. 1988. Abstract muscle action procedures for human face animation. The Visual Computer 3, 5, 290–297.Google ScholarCross Ref
38. Morishima, S., Ishikawa, T., and Terzopoulos, D. 1998. Facial muscle parameter decision from 2D frontal image. In Proc. of the Int. Conf. on Pattern Recognition, vol. 1, 160–162. Google ScholarDigital Library
39. Moving Picture Experts Group, 1998. Information technology – coding of audio-visual objects part 2: Visual. Final draft of international standard ISO/IEC JTC1/SC29/WG11 N2501 14496–2.Google Scholar
40. Na, K., and Jung, M. 2004. Hierarchical retargetting of fine facial motions. In Proc. of Eurographics, vol. 23.Google Scholar
41. Noh, J., and Neumann, U. 2001. Expression cloning. In Proc. of ACM SIGGRAPH, ACM Press, E. Fiume, Ed., 277–288. Google ScholarDigital Library
42. Parke, F. I., and Waters, K. 1996. Computer Facial Animation. AK Peters, Ltd. Google ScholarDigital Library
43. Parke, F. I. 1972. Computer generated animation of faces. In Proc. of ACM Conference, ACM Press, 451–457. Google ScholarDigital Library
44. Pieper, S., Rosen, J., and Zeltzer, D. 1992. Interactive graphics for plastic surgery: A task-level analysis and implementation. In Proc. of Symp. on Interactive 3D graphics, ACM Press, 127–134. Google ScholarDigital Library
45. Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. H. 1998. Synthesizing realistic facial expressions from photographs. In Proc. of ACM SIGGRAPH, ACM Press, 75–84. Google ScholarDigital Library
46. Pighin, F., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3D model-based tracking. In Proc. of Int. Conf. on Comput. Vision, 143–150.Google Scholar
47. Platt, S. M., and Badler, N. I. 1981. Animating facial expressions. Comput. Graph. (SIGGRAPH Proc.), 245–252. Google ScholarDigital Library
48. Pyun, H., Kim, Y., Chae, W., Kang, H. W., and Shin, S. Y. 2003. An example-based approach for facial expression cloning. In Proc. of ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 167–176. Google ScholarDigital Library
49. Roth, S. H., Gross, M., Turello, M. H., and Carls, S. 1998. A Bernstein-Bézier based approach to soft tissue simulation. In Proc. of Eurographics, vol. 17, 285–294.Google ScholarCross Ref
50. Sumner, R., and Popović, J. 2004. Deformation transfer for triangle meshes. In Proc. of ACM SIGGRAPH, vol. 23, 32–39. Google ScholarDigital Library
51. Teran, J., Blemker, S., Ng, V., and Fedkiw, R. 2003. Finite volume methods for the simulation of skeletal muscle. In Proc. of the 2003 ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 68–74. Google ScholarDigital Library
52. Teran, J., Sifakis, E., Irving, G., and Fedkiw, R. 2005. Robust quasistatic finite elements and flesh simulation. ACM Trans. on Graphics (to appear).Google Scholar
53. Teran, J., Sifakis, E., Salinas-Blemker, S., Ng-Thow-Hing, V., Lau, C., and Fedkiw, R. 2005. Creating and simulating skeletal muscle from the visible human data set. IEEE Trans. on Vis. and Comput. Graph. 11, 3, 317–328. Google ScholarDigital Library
54. Terzopoulos, D., and Waters, K. 1993. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Trans. on Pattern Analysis and Machine Intelligence 15, 6. Google ScholarDigital Library
55. Teschner, M., Girod, S., and Girod, B. 2000. Direct computation of nonlinear soft-tissue deformation. In Proc. of Vision, Modeling, and Visualization, 383–390.Google Scholar
56. Teschner, M., Heidelberger, B., Müller, M., Pomeranets, D., and Gross, M. 2003. Optimized spatial hashing for collision detection of deformable objects. In Proc. of Vision, Modeling, Visualization (VMV), 47–54.Google Scholar
57. U.S. NATIONAL LIBRARY OF MEDICINE, 1994. The visible human project. http://www.nlm.nih.gov/research/visible/.Google Scholar
58. Wang, Y., Huang, X., Lee, C. S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dynamic 3-D facial expressions. In Proc. of Eurographics, 677–686.Google Scholar
59. Waters, K., and Frisbie, J. 1995. A coordinated muscle model for speech animation. In Proc. of Graphics Interface, 163–170.Google Scholar
60. Waters, K. 1987. A muscle model for animating three-dimensional facial expressions. Comput. Graph. (SIGGRAPH Proc.), 17–24. Google ScholarDigital Library
61. Williams, L. 1990. Performance-driven facial animation. In Computer Graphics (Proc. of Int. Conf. on Computer Graphics and Interactive Techniques), ACM Press, 235–242. Google ScholarDigital Library
62. Wu, Y., Kalra, P., Magnenat-Thalmann, N., and Thalmann, D. 1999. Simulating wrinkles and skin aging. The Visual Computer 15, 4, 183–198.Google ScholarCross Ref
63. Zajac, F. 1989. Muscle and tendon: Properties, models, scaling, and application to biomechanics and motor control. Critical Reviews in Biomed. Eng. 17, 4, 359–411.Google Scholar
64. Zhang, Q., Liu, Z., Guo, B., and Shum, H. 2003. Geometry-driven photorealistic facial expression synthesis. In Proc. of ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., ACM Press, 16–22. Google ScholarDigital Library
65. Zhang, L., Snavely, N., Curless, B., and Seitz, S. 2004. Spacetime faces: High resolution capture for modeling and animation. In Proc. of ACM SIGGRAPH, ACM Press, vol. 23, 548–558. Google ScholarDigital Library