“Learning active quasistatic physics-based models from data” by Srinivasan, Wang, Rojas, Klár, Kavan, et al. …

Humans and animals can control their bodies to generate a wide range of motions via low-dimensional action signals representing high-level goals. As such, human bodies and faces are prime examples of active objects, which can affect their shape via an internal actuation mechanism. This paper explores the following proposition: given a training set of example poses of an active deformable object, can we learn a low-dimensional control space that could reproduce the training set and generalize to new poses? In contrast to popular machine learning methods for dimensionality reduction such as auto-encoders, we model our active objects in a physics-based way. We utilize a differentiable, quasistatic, physics-based simulation layer and combine it with a decoder-type neural network. Our differentiable physics layer naturally fits into deep learning frameworks and allows the decoder network to learn actuations that reach the desired poses after physics-based simulation. In contrast to modeling approaches where users build anatomical models from first principles, medical literature or medical imaging, we do not presume knowledge of the underlying musculature, but learn the structure and control of the actuation mechanism directly from the input data. We present a training paradigm and several scalability-oriented enhancements that allow us to train effectively while accommodating high-resolution volumetric models, with as many as a quarter million simulation elements. The prime demonstration of the efficacy of our example-driven modeling framework targets facial animation, where we train on a collection of input expressions while generalizing to unseen poses, drive detailed facial animation from sparse motion capture input, and facilitate expression sculpting via direct manipulation.

1. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In International conference on machine learning. 214–223.Google ScholarDigital Library
2. Timur Bagautdinov, Chenglei Wu, Jason Saragih, Pascal Fua, and Yaser Sheikh. 2018. Modeling Facial Geometry Using Compositional VAEs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
3. Stephen W. Bailey, Dalton Omens, Paul Dilorenzo, and James F. O’Brien. 2020. Fast and Deep Facial Deformations. ACM Trans. Graph. 39, 4, Article 94 (July 2020).Google ScholarDigital Library
4. Michael Bao, Matthew Cong, Stephane Grabli, and Ronald Fedkiw. 2019. High-Quality Face Capture Using Anatomical Muscles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
5. GG Barbarino, M Jabareen, J Trzewik, A Nkengne, G Stamatas, and E Mazza. 2009. Development and validation of a three-dimensional finite element model of the face. Journal of biomechanical engineering 131, 4 (2009), 041006.Google ScholarCross Ref
6. James Bern, Grace Kumagai, and Stelian Coros. 2017b. Fabrication, Modeling, and Control of Plush Robots. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 3739 — 3746. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017); Conference Location: Vancouver, Canada; Conference Date: September 24-28, 2017.Google Scholar
7. James M Bern, Pol Banzet, Roi Poranne, and Stelian Coros. 2019. Trajectory optimization for cable-driven soft robot locomotion. Proceedings of Robotics: Science and Systems (2019).Google ScholarCross Ref
8. James M. Bern, Kai-Hung Chang, and Stelian Coros. 2017a. Interactive Design of Animated Plushies. ACM Trans. Graph. 36, 4, Article 80 (July 2017), 11 pages.Google ScholarDigital Library
9. Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. 187–194.Google ScholarDigital Library
10. Silvia Salinas Blemker. 2004. 3D modeling of complex muscle architecture and geometry. Ph.D. Dissertation. Stanford University.Google Scholar
11. Sofien Bouaziz, Sebastian Martin, Tiantian Liu, Ladislav Kavan, and Mark Pauly. 2014. Projective dynamics: fusing constraint projections for fast simulation. Proc. of ACM SIGGRAPH 33, 4 (2014).Google ScholarDigital Library
12. Matthew Cong, Kiran S Bhat, and Ronald Fedkiw. 2016. Art-directed muscle simulation for high-end facial animation. 119–127.Google Scholar
13. Stelian Coros, Sebastian Martin, Bernhard Thomaszewski, Christian Schumacher, Robert Sumner, and Markus Gross. 2012. Deformable objects alive! ACM Transactions on Graphics (TOG) 31, 4 (2012), 69.Google Scholar
14. Kai Ding, Libin Liu, Michiel van de Panne, and KangKang Yin. 2015. Learning Reduced-Order Feedback Policies for Motion Skills. In Proceedings of the 14th ACM SIGGRAPH / Eurographics Symposium on Computer Animation (Los Angeles, California) (SCA ’15). Association for Computing Machinery, New York, NY, USA, 83–92.Google Scholar
15. Carl Doersch. 2016. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908 (2016).Google Scholar
16. Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. 2021. DiffPD: Differentiable Projective Dynamics with Contact. arXiv:2101.05917 [cs.LG]Google Scholar
17. Paul Ekman and Wallace V Friesen. 1977. Facial action coding system. (1977).Google Scholar
18. Ye Fan, Joshua Litven, and Dinesh K Pai. 2014. Active volumetric musculoskeletal systems. Proc. of ACM SIGGRAPH 33, 4 (2014), 152.Google ScholarDigital Library
19. François Faure, Christian Duriez, Hervé Delingette, Jérémie Allard, Benjamin Gilles, Stéphanie Marchesseau, Hugo Talbot, Hadrien Courtecuisse, Guillaume Bousquet, Igor Peterlik, et al. 2012. Sofa: A multi-model framework for interactive physical simulation. In Soft tissue biomechanical modeling for computer assisted surgery. Springer, 283–321.Google Scholar
20. Cormac Flynn, Ian Stavness, John Lloyd, and Sidney Fels. 2015. A finite element model of the face including an orthotropic skin model under in vivo tension. Computer methods in biomechanics and biomedical engineering 18, 6 (2015), 571–582.Google Scholar
21. Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. 2020. ADD: analytically differentiable dynamics for multi-body systems with frictional contact. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–15.Google ScholarDigital Library
22. Zhenglin Geng, Daniel Johnson, and Ronald Fedkiw. 2020. Coercing machine learning to output physically accurate results. J. Comput. Phys. 406 (Apr 2020), 109099.Google ScholarCross Ref
23. Evgeny Gladilin. 2003. Biomechanical modeling of soft tissue and facial expressions for craniofacial surgery planning. Freien University, Berlin (2003).Google Scholar
24. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672–2680.Google Scholar
25. David Hahn, Pol Banzet, James M Bern, and Stelian Coros. 2019. Real2sim: Visco-elastic parameter estimation from dynamic motion. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1–13.Google ScholarDigital Library
26. I. Higgins, Loïc Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. 2017. beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In ICLR.Google Scholar
27. Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. 2020. DiffTaichi: Differentiable Programming for Physical Simulation. In International Conference on Learning Representations.Google Scholar
28. Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua B Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojciech Matusik. 2019. Chainqueen: A real-time differentiable physical simulator for soft robotics. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 6265–6271.Google ScholarDigital Library
29. Alexandru Ichim, Ladislav Kavan, Merlin Nimier-David, and Mark Pauly. 2016. Building and Animating User-Specific Volumetric Face Rigs.Google Scholar
30. Alexandru-Eugen Ichim, Petr Kadleček, Ladislav Kavan, and Mark Pauly. 2017. Phace: Physics-based face modeling and animation. ACM Transactions on Graphics (TOG) 36, 4 (2017), 153.Google ScholarDigital Library
31. Petr Kadlecek and Ladislav Kavan. 2019. Building Accurate Physics-Based Face Models from Data. In Symposium on Computer Animation.Google ScholarDigital Library
32. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.Google Scholar
33. Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.Google Scholar
34. Gergely Klár, Andrew Moffat, Ken Museth, and Eftychios Sifakis. 2020. Shape Targeting: A Versatile Active Elasticity Constitutive Model. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Talks (SIGGRAPH ’20). Association for Computing Machinery, Article 59, 2 pages.Google Scholar
35. Yeara Kozlov, Derek Bradley, Moritz Bächer, Bernhard Thomaszewski, Thabo Beeler, and Markus Gross. 2017. Enriching Facial Blendshape Rigs with Physical Simulation. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 75–84.Google Scholar
36. Lana Lan, Matthew Cong, and Ronald Fedkiw. 2017. Lessons from the evolution of an anatomical facial muscle model. In Proceedings of the ACM SIGGRAPH Digital Production Symposium. ACM, 11.Google ScholarDigital Library
37. John P Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frederic H Pighin, and Zhigang Deng. 2014. Practice and Theory of Blendshape Facial Models. Eurographics (State of the Art Reports) 1, 8 (2014), 2.Google Scholar
38. Jiaman Li, Zhengfei Kuang, Yajie Zhao, Mingming He, Karl Bladin, and Hao Li. 2020b. Dynamic facial asset and rig generation from a single scan. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–18.Google ScholarDigital Library
39. Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al. 2020a. Learning Formation of Physically-Based Face Attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3410–3419.Google ScholarCross Ref
40. John E Lloyd, Ian Stavness, and Sidney Fels. 2012. ArtiSynth: A fast interactive biomechanical modeling toolkit combining multibody and finite element simulation. In Soft tissue biomechanical modeling for computer assisted surgery. Springer, 355–394.Google Scholar
41. Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser Sheikh. 2018. Deep appearance models for face rendering. ACM Transactions on Graphics (TOG) 37, 4 (2018), 68.Google ScholarDigital Library
42. Steve A Maas, Gerard A Ateshian, and Jeffrey A Weiss. 2017. FEBio: History and advances. Annual review of biomedical engineering 19 (2017), 279–299.Google Scholar
43. Sebastian Martin, Bernhard Thomaszewski, Eitan Grinspun, and Markus Gross. 2011. Example-based elastic materials. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 72.Google ScholarDigital Library
44. Aleka McAdams, Yongning Zhu, Andrew Selle, Mark Empey, Rasmus Tamstorf, Joseph Teran, and Eftychios Sifakis. 2011. Efficient elasticity for character skinning with contact and collisions. ACM Transactions on Graphics (TOG) 30, 4 (2011), 37.Google ScholarDigital Library
45. Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. 2004. Fluid Control Using the Adjoint Method. ACM Trans. Graph. 23, 3 (Aug. 2004), 449–456.Google ScholarDigital Library
46. Nathan Mitchell, Court Cutting, and Eftychios Sifakis. 2015. GRIDiron: An interactive authoring and cognitive training foundation for reconstructive plastic surgery procedures. ACM Trans. Graph. (Proceedings of ACM SIGGRAPH) (2015).Google ScholarDigital Library
47. Paola Nardinocchi and Luciano Teresi. 2007. On the active response of soft living tissues. Journal of Elasticity 88, 1 (2007), 27–39.Google ScholarCross Ref
48. Thomas Neumann, Kiran Varanasi, Stephan Wenger, Markus Wacker, Marcus Magnor, and Christian Theobalt. 2013. Sparse Localized Deformation Components. ACM Trans. Graph. 32, 6, Article 179 (Nov. 2013), 10 pages.Google ScholarDigital Library
49. Michael Schmidt and Hod Lipson. 2009. Distilling Free-Form Natural Laws from Experimental Data. Science 324, 5923 (2009), 81–85.Google Scholar
50. Gabriel Schwartz, Shih-En Wei, Te-Li Wang, Stephen Lombardi, Tomas Simon, Jason Saragih, and Yaser Sheikh. 2020. The eyes have it: an integrated eye and face model for photorealistic facial animation. ACM Transactions on Graphics (TOG) 39, 4 (2020), 91–1.Google ScholarDigital Library
51. Eftychios Sifakis and Jernej Barbič. 2012. FEM Simulation of 3D Deformable Solids: A practitioner’s guide to theory, discretization and model reduction. http://www.femdefo.org.Google Scholar
52. Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. In Proc. of ACM SIGGRAPH, Vol. 24. 417–425.Google ScholarDigital Library
53. Eftychios Sifakis, Tamar Shinar, Geoffrey Irving, and Ronald Fedkiw. 2007. Hybrid simulation of deformable solids. In Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Computer animation. Eurographics Association, 81–90.Google ScholarDigital Library
54. Breannan Smith, Chenglei Wu, He Wen, Patrick Peluse, Yaser Sheikh, Jessica K Hodgins, and Takaaki Shiratori. 2020. Constraining dense hand surface tracking with elasticity. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–14.Google ScholarDigital Library
55. Martin Spüler, Nerea Irastorza Landa, Andrea Sarasola Sanz, and Ander Ramos-Murguialday. 2016. Extracting Muscle Synergy Patterns from EMG Data Using Autoencoders. In Artificial Neural Networks and Machine Learning – ICANN 2016. 47–54.Google Scholar
56. Ian Stavness, Mohammad Ali Nazari, Cormac Flynn, Pascal Perrier, Yohan Payan, John E Lloyd, and Sidney Fels. 2014. Coupled biomechanical modeling of the face, jaw, skull, tongue, and hyoid bone. In 3D Multiscale Physiological Human. Springer, 253–274.Google Scholar
57. Jie Tan, Greg Turk, and C Karen Liu. 2012. Soft body locomotion. ACM Transactions on Graphics (TOG) 31, 4 (2012), 26.Google ScholarDigital Library
58. J Rafael Tena, Fernando De la Torre, and Iain Matthews. 2011. Interactive region-based linear 3d face models. In ACM SIGGRAPH 2011 papers. 1–10.Google ScholarDigital Library
59. Joseph Teran, Sylvia Blemker, V Hing, and Ronald Fedkiw. 2003. Finite volume methods for the simulation of skeletal muscle. Eurographics Association, 68–74.Google Scholar
60. Joseph Teran, Eftychios Sifakis, Silvia S Blemker, Victor Ng-Thow-Hing, Cynthia Lau, and Ronald Fedkiw. 2005. Creating and simulating skeletal muscle from the visible human data set. IEEE TVCG 11, 3 (2005), 317–328.Google Scholar
61. Shih-En Wei, Jason Saragih, Tomas Simon, Adam W Harley, Stephen Lombardi, Michal Perdoch, Alexander Hypes, Dawei Wang, Hernan Badino, and Yaser Sheikh. 2019. Vr facial animation via multiview image translation. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–16.Google ScholarDigital Library
62. Jeffrey A Weiss, Bradley N Maker, and Sanjay Govindjee. 1996. Finite element implementation of incompressible, transversely isotropic hyperelasticity. Computer methods in applied mechanics and engineering 135, 1 (1996), 107–128.Google Scholar
63. Chris Wojtan, Peter J. Mucha, and Greg Turk. 2006. Keyframe control of complex particle systems using the adjoint method. In SCA ’06: Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation (Vienna, Austria). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 15–23.Google ScholarDigital Library
64. Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. An anatomically-constrained local deformation model for monocular face capture. ACM transactions on graphics (TOG) 35, 4 (2016), 1–12.Google Scholar
65. Felix E Zajac. 1989. Muscle and tendon Properties models scaling and application to biomechanics and motor. Critical reviews in biomedical engineering 17, 4 (1989), 359–411.Google Scholar
66. Xiaotian Zhang, Fan Kiat Chan, Tejaswin Parthasarathy, and Mattia Gazzola. 2019. Modeling and simulation of complex dynamic musculoskeletal architectures. Nature communications 10, 1 (2019), 1–12.Google Scholar