“Contact-centric deformation learning” by Romero, Casas, Chiaramonte and Otaduy

We propose a novel method to machine-learn highly detailed, nonlinear contact deformations for real-time dynamic simulation. We depart from previous deformation-learning strategies, and model contact deformations in a contact-centric manner. This strategy shows excellent generalization with respect to the object’s configuration space, and it allows for simple and accurate learning. We complement the contact-centric learning strategy with two additional key ingredients: learning a continuous vector field of contact deformations, instead of a discrete approximation; and sparsifying the mapping between the contact configuration and contact deformations. These two ingredients further contribute to the accuracy, efficiency, and generalization of the method. We integrate our learning-based contact deformation model with subspace dynamics, showing real-time dynamic simulations with fine contact deformation detail.

1. Thiemo Alldieck, Hongyi Xu, and Cristian Sminchisescu. 2021. imGHUM: Implicit Generative Models of 3D Human Shape and Articulated Pose. In Proc. of Computer Vision and Pattern Recognition (CVPR). 5461–5470.Google ScholarCross Ref
2. Steven S. An, Theodore Kim, and Doug L. James. 2008. Optimizing Cubature for Efficient Integration of Subspace Deformations. ACM Trans. Graph. 27, 5, Article 165 (2008).Google ScholarDigital Library
3. Uri M. Ascher and Eddy Boxerman. 2003. On the Modified Conjugate Gradient Method in Cloth Simulation. Vis. Comput. 19, 7–8 (2003).Google ScholarDigital Library
4. Matan Atzmon and Yaron Lipman. 2020. SAL: Sign Agnostic Learning of Shapes from Raw Data. In Proc. of Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
5. Stephen W. Bailey, Dave Otte, Paul Dilorenzo, and James F. O’Brien. 2018. Fast and Deep Deformation Approximations. ACM Trans. Graph. 37, 4 (2018).Google ScholarDigital Library
6. Jernej Barbič and Doug L. James. 2005. Real-Time Subspace Integration for St. Venant-Kirchhoff Deformable Models. ACM Trans. Graph. 24, 3 (July 2005), 982–990.Google ScholarDigital Library
7. Jernej Barbič and Yili Zhao. 2011. Real-Time Large-Deformation Substructuring. In ACM SIGGRAPH 2011 Papers (Vancouver, British Columbia, Canada) (SIGGRAPH ’11). Association for Computing Machinery, New York, NY, USA, Article 91, 8 pages.Google Scholar
8. Nuri Benbarka, Timon Höfer, Hamd ul-Moqeet Riaz, and Andreas Zell. 2022. Seeing Implicit Neural Representations As Fourier Series. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2041–2050.Google ScholarCross Ref
9. Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. 2020. LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration. In Advances in Neural Information Processing Systems (NeurIPS).Google Scholar
10. Christopher Brandt, Elmar Eisemann, and Klaus Hildebrandt. 2018. Hyper-Reduced Projective Dynamics. ACM Trans. Graph. 37, 4, Article 80 (2018).Google ScholarDigital Library
11. Dan Casas and Miguel A Otaduy. 2018. Learning nonlinear soft-tissue dynamics for interactive avatars. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 1 (2018), 10.Google ScholarDigital Library
12. Peter Yichen Chen, Maurizio Chiaramonte, Eitan Grinspun, and Kevin Carlberg. 2021. Model reduction for the material point method via learning the deformation map and its spatial-temporal gradients. arXiv preprint arXiv:2109.12390 (2021).Google Scholar
13. Zhiqin Chen and Hao Zhang. 2019. Learning implicit fields for generative shape modeling. In Proc. of Computer Vision and Pattern Recognition (CVPR). 5939–5948.Google ScholarCross Ref
14. Julian Chibane, Aymen Mir, and Gerard Pons-Moll. 2020. Neural Unsigned Distance Fields for Implicit Function Learning. In Advances in Neural Information Processing Systems (NeurIPS).Google Scholar
15. Enric Corona, Albert Pumarola, Guillem Alenyà, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2021. SMPLicit: Topology-aware Generative Model for Clothed People. In Proc. of Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
16. Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. 2020. NASA: Neural Articulated Shape Approximation. In Proc. of European Conference on Computer Vision (ECCV).Google ScholarDigital Library
17. Eric Ferley, Marie-Paule Cani, and Jean-Dominique Gascuel. 1999. Practical Volumetric Sculpting. The Visual Computer 16 (09 1999).Google Scholar
18. Lawson Fulton, Vismay Modi, David Duvenaud, David I. W. Levin, and Alec Jacobson. 2019. Latent-space Dynamics for Reduced Deformable Simulation. Computer Graphics Forum 38, 2 (2019), 379–391.Google ScholarCross Ref
19. Theodore F Gast, Craig Schroeder, Alexey Stomakhin, Chenfanfu Jiang, and Joseph M Teran. 2015. Optimization integrator for large time steps. IEEE Transactions on Visualization and Computer Graphics (TVCG) 21, 10 (2015), 1103–1115.Google ScholarDigital Library
20. Benjamin Gilles, Guillaume Bousquet, Francois Faure, and Dinesh K. Pai. 2011. Frame-Based Elastic Models. ACM Trans. Graph. 30, 2 (2011).Google ScholarDigital Library
21. Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. 2020. Implicit Geometric Regularization for Learning Shapes. In Proceedings of Machine Learning and Systems 2020. 3569–3579.Google Scholar
22. Fabian Hahn, Sebastian Martin, Bernhard Thomaszewski, Robert Sumner, Stelian Coros, and Markus Gross. 2012. Rig-Space Physics. ACM Trans. Graph. 31, 4, Article 72 (July 2012), 8 pages.Google ScholarDigital Library
23. Fabian Hahn, Bernhard Thomaszewski, Stelian Coros, Robert W. Sumner, Forrester Cole, Mark Meyer, Tony DeRose, and Markus Gross. 2014. Subspace Clothing Simulation Using Adaptive Bases. ACM Trans. Graph. 33, 4 (2014).Google ScholarDigital Library
24. David Harmon and Denis Zorin. 2013. Subspace Integration with Local Deformations. ACM Trans. Graph. 32, 4, Article 107 (July 2013), 10 pages.Google ScholarDigital Library
25. Daniel Holden, Bang Chi Duong, Sayantan Datta, and Derek Nowrouzezahrai. 2019. Subspace Neural Physics: Fast Data-Driven Interactive Simulation. In Proceedings of the 18th Annual ACM SIGGRAPH/Eurographics Symposium on Computer Animation.Google ScholarDigital Library
26. Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. 2020. ARCH: Animatable Reconstruction of Clothed Humans. In Proc. of Computer Vision and Pattern Recognition (CVPR). 3093–3102.Google ScholarCross Ref
27. Alec Jacobson, Zhigang Deng, Ladislav Kavan, and JP Lewis. 2014. Skinning: Real-time Shape Deformation. In ACM SIGGRAPH 2014 Courses.Google Scholar
28. Korrawe Karunratanakul, Adrian Spurr, Zicong Fan, Otmar Hilliges, and Siyu Tang. 2021. A Skeleton-Driven Neural Occupancy Representation for Articulated Hands. In International Conference on 3D Vision (3DV).Google ScholarCross Ref
29. Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang. 2020. Grasping Field: Learning Implicit Representations for Human Grasps. In International Conference on 3D Vision.Google Scholar
30. Meekyoung Kim, Gerard Pons-Moll, Sergi Pujades, Seungbae Bang, Jinwook Kim, Michael J. Black, and Sung-Hee Lee. 2017. Data-Driven Physics for Human Soft Tissue Animation. ACM Trans. Graph. 36, 4, Article 54 (2017), 12 pages. Google ScholarDigital Library
31. Theodore Kim and Doug L. James. 2011. Physics-Based Character Skinning Using Multi-Domain Subspace Deformations. In Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 63–72.Google Scholar
32. P. Krysl, S. Lall, and J. E. Marsden. 2001. Dimensional model reduction in non-linear finite element dynamics of solids and structures. Internat. J. Numer. Methods Engrg. 51, 4 (2001), 479–504.Google ScholarCross Ref
33. Tsuneya Kurihara and Natsuki Miyata. 2004. Modeling Deformable Human Hands from Medical Images. In Proc. of ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 355–363.Google ScholarDigital Library
34. Lei Lan, Ran Luo, Marco Fratarcangeli, Weiwei Xu, Huamin Wang, Xiaohu Guo, Junfeng Yao, and Yin Yang. 2020. Medial Elastics: Efficient and Collision-Ready Deformation via Medial Axis Transform. ACM Trans. Graph. 39, 3, Article 20 (2020).Google ScholarDigital Library
35. Kookjin Lee and Kevin T. Carlberg. 2021. Deep Conservation: A Latent-Dynamics Model for Exact Satisfaction of Physical Conservation Laws. In AAAI.Google Scholar
36. J. P. Lewis, Matt Cordner, and Nickson Fong. 2000. Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH). ACM Press/Addison-Wesley Publishing Co., USA, 165–172.Google ScholarDigital Library
37. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-person Linear Model. ACM Trans. Graph. 34, 6, Article 248 (Oct. 2015), 16 pages.Google ScholarDigital Library
38. Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black. 2020. Learning to Dress 3D People in Generative Clothing. In Proc. of Computer Vision and Pattern Recognition (CVPR).Google Scholar
39. N. Magnenat-Thalmann, R. Laperrière, and D. Thalmann. 1988. Joint-dependent Local Deformations for Hand Animation and Object Grasping. In Proceedings on Graphics Interface ’88 (Edmonton, Alberta, Canada). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 26–33.Google Scholar
40. Sebastian Martin, Bernhard Thomaszewski, Eitan Grinspun, and Markus Gross. 2011. Example-Based Elastic Materials. ACM Trans. Graph. 30, 4, Article 72 (2011).Google ScholarDigital Library
41. Aleka McAdams, Yongning Zhu, Andrew Selle, Mark Empey, Rasmus Tamstorf, Joseph Teran, and Eftychios Sifakis. 2011. Efficient Elasticity for Character Skinning with Contact and Collisions. In ACM SIGGRAPH 2011 Papers (Vancouver, British Columbia, Canada) (SIGGRAPH ’11). Association for Computing Machinery, New York, NY, USA, Article 37.Google Scholar
42. Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019. Occupancy Networks: Learning 3D Reconstruction in Function Space. In Proc. of Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
43. Marko Mihajlovic, Yan Zhang, Michael J Black, and Siyu Tang. 2021. LEAP: Learning Articulated Occupancy of People. In Proc. of Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
44. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Proc. of European Conference on Computer Vision (ECCV).Google ScholarDigital Library
45. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Love-grove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In Proc. of Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
46. Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. 2020. TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE.Google Scholar
47. Alex Pentland and John Williams. 1989. Good Vibrations: Modal Dynamics for Graphics and Animation. Computer Graphics 23, 3 (1989), 215–222.Google ScholarDigital Library
48. Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael J. Black. 2017. ClothCap: Seamless 4D Clothing Capture and Retargeting. ACM Transactions on Graphics (Proc. of SIGGRAPH) 36, 4, Article 73 (2017), 15 pages.Google Scholar
49. Gerard Pons-Moll, Javier Romero, Naureen Mahmood, and Michael J. Black. 2015. Dyna: A Model of Dynamic Human Shape in Motion. ACM Trans. Graph. 34, 4, Article 120 (July 2015), 14 pages.Google ScholarDigital Library
50. Cristian Romero, Dan Casas, Jesús Pérez, and Miguel Otaduy. 2021. Learning Contact Corrections for Handle-Based Subspace Dynamics. ACM Trans. Graph. 40, 4 (2021).Google ScholarDigital Library
51. Cristian Romero, Miguel A. Otaduy, Dan Casas, and Jesus Perez. 2020. Modeling and Estimation of Nonlinear Skin Mechanics for Animated Avatars. Computer Graphics Forum (Proc. Eurographics) 39, 2 (2020).Google Scholar
52. Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Trans. Graph. 36, 6, Article 245 (2017).Google ScholarDigital Library
53. Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. 2019. PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization.Google Scholar
54. Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. 2020. PIFuHD: MultiLevel Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization. In Proc. of Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
55. Shunsuke Saito, Jinlong Yang, Qianli Ma, and Michael J. Black. 2021. SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks. In Proc. of Computer Vision and Pattern Recognition (CVPR).Google Scholar
56. Igor Santesteban, Elena Garces, Miguel A. Otaduy, and Dan Casas. 2020. SoftSMPL: Data-driven Modeling of Nonlinear Soft-tissue Dynamics for Parametric Humans. Computer Graphics Forum 39, 2 (2020), 65–75.Google ScholarCross Ref
57. Igor Santesteban, Miguel A. Otaduy, and Dan Casas. 2019. Learning-Based Animation of Clothing for Virtual Try-On. Computer Graphics Forum 38, 2 (2019), 355–366.Google ScholarCross Ref
58. Igor Santesteban, Nils Thuerey, Miguel A Otaduy, and Dan Casas. 2021. Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021).Google ScholarCross Ref
59. Siyuan Shen, Yin Yang, Tianjia Shao, He Wang, Chenfanfu Jiang, Lei Lan, and Kun Zhou. 2021. High-Order Differentiable Autoencoder for Nonlinear Model Reduction. ACM Trans. Graph. 40, 4, Article 68 (2021).Google ScholarDigital Library
60. Eftychios Sifakis and Jernej Barbic. 2012. FEM Simulation of 3D Deformable Solids: A Practitioner’s Guide to Theory, Discretization and Model Reduction. In ACM SIGGRAPH 2012 Courses. 20:1–20:50.Google Scholar
61. Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. 2020. Implicit Neural Representations with Periodic Activation Functions. In Proc. NeurIPS.Google Scholar
62. Breannan Smith, Fernando De Goes, and Theodore Kim. 2018. Stable Neo-Hookean Flesh Simulation. ACM Trans. Graph. 37, 2, Article 12 (2018).Google ScholarDigital Library
63. Steven L. Song, Weiqi Shi, and Michael Reed. 2020. Accurate Face Rig Approximation with Deep Differential Subspace Reconstruction. ACM Trans. Graph. 39, 4 (2020). Google ScholarDigital Library
64. Javier Tapia, Cristian Romero, Jesús Pérez, and Miguel A. Otaduy. 2021. Parametric Skeletons with Reduced Soft-Tissue Deformations. Computer Graphics Forum (2021).Google Scholar
65. Yun Teng, Mark Meyer, Tony DeRose, and Theodore Kim. 2015. Subspace Condensation: Full Space Adaptivity for Subspace Deformations. ACM Trans. Graph. 34, 4, Article 76 (July 2015), 9 pages.Google ScholarDigital Library
66. Demetri Terzopoulos, John Platt, Alan Barr, and Kurt Fleischer. 1987. Elastically Deformable Models. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’87). Association for Computing Machinery, New York, NY, USA, 205–214.Google ScholarDigital Library
67. Garvita Tiwari, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll. 2021. Neural-GIF: Neural generalized implicit functions for animating people in clothing. In Proc. of Computer Vision and Pattern Recognition (CVPR). 11708–11718.Google ScholarCross Ref
68. Yu Wang, Alec Jacobson, Jernej Barbič, and Ladislav Kavan. 2015. Linear Subspace Design for Real-Time Shape Deformation. ACM Trans. Graph. 34, 4, Article 57 (2015).Google ScholarDigital Library
69. Xiaofeng Wu, Rajaditya Mukherjee, and Huamin Wang. 2015. A Unified Approach for Subspace Simulation of Deformable Bodies in Multiple Domains. ACM Trans. Graph. 34, 6, Article 241 (2015).Google ScholarDigital Library
70. Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. 2021. Neural Fields in Visual Computing and Beyond. arXiv:arXiv:2111.11426 https://neuralfields.cs.brown.edu/Google Scholar
71. Hongyi Xu and Jernej Barbič. 2016. Pose-Space Subspace Dynamics. ACM Trans. Graph. 35, 4, Article 35 (July 2016), 14 pages.Google ScholarDigital Library
72. Meng Zhang, Tuanfeng Y. Wang, Duygu Ceylan, and Niloy J. Mitra. 2021. Dynamic Neural Garments. ACM Trans. Graph. 40, 6, Article 235 (2021).Google ScholarDigital Library