“Learning skeletal articulations with neural blend shapes” by Zhang, Aberman, Hanocka, Liu, Sorkine-Hornung, et al. …

Animating a newly designed character using motion capture (mocap) data is a long standing problem in computer animation. A key consideration is the skeletal structure that should correspond to the available mocap data, and the shape deformation in the joint regions, which often requires a tailored, pose-specific refinement. In this work, we develop a neural technique for articulating 3D characters using enveloping with a pre-defined skeletal structure which produces high quality pose dependent deformations. Our framework learns to rig and skin characters with the same articulation structure (e.g., bipeds or quadrupeds), and builds the desired skeleton hierarchy into the network architecture. Furthermore, we propose neural blend shapes – a set of corrective pose-dependent shapes which improve the deformation quality in the joint regions in order to address the notorious artifacts resulting from standard rigging and skinning. Our system estimates neural blend shapes for input meshes with arbitrary connectivity, as well as weighting coefficients which are conditioned on the input joint rotations. Unlike recent deep learning techniques which supervise the network with ground-truth rigging and skinning parameters, our approach does not assume that the training data has a specific underlying deformation model. Instead, during training, the network observes deformed shapes and learns to infer the corresponding rig, skin and blend shapes using indirect supervision. During inference, we demonstrate that our network generalizes to unseen characters with arbitrary mesh connectivity, including unrigged characters built by 3D artists. Conforming to standard skeletal animation models enables direct plug-and-play in standard animation software, as well as game engines.

1. Kfir Aberman, Peizhuo Li, Olga Sorkine-Hornung, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. 2020. Skeleton-Aware Networks for Deep Motion Retargeting. ACM Transactions on Graphics (TOG) 39, 4 (2020), 62.Google ScholarDigital Library
2. Adobe Systems Inc. 2018. Mixamo. https://www.mixamo.com. https://www.mixamo.com Accessed: 2018-12-27.Google Scholar
3. Oscar Kin-Chung Au, Chiew-Lan Tai, Hung-Kuo Chu, Daniel Cohen-Or, and Tong-Yee Lee. 2008. Skeleton Extraction by Mesh Contraction. ACM Transactions on Graphics 27, 3 (Aug. 2008), 1–10. Google ScholarDigital Library
4. Stephen W. Bailey, Dalton Omens, Paul Dilorenzo, and James F. O’Brien. 2020. Fast and Deep Facial Deformations. ACM Trans. Graph. 39, 4, Article 94 (July 2020), 15 pages. Google ScholarDigital Library
5. Stephen W Bailey, Dave Otte, Paul Dilorenzo, and James F O’Brien. 2018. Fast and deep deformation approximations. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–12.Google ScholarDigital Library
6. Ilya Baran and Jovan Popović. 2007. Automatic rigging and animation of 3d characters. ACM Transactions on graphics (TOG) 26, 3 (2007), 72–es.Google Scholar
7. Gaurav Bharaj, Thorsten Thormählen, Hans-Peter Seidel, and Christian Theobalt. 2012. Automatically Rigging Multi-component Characters. Computer Graphics Forum 31, 2pt4 (2012), 755–764. : https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8659.2012.03034.x. Google ScholarCross Ref
8. Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. 2019. Multi-Garment Net: Learning to Dress 3D People from Images. In IEEE International Conference on Computer Vision (ICCV). IEEE.Google ScholarCross Ref
9. Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. 2010. Polygon mesh processing. CRC press.Google Scholar
10. Junjie Cao, Andrea Tagliasacchi, Matt Olson, Hao Zhang, and Zhinxun Su. 2010. Point Cloud Skeletons via Laplacian Based Contraction. In 2010 Shape Modeling International Conference. IEEE, Aix-en-Provence, France, 187–197. Google ScholarDigital Library
11. Edilson De Aguiar, Christian Theobalt, Sebastian Thrun, and Hans-Peter Seidel. 2008. Automatic conversion of mesh animations into skeleton-based animations. In Computer Graphics Forum, Vol. 27. Wiley Online Library, 389–397.Google Scholar
12. Olivier Dionne and Martin de Lasa. 2013. Geodesic voxel binding for production character meshes. In Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA ’13). Association for Computing Machinery, New York, NY, USA, 173–180. Google ScholarDigital Library
13. Stefan Fröhlich and Mario Botsch. 2011. Example-driven deformations based on discrete shells. Computer graphics forum 30, 8 (2011), 2246–2257.Google Scholar
14. Michael Gleicher. 1998. Retargetting motion to new characters. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 33–42.Google ScholarDigital Library
15. Fabian Hahn, Sebastian Martin, Bernhard Thomaszewski, Robert Sumner, Stelian Coros, and Markus Gross. 2012. Rig-Space Physics. ACM Transactions on Graphics 31, 4 (July 2012), 72:1–72:8. Google ScholarDigital Library
16. Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2019. MeshCNN: A Network with an Edge. ACM Trans. Graph. 38, 4, Article 90 (July 2019), 12 pages. Google ScholarDigital Library
17. Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. 2020. Point2Mesh: A Self-Prior for Deformable Meshes. ACM Trans. Graph. 39, 4, Article 126 (July 2020), 12 pages. Google ScholarDigital Library
18. Nils Hasler, Thorsten Thormählen, Bodo Rosenhahn, and Hans-Peter Seidel. 2010. Learning skeletons for shape and pose. In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. 23–30.Google ScholarDigital Library
19. Jim Hejl. 2004. Hardware skinning with quaternions. Game Programming Gems 4 (2004), 487–495.Google Scholar
20. Hugues Hoppe. 1996. Progressive meshes. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 99–108.Google ScholarDigital Library
21. Yixin Hu, Teseo Schneider, Bolun Wang, Denis Zorin, and Daniele Panozzo. 2020. Fast Tetrahedral Meshing in the Wild. ACM Trans. Graph. 39, 4, Article 117 (July 2020), 18 pages. Google ScholarDigital Library
22. Alec Jacobson, Ilya Baran, Jovan Popovic, and Olga Sorkine. 2011. Bounded biharmonic weights for real-time deformation. ACM Trans. Graph. 30, 4 (2011), 78.Google ScholarDigital Library
23. Doug L. James and Christopher D. Twigg. 2005. Skinning mesh animations. In ACM SIGGRAPH 2005 Papers (SIGGRAPH ’05). Association for Computing Machinery, New York, NY, USA, 399–407. Google ScholarDigital Library
24. Pushkar Joshi, Mark Meyer, Tony DeRose, Brian Green, and Tom Sanocki. 2007. Harmonic coordinates for character articulation. ACM Transactions on Graphics 26, 3 (July 2007), 71–es. Google ScholarDigital Library
25. Tao Ju, Scott Schaefer, and Joe Warren. 2005. Mean value coordinates for closed triangular meshes. ACM Transactions on Graphics 24, 3 (Jul 2005), 561–566. Google ScholarDigital Library
26. Ladislav Kavan, Steven Collins, Jiří Žára, and Carol O’Sullivan. 2007. Skinning with dual quaternions. In Proceedings of the 2007 symposium on Interactive 3D graphics and games. 39–46.Google ScholarDigital Library
27. Ladislav Kavan and Olga Sorkine. 2012. Elasticity-Inspired Deformers for Character Articulation. ACM Trans. Graph. 31, 6 (2012), 196:1–196:8.Google ScholarDigital Library
28. Ladislav Kavan and Jiří Žára. 2005. Spherical blend skinning: a real-time deformation of articulated models. In Proceedings of the 2005 symposium on Interactive 3D graphics and games. 9–16.Google ScholarDigital Library
29. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
30. Paul G. Kry, Doug L. James, and Dinesh K. Pai. 2002. EigenSkin: real time large deformation character skinning in hardware. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation (SCA ’02). Association for Computing Machinery, New York, NY, USA, 153–159. Google ScholarDigital Library
31. Binh Huy Le and Zhigang Deng. 2014. Robust and accurate skeletal rigging from mesh sequences. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–10.Google ScholarDigital Library
32. Binh Huy Le and Jessica K Hodgins. 2016. Real-time skeletal skinning with optimized centers of rotation. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1–10.Google ScholarDigital Library
33. John P Lewis, Matt Cordner, and Nickson Fong. 2000. Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 165–172.Google ScholarDigital Library
34. Tianxing Li, Rui Shi, and Takashi Kanai. 2020. DenseGATs: A Graph-Attention-Based Network for Nonlinear Character Deformation. In Symposium on Interactive 3D Graphics and Games. 1–9.Google Scholar
35. Lijuan Liu, Youyi Zheng, Di Tang, Yi Yuan, Changjie Fan, and Kun Zhou. 2019. Neuroskinning: Automatic skin binding for production characters with deep graph networks. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–12.Google ScholarDigital Library
36. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015. SMPL: A skinned multi-person linear model. ACM transactions on graphics (TOG) 34, 6 (2015), 1–16.Google ScholarDigital Library
37. Nadia Magnenat-Thalmann, Richard Laperrire, and Daniel Thalmann. 1988. Joint-dependent local deformations for hand animation and object grasping. In In Proceedings on Graphics interface’88. Citeseer.Google Scholar
38. Bruce Merry, Patrick Marais, and James Gain. 2006. Animation space: A truly linear framework for character animation. ACM Transactions on Graphics (TOG) 25, 4 (2006), 1400–1423.Google ScholarDigital Library
39. Christian Miller, Okan Arikan, and Don Fussell. 2010. Frankenrigs: building character rigs from multiple sources. In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. 31–38.Google ScholarDigital Library
40. Tomohiko Mukai and Shigeru Kuriyama. 2016. Efficient dynamic skinning with low-rank helper bone controllers. ACM Transactions on Graphics 35, 4 (July 2016), 36:1–36:11. Google ScholarDigital Library
41. Tina O’Hailey. 2018. Rig it right! Maya animation rigging concepts. Routledge.Google Scholar
42. Ahmed A A Osman, Timo Bolkart, and Michael J. Black. 2020. STAR: A Spare Trained Articulated Human Body Regressor. In European Conference on Computer Vision (ECCV). https://star.is.tue.mpg.deGoogle Scholar
43. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdfGoogle Scholar
44. Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 652–660.Google Scholar
45. Autodesk research. 2020. Meshmixer. https://www.meshmixer.com/Google Scholar
46. Scott Schaefer and Can Yuksel. 2007. Example-based skeleton extraction. In Symposium on Geometry Processing. 153–162.Google Scholar
47. Jaewoo Seo, Geoffrey Irving, J. P. Lewis, and Junyong Noh. 2011. Compression and direct manipulation of complex blendshape models. In Proceedings of the 2011 SIGGRAPH Asia Conference (SA ’11). Association for Computing Machinery, New York, NY, USA, 1–10. Google ScholarDigital Library
48. Peter-Pike J. Sloan, Charles F. Rose, and Michael F. Cohen. 2001. Shape by example. In Proceedings of the 2001 symposium on Interactive 3D graphics (I3D ’01). Association for Computing Machinery, New York, NY, USA, 135–143. Google ScholarDigital Library
49. Steven L Song, Weiqi Shi, and Michael Reed. 2020. Accurate Face Rig Approximation with Deep Differential Subspace Reconstruction. arXiv preprint arXiv:2006.01746 (2020).Google Scholar
50. Robert W Sumner, Matthias Zwicker, Craig Gotsman, and Jovan Popović. 2005. Meshbased inverse kinematics. ACM transactions on graphics (TOG) 24, 3 (2005), 488–495.Google Scholar
51. Xiaohuan Corina Wang and Cary Phillips. 2002. Multi-weight enveloping: least-squares approximation techniques for skin animation. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation. 129–138.Google ScholarDigital Library
52. Rich Wareham and Joan Lasenby. 2008. Bone Glow: An Improved Method for the Assignment of Weights for Mesh Deformation. In Articulated Motion and Deformable Objects (Lecture Notes in Computer Science), Francisco J. Perales and Robert B. Fisher (Eds.). Springer, Berlin, Heidelberg, 63–71. Google ScholarDigital Library
53. Ofir Weber, Olga Sorkine, Yaron Lipman, and Craig Gotsman. 2007. Context-aware skeletal shape deformation. In Computer Graphics Forum, Vol. 26. Wiley Online Library, 265–274.Google Scholar
54. Zhan Xu, Yang Zhou, Evangelos Kalogerakis, Chris Landreth, and Karan Singh. 2020. RigNet: Neural Rigging for Articulated Characters. ACM Transactions on Graphics 39, 4 (July 2020). Google ScholarDigital Library
55. Wang Yifan, Noam Aigerman, Vladimir G. Kim, Siddhartha Chaudhuri, and Olga Sorkine-Hornung. 2020. Neural Cages for Detail-Preserving 3D Deformations. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Seattle, WA, USA, 72–80. Google ScholarCross Ref
56. Jiayi Eris Zhang, Seungbae Bang, David I. W. Levin, and Alec Jacobson. 2020. Complementary Dynamics. ACM Transactions on Graphics 39, 6 (Nov. 2020), 179:1–179:11. Google ScholarDigital Library