“SCULPTOR: Skeleton-Consistent Face Creation Using a Learned Parametric Generator” by Qiu, Li, He, Zhang, Zhang, et al. …
Conference:
Type(s):
Title:
- SCULPTOR: Skeleton-Consistent Face Creation Using a Learned Parametric Generator
Session/Category Title: Faces and Avatars
Presenter(s)/Author(s):
Abstract:
Recent years have seen growing interest in 3D human face modeling due to its wide applications in digital human, character generation and animation. Existing approaches overwhelmingly emphasized on modeling the exterior shapes, textures and skin properties of faces, ignoring the inherent correlation between inner skeletal structures and appearance. In this paper, we present SCULPTOR, 3D face creations with Skeleton Consistency Using a Learned Parametric facial generaTOR, aiming to facilitate the easy creation of both anatomically correct and visually convincing face models via a hybrid parametric-physical representation. At the core of SCULPTOR is LUCY, the first large-scale shape-skeleton face dataset in collaboration with plastic surgeons. Named after the fossils of one of the oldest known human ancestors, our LUCY dataset contains high-quality Computed Tomography (CT) scans of the complete human head before and after orthognathic surgeries, which are critical for evaluating surgery results. LUCY consists of 144 scans of 72 subjects (31 male and 41 female), where each subject has two CT scans taken pre- and post-orthognathic operations. Based on our LUCY dataset, we learned a novel skeleton consistent parametric facial generator, SCULPTOR, which can create unique and nuanced facial features that help define a character and at the same time maintain physiological soundness. Our SCULPTOR jointly models the skull, face geometry and face appearance under a unified data-driven framework by separating the depiction of a 3D face into shape blend shape, pose blend shape and facial expression blend shape. SCULPTOR preserves both anatomic correctness and visual realism in facial generation tasks compared with existing methods. Finally, we showcase the robustness and effectiveness of SCULPTOR in various fancy applications unseen before, like archaeological skeletal facial completion, bone-aware character fusion, skull inference from images, face generation with lipo-Level change and facial animations, etc.
References:
1. 3dMD. 2022. 3dMDface™ System. https://3dmd.com/
2. 3DSCANSTORE. 2022. 3D Scan Store: Captured Assets for Digital Artists. https://www.3dscanstore.com/
3. Jiwanasha Manish Agrawal, Manish Suresh Agrawal, Lalita Girish Nanjannawar, and Anita D Parushetti. 2013. CBCT in orthodontics: the wave of future. The journal of contemporary dental practice 14, 1 (2013), 153.
4. Linchao Bao, Xiangkai Lin, Yajing Chen, Haoxian Zhang, Sheng Wang, Xuefei Zhe, Di Kang, Haozhi Huang, Xinwei Jiang, Jue Wang, Dong Yu, and Zhengyou Zhang. 2021. High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies. ACM Trans. Graph. 41, 1, Article 3 (nov 2021), 21 pages.
5. Thabo Beeler and Derek Bradley. 2014. Rigid stabilization of facial expressions. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–9.
6. Amit Bermano, Thabo Beeler, Yeara Kozlov, Derek Bradley, Bernd Bickel, and Markus Gross. 2015. Detailed Spatio-Temporal Reconstruction of Eyelids. ACM Transactions on Graphics 34 (07 2015), 44:1–44:11.
7. P.J. Besl and Neil D. McKay. 1992. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 2 (1992), 239–256.
8. Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. 187–194.
9. Blender. 2021. Cycles renderer.
10. Gunilla Borgefors. 1983. Chamfering: A fast method for obtaining approximations of the Euclidean distance in N dimensions. In Proc. 3rd Scand. Conf. on Image Analysis (SCIA3). 250–255.
11. Alan Brunton, Timo Bolkart, and Stefanie Wuhrer. 2014. Multilinear Wavelets: A Statistical Shape Space for Human Faces, Vol. 8689.
12. Pascal Bérard, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. Lightweight eye capture using a parametric model. ACM Transactions on Graphics 35 (07 2016), 1–12.
13. Chen Cao, Derek Bradley, Kun Zhou, and Thabo Beeler. 2015. Real-Time High-Fidelity Facial Performance Capture. ACM Transactions on Graphics 34 (07 2015), 46:1–46:9.
14. Peter Claes, Dirk Vandermeulen, Sven Greef, Guy Willems, John Clement, and Paul Suetens. 2010. Computerized craniofacial reconstruction: Conceptual framework and review. Forensic science international 201 (03 2010), 138–45.
15. Juliette P. Driessen, Hadé Vuyk, and Johannes Borgstein. 2011. New insights into facial anthropometry in digital photographs using iris dependent calibration. International Journal of Pediatric Otorhinolaryngology 75, 4 (2011), 579–584.
16. Fuqing Duan, Donghua Huang, Yun Tian, Ke Lu, Zhongke Wu, and Mingquan Zhou. 2015. 3D face reconstruction from skull by regression modeling in shape parameter spaces. Neurocomputing 151 (2015), 674–682.
17. Fuqing Duan, Sen Yang, Donghua Huang, Yongli Hu, Zhongke Wu, and Mingquan Zhou. 2014. Craniofacial reconstruction based on multi-linear subspace analysis. Multimedia Tools and Applications 73, 2 (2014), 809–823.
18. Bernhard Egger, William Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhöfer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter. 2020. 3D Morphable Face Models—Past, Present, and Future. ACM Transactions on Graphics 39 (06 2020), 1–38.
19. Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. 2021. Learning an animatable detailed 3D face model from in-the-wild images. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–13.
20. Pablo Garrido, Michael Zollhöfer, Chenglei Wu, Derek Bradley, Patrick Pérez, Thabo Beeler, and Christian Theobalt. 2016. Corrective 3D reconstruction of lips from monocular video. ACM Transactions on Graphics 35 (11 2016), 1–11.
21. Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview Face Capture Using Polarized Spherical Gradient Illumination. ACM Trans. Graph. 30, 6 (dec 2011), 1–10.
22. Paulo Gotardo, Jeremy Riviere, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2018. Practical dynamic facial appearance modeling and acquisition. ACM Transactions on Graphics 37, 1–13.
23. Aurel Gruber, Marco Fratarcangeli, Gaspard Zoss, Roman Cattaneo, Thabo Beeler, Markus Gross, and Derek Bradley. 2020. Interactive Sculpting of Digital Faces Using an Anatomical Modeling Paradigm. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 93–102.
24. Marc Habermann, Weipeng Xu, Michael Zollhöfer, Gerard Pons-Moll, and Christian Theobalt. 2019. LiveCap: Real-Time Human Performance Capture From Monocular Video. ACM Transactions on Graphics 38 (03 2019), 1–17.
25. Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. 2019. Learning Joint Reconstruction of Hands and Manipulated Objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
26. Maya Hoole, Alison Sheridan, Angela Boyle, Thomas Booth, Selina Brace, Yoan Diekmann, Iñigo Olalde, Mark Thomas, Ian Barnes, Jane Evans, Carolyn Chenery, Hilary Sloane, Hew Morrison, Sheena Fraser, Scott Timpany, and Derek Hamilton. 2018. ‘Ava’: a Beaker-associated woman from a cist at Achavanich, Highland, and the story of her (re-)discovery and subsequent study. Proceedings of the Society of Antiquaries of Scotland 147 (Nov. 2018), 73–118.
27. Liwen Hu, Chongyang Ma, Linjie Luo, and Hao Li. 2015. Single-View Hair Modeling Using A Hairstyle Database. ACM Transactions on Graphics 34 (07 2015), 125:1–125:9.
28. Alexandru-Eugen Ichim, Petr Kadleček, Ladislav Kavan, and Mark Pauly. 2017. Phace: Physics-Based Face Modeling and Animation. ACM Trans. Graph. 36, 4, Article 153 (jul 2017), 14 pages.
29. Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4401–4410.
30. Leif Kobbelt, Swen Campagna, Jens Vorsatz, and Hans-Peter Seidel. 1998. Interactive Multi-Resolution Modeling on Arbitrary Meshes. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’98). Association for Computing Machinery, New York, NY, USA, 105–114.
31. Yeara Kozlov, Derek Bradley, Moritz Bächer, Bernhard Thomaszewski, Thabo Beeler, and Markus Gross. 2017. Enriching facial blendshape rigs with physical simulation. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 75–84.
32. Ágnes Kustár, Laszlo Forró, Ildikó Kalina, Ferenc Fazekas, Szabolcs Honti, Sz Makra, and Martin Friess. 2013. FACE-R-A 3D Database of 400 Living Individuals’ Full Head CT- and Face Scans and Preliminary GMM Analysis for Craniofacial Reconstruction. Journal of forensic sciences 58 (09 2013).
33. Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, and Hao Li. 2020. Learning Formation of Physically-Based Face Attributes. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3407–3416.
34. Tianye Li, Timo Bolkart, Michael J. Black, Hao Li, and Javier Romero. 2017. Learning a Model of Facial Shape and Expression from 4D Scans. ACM Trans. Graph. 36, 6, Article 194 (nov 2017), 17 pages.
35. Yuwei Li, Minye Wu, Yuyao Zhang, Lan Xu, and Jingyi Yu. 2021. PIANO: A Parametric Hand Bone Model from Magnetic Resonance Imaging. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. 816–822.
36. Yuwei Li, Longwen Zhang, Zesong Qiu, Yingwenqi Jiang, Nianyi Li, Yuexin Ma, Yuyao Zhang, Lan Xu, and Jingyi Yu. 2022. NIMBLE: A Non-Rigid Hand Model with Bones and Muscles. ACM Trans. Graph. 41, 4, Article 120 (jul 2022), 16 pages.
37. Anthony Little, Ben Jones, and Lisa DeBruine. 2011. Facial attractiveness: Evolutionary based research. Philosophical transactions of the Royal Society of London. Series B, Biological sciences 366 (06 2011), 1638–59.
38. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34, 6 (Oct. 2015), 248:1–248:16.
39. Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec. 2007. Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination. 183–194.
40. Dennis Madsen, Marcel Lüthi, Andreas Schneider, and Thomas Vetter. 2018. Probabilistic Joint Face-Skull Modelling for Facial Reconstruction. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5295–5303.
41. Masaya Misaki, Jonathan Savitz, Vadim Zotev, Raquel Phillips, Han Yuan, Kymberly Young, Wayne Drevets, and Jerzy Bodurka. 2014. Contrast Enhancement by Combining T1-and T2-Weighted Structural Brain MR Images. Magnetic Resonance in Medicine 74 (12 2014).
42. Koki Nagano, Graham Fyffe, Oleg Alexander, Jernej Barbiç, Hao Li, Abhijeet Ghosh, and Paul Debevec. 2015. Skin Microstructure Deformation with Displacement Map Convolution. ACM Transactions on Graphics 34 (07 2015), 109:1–109:10.
43. Thomas Neumann, Kiran Varanasi, Stephan Wenger, Markus Wacker, Marcus Magnor, and Christian Theobalt. 2013. Sparse Localized Deformation Components. ACM Transactions on Graphics (TOG) 32 (11 2013).
44. Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE conference on computer vision and pattern recognition. 343–352.
45. Kyle Olszewski, Duygu Ceylan, Jun Xing, Jose Echevarria, Zhili Chen, Weikai Chen, and Hao Li. 2020. Intuitive, Interactive Beard and Hair Synthesis With Generative Models. 7444–7454.
46. Stylianos Ploumpis, Evangelos Ververas, Eimear O’ Sullivan, Stylianos Moschoglou, Nick Pears, William Smith, Baris Gecer, and Stefanos Zafeiriou. 2020. Towards a Complete 3D Morphable Model of the Human Head. IEEE Transactions on Pattern Analysis and Machine Intelligence PP (04 2020), 1–1.
47. Neng Qian, Jiayi Wang, Franziska Mueller, Florian Bernard, Vladislav Golyanik, and Christian Theobalt. 2020. HTML: A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization. In European Conference on Computer Vision. Springer, 54–71.
48. Rodrigo Quian. 2017. How Do We Recognize a Face? Cell 169 (06 2017), 975–977.
49. Mallikarjun R, Ayush Tewari, Hans-Peter Seidel, Mohamed Elgharib, and Christian Theobalt. 2021. Learning Complete 3D Morphable Face Models from Images and Videos. 3360–3370.
50. Taehyun Rhee, J.P. Lewis, Ulrich Neumann, and Krishna Nayak. 2007. Soft-Tissue Deformation for In Vivo Volume Animation. In 15th Pacific Conference on Computer Graphics and Applications (PG’07). 435–438.
51. Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) 36, 6 (Nov. 2017).
52. Wuyang Shui, Mingquan Zhou, Steve Maddock, Taiping He, Xingce Wang, and Qingqiong Deng. 2017. A PCA-Based Method for Determining Craniofacial Relationship and Sexual Dimorphism of Facial Shapes. Computers in Biology and Medicine 90 (08 2017).
53. Arthur Swift, Steven Liew, Susan Weinkle, Julie K Garcia, and Michael B Silberberg. 2021. The facial aging process from the “inside out”. Aesthetic Surgery Journal 41, 10 (2021), 1107–1119.
54. Quan Wen, Feng Xu, Ming Lu, and Jun-Hai Yong. 2017. Real-time 3D eyelids tracking from semantic edges. ACM Transactions on Graphics 36 (11 2017), 1–11.
55. Chenglei Wu, Derek Bradley, Pablo Garrido, Michael Zollhöfer, Christian Theobalt, Markus Gross, and Thabo Beeler. 2016a. Model-based teeth reconstruction. ACM Transactions on Graphics 35 (11 2016), 1–13.
56. Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016b. An anatomically-constrained local deformation model for monocular face capture. ACM Transactions on Graphics 35 (07 2016), 1–12.
57. Woffles Wu. 2020. Invited Update: Consensus on Changing Trends, Attitudes, and Concepts of Asian Beauty and Consensus on Current Injectable Treatment Strategies in the Asian Face. Aesthetic Plastic Surgery 44 (08 2020).
58. Lan Xu, Wei Cheng, Kaiwen Guo, Lei Han, Yebin Liu, and Lu Fang. 2019. Flyfusion: Realtime dynamic scene reconstruction using a flying depth camera. IEEE transactions on visualization and computer graphics 27, 1 (2019), 68–82.
59. Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao. 2020. Facescape: a large-scale high quality 3d face dataset and detailed riggable 3d face prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 601–610.
60. Gaspard Zoss, Derek Bradley, Pascal Bérard, and Thabo Beeler. 2018. An Empirical Rig for Jaw Animation. ACM Trans. Graph. 37, 4, Article 59 (jul 2018), 12 pages.
