“An Implicit Parametric Morphable Dental Model” by Zhang, Elgharib, Fox, Gu, Theobalt, et al. …
Conference:
Type(s):
Title:
- An Implicit Parametric Morphable Dental Model
Session/Category Title: Faces and Avatars
Presenter(s)/Author(s):
Abstract:
3D Morphable models of the human body capture variations among subjects and are useful in reconstruction and editing applications. Current dental models use an explicit mesh scene representation and model only the teeth, ignoring the gum. In this work, we present the first parametric 3D morphable dental model for both teeth and gum. Our model uses an implicit scene representation and is learned from rigidly aligned scans. It is based on a component-wise representation for each tooth and the gum, together with a learnable latent code for each of such components. It also learns a template shape thus enabling several applications such as segmentation, interpolation and tooth replacement. Our reconstruction quality is on par with the most advanced global implicit representations while enabling novel applications. The code will be available at https://github.com/cong-yi/DMM
References:
1. Aly S. Abdelrehim, Aly A. Farag, Ahmed M. Shalaby, and Moumen T. El-Melegy. 2014. 2D-PCA Shape Models: Application to 3D Reconstruction of the Human Teeth from a Single Image. In Medical Computer Vision. Large Data in Medical Imaging, Bjoern Menze, Georg Langs, Albert Montillo, Michael Kelm, Henning Müller, and Zhuowen Tu (Eds.). Springer International Publishing, Cham, 44–52.
2. Thiemo Alldieck, Hongyi Xu, and Cristian Sminchisescu. 2021. imGHUM: Implicit Generative Models of 3D Human Shape and Articulated Pose. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 5441–5450.
3. K. S. Arun, T. S. Huang, and S. D. Blostein. 1987. Least-Squares Fitting of Two 3-D Point Sets. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-9, 5 (1987), 698–700.
4. Thabo Beeler and Derek Bradley. 2014. Rigid Stabilization of Facial Expressions. ACM Trans. Graph. 33, 4, Article 44 (July 2014), 9 pages.
5. Achraf Ben-Hamadou, Oussama Smaoui, Houda Chaabouni-Chouayakh, Ahmed Rekik, Sergi Pujades, Edmond Boyer, Julien Strippoli, Aurélien Thollot, Hugo Setbon, Cyril Trosset, and Edouard Ladroit. 2022. Teeth3DS: a benchmark for teeth segmentation and labeling from intra-oral 3D scans. arXiv preprint (2022).
6. Rohan Chabra, Jan E. Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. 2020. Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction. In Computer Vision – ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 608–625.
7. Zhiqin Chen and Hao Zhang. 2019. Learning Implicit Fields for Generative Shape Modeling. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5932–5941.
8. Zhang Chen, Yinda Zhang, Kyle Genova, Sean Fanello, Sofien Bouaziz, Christian Häne, Ruofei Du, Cem Keskin, Thomas Funkhouser, and Danhang Tang. 2021. Multiresolution Deep Implicit Functions for 3D Shape Representation. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 13067–13076.
9. Enric Corona, Tomas Hodan, Minh Vo, Francesc Moreno-Noguer, Chris Sweeney, Richard Newcombe, and Lingni Ma. 2022. LISA: Learning Implicit Shape and Appearance of Hands. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 20533–20543.
10. Zhiming Cui, Changjian Li, Nenglun Chen, Guodong Wei, Runnan Chen, Yuanfeng Zhou, and Wenping Wang. 2021. TSegNet: An efficient and accurate tooth segmentation network on 3D dental model. Medical Image Analysis 69 (2021), 101949.
11. Boyang Deng, J. P. Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. 2020. NASA Neural Articulated Shape Approximation. In Computer Vision – ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 612–628.
12. Yu Deng, Jiaolong Yang, and Xin Tong. 2021. Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10281–10291.
13. Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter. 2020. 3D Morphable Face Models—Past, Present, and Future. ACM Trans. Graph. 39, 5, Article 157 (jun 2020), 38 pages.
14. Aly Farag, Shireen Elhabian, Aly Abdelrehim, Wael Aboelmaaty, Allan Farman, and David Tasman. 2013. Model-Based Human Teeth Shape Recovery from a Single Optical Image with Unknown Illumination. In Medical Computer Vision. Recognition Techniques and Applications in Medical Imaging, Bjoern H. Menze, Georg Langs, Le Lu, Albert Montillo, Zhuowen Tu, and Antonio Criminisi (Eds.). 263–272.
15. Flawless 2022. Flawless AI. https://www.flawlessai.com/.
16. Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. 2020. Local Deep Implicit Functions for 3D Shape. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 4856–4865.
17. John C Gower. 1975. Generalized procrustes analysis. Psychometrika 40, 1 (1975), 33–51.
18. Liwen Hu, Shunsuke Saito, Lingyu Wei, Koki Nagano, Jaewoo Seo, Jens Fursund, Iman Sadeghi, Carrie Sun, Yen-Chun Chen, and Hao Li. 2017. Avatar Digitization from a Single Image for Real-Time Rendering. ACM Trans. Graph. 36, 6, Article 195 (nov 2017), 14 pages.
19. ISO-3950 2016. Dentistry — Designation system for teeth and areas of the oral cavity. https://www.iso.org/standard/68292.html.
20. Maik Keller, Damien Lefloch, Martin Lambers, Shahram Izadi, Tim Weyrich, and Andreas Kolb. 2013. Real-Time 3D Reconstruction in Dynamic Scenes Using Point-Based Fusion. In 2013 International Conference on 3D Vision – 3DV 2013. 1–8.
21. Hyeongwoo Kim, Mohamed Elgharib, Michael Zollhöfer, Hans-Peter Seidel, Thabo Beeler, Christian Richardt, and Christian Theobalt. 2019. Neural Style-Preserving Visual Dubbing. ACM Trans. Graph. 38, 6, Article 178 (nov 2019), 13 pages.
22. Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser Sheikh. 2018. Deep Appearance Models for Face Rendering. ACM Trans. Graph. 37, 4, Article 68 (July 2018), 13 pages.
23. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34, 6 (Oct. 2015), 248:1–248:16.
24. Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019. Occupancy Networks: Learning 3D Reconstruction in Function Space. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 4455–4465.
25. Eslam Mostafa, Shireen Elhabian, Aly Abdelrahim, Salwa Elshazly, and Aly Farag. 2014. Statistical morphable model for human teeth restoration. In 2014 IEEE International Conference on Image Processing (ICIP). 4285–4288.
26. Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. 2021. A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 13001–13011.
27. Yahini Prabha Murugesan, Abeer Alsadoon, Paul Manoranjan, and P. W.C. Prasad. 2018. A novel rotational matrix and translation vector algorithm: Geometric accuracy for augmented reality in oral and maxillofacial surgeries. International Journal of Medical Robotics and Computer Assisted Surgery 14, 3 (June 2018), 1–14.
28. Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013. RealTime 3D Reconstruction at Scale Using Voxel Hashing. ACM Trans.Graph. 32, 6, Article 169 (nov 2013), 11 pages.
29. Pablo Palafox, Aljaž Božič, Justus Thies, Matthias Nießner, and Angela Dai. 2021. NPMs: Neural Parametric Models for 3D Deformable Shapes. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 12675–12685.
30. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 165–174.
31. Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. 2021. Nerfies: Deformable Neural Radiance Fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 5865–5874.
32. Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. 2020. Convolutional Occupancy Networks. In Computer Vision – ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 523–540.
33. Yuhan Ping, Guodong Wei, Lei Yang, Zhiming Cui, and Wenping Wang. 2021. Self-attention implicit function networks for 3D dental data completion. Computer Aided Geometric Design 90 (2021), 102026.
34. Olinde Rodrigues. 1816. De l’attraction des sphéroïdes. In Correspondence Sur l’École Impériale Polytechnique. 361–385.
35. Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Trans. Graph. 36, 6, Article 245 (nov 2017), 17 pages.
36. Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. 2020. Implicit Neural Representations with Periodic Activation Functions. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 7462–7473. https://proceedings.neurips.cc/paper/2020/file/53c04118df112c13a8c34b38343b9c10-Paper.pdf
37. Vincent Sitzmann, Michael Zollhoefer, and Gordon Wetzstein. 2019. Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2019/file/b5dc4e5d9b495d0196f61d45b26ef33e-Paper.pdf
38. Synthesia 2022. Synthesia. https://www.synthesia.io/.
39. Ayush Tewari, Mohamed Elgharib, Mallikarjun B R, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt. 2020. PIE: Portrait Image Embedding for Semantic Control. ACM Trans. Graph. 39, 6, Article 223 (nov 2020), 14 pages.
40. Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred Neural Rendering: Image Synthesis Using Neural Textures. ACM Trans. Graph. 38, 4, Article 66 (jul 2019), 12 pages.
41. Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2016. Face2Face: Real-Time Face Capture and Reenactment of RGB Videos. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2387–2395.
42. Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Carsten Stoll, and Christian Theobalt. 2020. PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations. In Computer Vision – ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 293–309.
43. Zdravko Velinov, Marios Papas, Derek Bradley, Paulo Gotardo, Parsa Mirdehghan, Steve Marschner, Jan Novák, and Thabo Beeler. 2018. Appearance Capture and Modeling of Human Teeth. ACM Trans. Graph. 37, 6, Article 207 (2018), 13 pages.
44. Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. 2021. One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10034–10044.
45. Guodong Wei, Zhiming Cui, Yumeng Liu, Nenglun Chen, Runnan Chen, Guiqing Li, and Wenping Wang. 2020. TANet: Towards Fully Automatic Tooth Arrangement. In Computer Vision – ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). 481–497.
46. Andreas Wirtz, Florian Jung, Matthias Noll, Anqi Wang, and Stefan Wesarg. 2021. Automatic model-based 3-D reconstruction of the teeth from five photographs with predefined viewing directions. In Medical Imaging 2021: Image Processing, Ivana Išgum and Bennett A. Landman (Eds.), Vol. 11596. International Society for Optics and Photonics, SPIE, 198–212.
47. Chenglei Wu, Derek Bradley, Pablo Garrido, Michael Zollhöfer, Christian Theobalt, Markus Gross, and Thabo Beeler. 2016. Model-Based Teeth Reconstruction. ACM Trans. Graph. 35, 6, Article 220 (nov 2016), 13 pages.
48. Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. 2020. PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 826–835.
49. Lingchen Yang, Zefeng Shi, Yiqian Wu, Xiang Li, Kun Zhou, Hongbo Fu, and Youyi Zheng. 2020. iOrthoPredictor: Model-Guided Deep Prediction of Teeth Alignment. ACM Trans. Graph. 39, 6, Article 216 (nov 2020), 15 pages.
50. Wenwu Yang, Nathan Marshak, Daniel Sỳkora, Srikumar Ramalingam, and Ladislav Kavan. 2019. Building anatomically realistic jaw kinematics model from data. The Visual Computer 35, 6–8 (2019), 1105–1118.
51. Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans-Peter Seidel, Mohamed Elgharib, Daniel Cremers, and Christian Theobalt. 2021. i3DMM: Deep Implicit 3D Morphable Model of Human Heads. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12798–12808.
52. K. Yin, Z. Chen, S. Chaudhuri, M. Fisher, V. G. Kim, and H. Zhang. 2020. COALESCE: Component Assembly by Learning to Synthesize Connections. In 2020 International Conference on 3D Vision (3DV). IEEE Computer Society, Los Alamitos, CA, USA, 61–70.
53. Lingming Zhang, Yue Zhao, Deyu Meng, Zhiming Cui, Chenqiang Gao, Xinbo Gao, Chunfeng Lian, and Dinggang Shen. 2021. TSGCNet: Discriminative Geometric Feature Learning With Two-Stream Graph Convolutional Network for 3D Dental Model Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 6699–6708.
54. Mingwu Zheng, Hongyu Yang, Di Huang, and Liming Chen. 2022. ImFace: A Nonlinear 3D Morphable Face Model With Implicit Neural Representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 20343–20352.
55. Shu-Xian Zheng, Jia Li, and Qing-Feng Sun. 2011. A novel 3D morphing approach for tooth occlusal surface reconstruction. Computer-Aided Design 43, 3 (2011), 293–302.
56. Zerong Zheng, Tao Yu, Qionghai Dai, and Yebin Liu. 2021. Deep Implicit Templates for 3D Shape Representation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1429–1439.
57. Michael Zollhöfer, Patrick Stotko, Andreas Görlitz, Christian Theobalt, Matthias Nießner, Reinhard Klein, and Andreas Kolb. 2018a. State of the Art on 3D Reconstruction with RGB-D Cameras. Computer Graphics Forum 37, 2 (2018), 625–652. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.13386
58. M. Zollhöfer, J. Thies, P. Garrido, D. Bradley, T. Beeler, P. Pérez, M. Stamminger, M. Nießner, and C. Theobalt. 2018b. State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications. Computer Graphics Forum 37, 2 (2018), 523–550. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.13382
