“Model-based teeth reconstruction”
Conference:
Type(s):
Title:
- Model-based teeth reconstruction
Session/Category Title: Scanning & Tracking People
Presenter(s)/Author(s):
Abstract:
In recent years, sophisticated image-based reconstruction methods for the human face have been developed. These methods capture highly detailed static and dynamic geometry of the whole face, or specific models of face regions, such as hair, eyes or eye lids. Unfortunately, image-based methods to capture the mouth cavity in general, and the teeth in particular, have received very little attention. The accurate rendering of teeth, however, is crucial for the realistic display of facial expressions, and currently high quality face animations resort to tooth row models created by tedious manual work. In dentistry, special intra-oral scanners for teeth were developed, but they are invasive, expensive, cumbersome to use, and not readily available. In this paper, we therefore present the first approach for non-invasive reconstruction of an entire person-specific tooth row from just a sparse set of photographs of the mouth region. The basis of our approach is a new parametric tooth row prior learned from high quality dental scans. A new model-based reconstruction approach fits teeth to the photographs such that visible teeth are accurately matched and occluded teeth plausibly synthesized. Our approach seamlessly integrates into photogrammetric multi-camera reconstruction setups for entire faces, but also enables high quality teeth modeling from normal uncalibrated photographs and even short videos captured with a mobile phone.
References:
1. Abdelmunim, H., Chen, D., Farag, A., Pusateri, R., Carter, C., Miller, M., Farman, A., and Tasman, D. 2011. A 3d human teeth database construction based on a point-based shape registration. In IEEE ICIP, 1617–1620.
2. Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital emily project: Photoreal facial modeling and animation. In ACM SIGGRAPH 2009 Courses, 12:1–12:15.
3. Alexander, O., Fyffe, G., Busch, J., Yu, X., Ichikari, R., Jones, A., Debevec, P., Jimenez, J., Danvoye, E., Antionazzi, B., Eheler, M., Kysela, Z., and von der Pahlen, J. 2013. Digital ira: Creating a real-time photoreal digital actor. In ACM SIGGRAPH 2013 Posters, 1:1–1:1.
4. Beeler, T., Bickel, B., Beardsley, P., Sumner, B., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, 40:1–40:9.
5. Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, 75:1–75:10.
6. Beeler, T., Bickel, B., Noris, G., Marschner, S., Beardsley, P., Sumner, R. W., and Gross, M. 2012. Coupled 3d reconstruction of sparse facial hair and skin. ACM Trans. Graphics (Proc. SIGGRAPH) 31, 117:1–117:10.
7. Bérard, P., Bradley, D., Nitti, M., Beeler, T., and Gross, M. 2014. High-quality capture of eyes. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33, 6, 223:1–223:12.
8. Bermano, A., Beeler, T., Kozlov, Y., Bradley, D., Bickel, B., and Gross, M. 2015. Detailed spatio-temporal reconstruction of eyelids. ACM Trans. Graphics (Proc. SIGGRAPH) 34, 4, 44:1–44:11.
9. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In SIGGRAPH ’99, 187–194.
10. Blanz, V., Mehl, A., Vetter, T., and Seidel, H.-P. 2004. A statistical method for robust 3d surface reconstruction from sparse data. In 3DPVT, 293–300.
11. Buchaillard, S. I., Ong, S. H., Payan, Y., and Foong, K. 2007. 3d statistical models for tooth surface reconstruction. Comput. Biol. Med. 37, 10, 1461–1471.
12. Cao, C., Hou, Q., and Zhou, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans. Graphics (Proc. SIGGRAPH) 33, 4, 43:1–43:10.
13. Cao, C., Bradley, D., Zhou, K., and Beeler, T. 2015. Real-time high-fidelity facial performance capture. ACM Trans. Graphics (Proc. SIGGRAPH) 34, 4, 46:1–46:9.
14. Carter, C., Pusateri, R., Chen, D., Ahmed, A., and Farag, A. 2010. Shape from shading for hybrid surfaces as applied to tooth reconstruction. In IEEE ICIP, 4049–4052.
15. Dale, K., Sunkavalli, K., Johnson, M. K., Vlasic, D., Matusik, W., and Pfister, H. 2011. Video face replacement. In ACM Trans. Graphics (Proc. SIGGRAPH Asia), vol. 30, 130:1–130:10.
16. Dempster, A. P., Laird, N. M., and Rubin, D. B. 1977. Maximum likelihood from incomplete data via the em algorithm. JOURNAL OF THE ROYAL STATISTICAL SOCIETY 39, 1, 1–38.
17. Dollár, P., Tu, Z., and Belongie, S. 2006. Supervised learning of edges and object boundaries. In IEEE CVPR, 1964–1971.
18. Echevarria, J. I., Bradley, D., Gutierrez, D., and Beeler, T. 2014. Capturing and stylizing hair for 3d fabrication. ACM Trans. Graphics (Proc. SIGGRAPH) 33, 4, 125:1–125:11.
19. El Munim, H., and Farag, A. 2007. Shape representation and registration using vector distance functions. In IEEE CVPR, 1–8.
20. Farag, A., Elhabian, S., Abdelrehim, A., Aboelmaaty, W., Farman, A., and Tasman, D. 2013. Model-based human teeth shape recovery from a single optical image with unknown illumination. In Medical Computer Vision: Recognition Techniques and Applications in Medical Imaging (MCV ’12), 263–272.
21. Garrido, P., Valgaert, L., Wu, C., and Theobalt, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 32, 6, 158:1–158:10.
22. Garrido, P., Valgaerts, L., Rehmsen, O., Thormählen, T., Perez, P., and Theobalt, C. 2014. Automatic face reenactment. In IEEE CVPR, 4217–4224.
23. Garrido, P., Valgaerts, L., Sarmadi, H., Steiner, I., Varanasi, K., Perez, P., and Theobalt, C. 2015. Vdub: Modifying face video of actors for plausible visual alignment to a dubbed audio track. Comput. Graph. Forum 34, 2, 193–204.
24. Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 6, 129:1–129:10.
25. Grigorescu, S., Petkov, N., and Kruizinga, P. 2002. Comparison of texture features based on gabor filters. IEEE Trans. Image Proc. 11, 10 (Oct), 1160–1167.
26. Hewer, A., Steiner, I., and Wuhrer, S. 2014. A hybrid approach to 3D tongue modeling from vocal tract MRI using unsupervised image segmentation and mesh deformation. In Interspeech, 418–421.
27. Hewer, A., Steiner, I., Bolkart, T., Wuhrer, S., and Richmond, K. 2015. A statistical shape space model of the palate surface trained on 3D MRI scans of the vocal tract. In 18th International Congress of Phonetic Sciences (ICPhS).
28. Hu, L., Ma, C., Luo, L., and Li, H. 2015. Single-view hair modeling using a hairstyle database. ACM Trans. Graphics (Proc. SIGGRAPH) 34, 4.
29. Ichim, A. E., Bouaziz, S., and Pauly, M. 2015. Dynamic 3d avatar creation from hand-held video input. ACM Trans. Graphics (Proc. SIGGRAPH) 34, 4, 45:1–45:14.
30. Kawai, M., Iwao, T., Mima, D., Maejima, A., and Morishima, S. 2014. Data-driven speech animation synthesis focusing on realistic inside of the mouth. Journal of Information Processing 22, 2, 401–409. Cross Ref
31. Kim, T., Yue, Y., Taylor, S., and Matthews, I. 2015. A decision tree framework for spatiotemporal sequence prediction. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 577–586.
32. Le, B. H., Deng, Z., Xia, J., Chang, Y.-B., and Zhou, X. 2009. An interactive geometric technique for upper and lower teeth segmentation. In Proceedings of the 12th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI ’09, 968–975.
33. Mehl, A., and Blanz, V. 2005. A new approach for automatic reconstruction of occlusal surfaces with the biogeneric tooth model. Int. J. Comput. Dent. 8, 13–25.
34. Mehl, A., Blanz, V., and Hickel, R. 2005. Biogeneric tooth: a new mathematical representation for tooth morphology in lower first molars. Eur. J. Oral. Sci. 113, 333–340. Cross Ref
35. Mostafa, E., Elhabian, S., Abdelrahim, A., Elshazly, S., and Farag, A. 2014. Statistical morphable model for human teeth restoration. In IEEE ICIP, 4285–4288.
36. Munim, H. E. A. E., Farag, A., and Farman, A. 2007. A new variational approach for 3d shape registration. In International Symposium on Biomedical Imaging: From Nano to Macro (ISBI ’07), 1324–1327.
37. Nagano, K., Fyffe, G., Alexander, O., Barbič, J., Li, H., Ghosh, A., and Debevec, P. 2015. Skin microstructure deformation with displacement map convolution. ACM Trans. Graphics (Proc. SIGGRAPH) 34, 4.
38. Omachi, S., Saito, K., Aso, H., Kasahara, S., Yamada, S., and Kimura, K. 2007. Tooth shape reconstruction from ct images using spline curves. In Wavelet Analysis and Pattern Recognition, vol. 1, 393–396.
39. Rousseeuw, P. J., and Leroy, A. M. 1987. Robust Regression and Outlier Detection. John Wiley & Sons, Inc., New York, NY.
40. Shi, F., Wu, H.-T., Tong, X., and Chai, J. 2014. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33, 6, 222:1–222:13.
41. Sorkine, O., Cohen-Or, D., Lipman, Y., Alexa, M., Rössl, C., and Seidel, H.-P. 2004. Laplacian surface editing. In Proc. SGP, 179–188.
42. Suwajanakorn, S., Kemelmacher-Shlizerman, I., and Seitz, S. M. 2014. Total moving face reconstruction. In ECCV.
43. Suwajanakorn, S., Seitz, S. M., and Kemelmacher-Shlizerman, I. 2015. What makes tom hanks look like tom hanks. In Proc. ICCV.
44. Thies, J., Zollhöfer, M., Niessner, M., Valgaerts, L., Stamminger, M., and Theobalt, C. 2015. Real-time expression transfer for facial reenactment. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34, 6.
45. Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., and Niessner, M. 2016. Face2Face: Real-time Face Capture and Reenactment of RGB Videos. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.
46. Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 31, 6, 187:1–187:11.
47. Webb, J. A., and Aggarwal, J. K. 1982. Structure from motion of rigid and jointed objects. Artificial Intelligence 19, 1, 107–130.
48. Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., McAndless, J., Lee, J., Ngan, A., Jensen, H. W., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. In ACM SIGGRAPH ’06, 1013–1024.
49. Wu, C., Bradley, D., Gross, M., and Beeler, T. 2016. An anatomically-constrained local deformation model for monocular face capture. ACM Trans. Graph. 35, 4 (July), 115:1–115:12.
50. Yanagisawa, R., Sugaya, Y., Kasahara, S., and Omachi, S. 2014. Tooth shape reconstruction from dental ct images with the region-growing method. Dentomaxillofacial Radiology 43, 6, 20140080. Cross Ref
51. Zheng, S.-X., Li, J., and Sun, Q.-F. 2011. A novel 3d morphing approach for tooth occlusal surface reconstruction. Comput. Aided Des. 43, 3, 293–302.
52. Zollhöfer, M., Niessner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., et al. 2014. Real-time non-rigid reconstruction using an rgb-d camera. ACM Transactions on Graphics (TOG) 33, 4, 156.
