“Physical face cloning” by Bickel, Kaufmann, Skouras, Thomaszewski, Bradley, et al. …

We propose a complete process for designing, simulating, and fabricating synthetic skin for an animatronics character that mimics the face of a given subject and its expressions. The process starts with measuring the elastic properties of a material used to manufacture synthetic soft tissue. Given these measurements we use physics-based simulation to predict the behavior of a face when it is driven by the underlying robotic actuation. Next, we capture 3D facial expressions for a given target subject. As the key component of our process, we present a novel optimization scheme that determines the shape of the synthetic skin as well as the actuation parameters that provide the best match to the target expressions. We demonstrate this computational skin design by physically cloning a real human face onto an animatronics figure.

1. Becker, M., and Teschner, M. 2007. Robust and efficient estimation of elasticity parameters using the linear finite element method. In SimVis, 15–28.Google Scholar
2. Beeler, T., Bickel, B., Beardsley, P., Sumner, B., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graph. 29, 4 (July), 40:1–40:9. Google ScholarDigital Library
3. Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30, 4 (August), 75:1–75:10. Google ScholarDigital Library
4. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graph. 26, 3 (July), 33:1–33:10. Google ScholarDigital Library
5. Bickel, B., Bächer, M., Otaduy, M. A., Matusik, W., Pfister, H., and Gross, M. 2009. Capture and modeling of non-linear heterogeneous soft tissue. ACM Trans. Graph. 28, 3 (July), 89:1–89:9. Google ScholarDigital Library
6. Bickel, B., Bächer, M., Otaduy, M. A., Lee, H. R., Pfister, H., Gross, M., and Matusik, W. 2010. Design and fabrication of materials with desired deformation behavior. ACM Trans. Graph. 29, 4 (July), 63:1–63:10. Google ScholarDigital Library
7. Bonet, J., and Wood, R. D. 1997. Nonlinear Continuum Mechanics for Finite Element Analysis. Cambridge Uni. Press.Google Scholar
8. Bradley, D., Boubekeur, T., and Heidrich, W. 2008. Accurate multi-view reconstruction using robust binocular stereo and surface meshing. In Proc. CVPR.Google Scholar
9. Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 4 (July), 41:1–41:10. Google ScholarDigital Library
10. Bucur, D., and Buttazzo, G. 2005. Variational Methods in Shape Optimization Problems. Birkhuser Mathematics.Google Scholar
11. Chentanez, N., Alterovitz, R., Ritchie, D., Cho, L., Hauser, K. K., Goldberg, K., Shewchuk, J. R., and O’Brien, J. F. 2009. Interactive simulation of surgical needle insertion and steering. ACM Trans. Graph. 28, 3 (July), 88:1–88:10. Google ScholarDigital Library
12. Clarberg, P., Jarosz, W., Akenine-Möller, T., and Jensen, H. W. 2005. Wavelet importance sampling: efficiently evaluating products of complex functions. ACM Trans. Graph. 24, 3 (Aug.), 1166–1175. Google ScholarDigital Library
13. Dong, Y., Wang, J., Pellacini, F., Tong, X., and Guo, B. 2010. Fabricating spatially-varying subsurface scattering. ACM Trans. Graph. 29 (July), 62:1–62:10. Google ScholarDigital Library
14. Gourret, J.-P., Thalmann, N. M., and Thalmann, D. 1989. Simulation of object and human skin deformations in a grasping task. In Comp. Graph. (Proc. SIGGRAPH), 21–30. Google ScholarDigital Library
15. Hara, F., Akazawa, H., and Kobayashi, H. 2001. Realistic facial expressions by sma driven face robot. In Proc. of IEEE International Workshop on Robot and Human Interactive Communication, 504–511.Google Scholar
16. Hašan, M., Fuchs, M., Matusik, W., Pfister, H., and Rusinkiewicz, S. 2010. Physical reproduction of materials with specified subsurface scattering. ACM Trans. Graph. 29 (July), 61:1–61:10. Google ScholarDigital Library
17. Irving, G., Teran, J., and Fedkiw, R. 2004. Invertible finite elements for robust simulation of large deformation. In 2004 ACM SIGGRAPH/Eurographics SCA, 131–140. Google ScholarDigital Library
18. Kauer, M., Vuskovic, V., Dual, J., Szekely, G., and Bajka, M. 2002. Inverse finite element characterization of soft tissues. Medical Image Analysis 6, 3, 257–287.Google ScholarCross Ref
19. Kharevych, L., Mullen, P., Owhadi, H., and Desbrun, M. 2009. Numerical coarsening of inhomogeneous elastic materials. ACM Trans. Graph. 28 (July), 51:1–51:8. Google ScholarDigital Library
20. Koch, R. M., Gross, M. H., Carls, F. R., von Büren, D. F., Fankhauser, G., and Parish, Y. I. H. 1996. Simulating facial surgery using finite element models. In Proc. of Comp. graph. and int. tech., ACM, SIGGRAPH ’96, 421–428. Google ScholarDigital Library
21. Lee, S.-H., and Terzopoulos, D. 2006. Heads up!: biomechanical modeling and neuromuscular control of the neck. In ACM SIGGRAPH 2006 Papers, ACM, 1188–1198. Google ScholarDigital Library
22. Lee, S.-H., Sifakis, E., and Terzopoulos, D. 2009. Comprehensive biomechanical modeling and simulation of the upper body. ACM Trans. Graph. 28 (September), 99:1–99:17. Google ScholarDigital Library
23. Levin, D. 1998. The approximation power of moving least-squares. Math. Comput. 67 (October), 1517–1531. Google ScholarDigital Library
24. Li, X.-Y., Shen, C.-H., Huang, S.-S., Ju, T., and Hu, S.-M. 2010. Popup: automatic paper architectures from 3d models. ACM Trans. Graph. 29 (July), 111:1–111:9. Google ScholarDigital Library
25. Mori, Y., and Igarashi, T. 2007. Plushie: an interactive design system for plush toys. In ACM SIGGRAPH 2007 papers, ACM, SIGGRAPH ’07. Google ScholarDigital Library
26. Nesme, M., Kry, P. G., Jeřábková, L., and Faure, F. 2009. Preserving topology and elasticity for embedded deformable models. In ACM SIGGRAPH 2009 papers, ACM, 52:1–52:9. Google ScholarDigital Library
27. Nishio, S., Ishiguro, H., and Hagita, N. 2007. Humanoid Robots: New Developments. I-Tech, ch. Geminoid: Teleoperated Android of an Existing Person.Google Scholar
28. Nocedal, J., and Wright, S. J. 2000. Numerical Optimization. Springer, August.Google Scholar
29. Oh, J.-H., Hanson, D., Kim, W.-S., Han, I. Y., Kim, J.-Y., and Park, I.-W. 2006. Design of android type humanoid robot albert hubo. In Proc. of IEEE/RSJ Int. Conference on Intelligent Robots and Systems, 1428–1433.Google Scholar
30. Pai, D. K., Doel, K. v. d., James, D. L., Lang, J., Lloyd, J. E., Richmond, J. L., and Yau, S. H. 2001. Scanning physical interaction behavior of 3d objects. In Proc. of Comp. graph. and int. tech., ACM, SIGGRAPH ’01, 87–96. Google ScholarDigital Library
31. Schenk, O., and Gärtner, K. 2006. On fast factorization pivoting methods for symmetric indefinite systems. In Elec. Trans. Numer., no. 23, 158–179.Google Scholar
32. Sifakis, E., Neverov, I., and Fedkiw, R. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. In ACM SIGGRAPH 2005, 417–425. Google ScholarDigital Library
33. Sueda, S., Kaufman, A., and Pai, D. K. 2008. Musculotendon simulation for hand animation. ACM Trans. Graph. (Proc. SIGGRAPH) 27, 3. Google ScholarDigital Library
34. Teran, J., Sifakis, E., Blemker, S. S., Ng-Thow-Hing, V., Lau, C., and Fedkiw, R. 2005. Creating and simulating skeletal muscle from the visible human data set. IEEE Trans. on Vis. and Comp. Graph. 11, 3, 317–328. Google ScholarDigital Library
35. Terzopoulos, D., and Waters, K. 1993. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Trans. Pattern Anal. Mach. Intell. 15 (June), 569–579. Google ScholarDigital Library
36. Terzopoulos, D., Platt, J., Barr, A., and Fleischer, K. 1987. Elastically deformable models. In Proc. Comp. graph. and int. tech., ACM, SIGGRAPH ’87, 205–214. Google ScholarDigital Library
37. Thoutireddy, P., and Ortiz, M. 2004. A variational r-adaption and shape-optimization method for finite-deformation elasticity. Int. J. Numer. Meth. Engng. 61, 1–21.Google ScholarCross Ref
38. Van Gelder, A. 1998. Approximate simulation of elastic membranes by triangulated spring meshes. J. Graph. Tools 3, 2, 21–42. Google ScholarDigital Library
39. Wang, Y., Huang, X., Lee, C.-S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dyanmic 3-d facial expressions. In Comp. Graph. Forum, 677–686.Google Scholar
40. Weyrich, T., Peers, P., Matusik, W., and Rusinkiewicz, S. 2009. Fabricating microgeometry for custom surface reflectance. ACM Trans. Graph. 28 (July), 32:1–32:6. Google ScholarDigital Library
41. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3 (Aug.), 548–558. Google ScholarDigital Library