“Multi-scale capture of facial geometry and motion” by Bickel, Botsch, Angst, Matusik, Otaduy, et al. …

We present a novel multi-scale representation and acquisition method for the animation of high-resolution facial geometry and wrinkles. We first acquire a static scan of the face including reflectance data at the highest possible quality. We then augment a traditional marker-based facial motion-capture system by two synchronized video cameras to track expression wrinkles. The resulting model consists of high-resolution geometry, motion-capture data, and expression wrinkles in 2D parametric form. This combination represents the facial shape and its salient features at multiple scales. During motion synthesis the motion-capture data deforms the high-resolution geometry using a linear shell-based mesh-deformation method. The wrinkle geometry is added to the facial base mesh using nonlinear energy optimization. We present the results of our approach for performance replay as well as for wrinkle editing.

1. Bando, Y., Kuratate, T., and Nishita, T. 2002. A simple method for modeling wrinkles on human skin. In Proc. of Pacific Conference on Computer Graphics and Applications. Google ScholarDigital Library
2. Blanz, V., Basso, C., Poggio, T., and Vetter, T. 2003. Reanimating faces in images and video. Computer Graphics Forum 22, 3, 641–650.Google ScholarCross Ref
3. Borshukov, G., and Lewis, J. 2003. Realistic human face rendering for “The Matrix Reloaded”. In ACM SIGGRAPH 03 Sketches & Applications. Google ScholarDigital Library
4. Borshukov, G., Piponi, D., Larsen, O., Lewis, J., and Tempelaar-Lietz, C. 2003. Universal capture – Image-based facial animation for “The Matrix Reloaded”. In ACM SIGGRAPH 03 Sketches & Applications. Google ScholarDigital Library
5. Botsch, M., and Kobbelt, L. 2004. An intuitive framework for real-time freeform modeling. ACM Transactions on Graphics 23, 3, 630–634. Google ScholarDigital Library
6. Botsch, M., and Kobbelt, L. 2005. Real-time shape editing using radial basis functions. Computer Graphics Forum 24, 3, 611–621.Google ScholarCross Ref
7. Botsch, M., and Sorkine, O. 2007. On linear variational surface deformation methods. IEEE Transactions on Visualization and Computer Graphics (TVCG), to appear. Google ScholarDigital Library
8. Botsch, M., Bommes, D., and Kobbelt, L. 2005. Efficient linear system solvers for geometry processing. In 11th IMA conference on the Mathematics of Surfaces, 62–83. Google ScholarDigital Library
9. Bridson, R., Marino, S., and Fedkiw, R. 2003. Simulation of clothing with folds and wrinkles. In Proc. of ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA), 28–36. Google ScholarDigital Library
10. Carr, J. C., Beatson, R. K., Cherrie, J. B., Mitchell, T. J., Fright, W. R., McCallum, B. C., and Evans, T. R. 2001. Reconstruction and representation of 3D objects with radial basis functions. In Proc. of ACM SIGGRAPH 01, 67–76. Google ScholarDigital Library
11. Celniker, G., and Gossard, D. 1991. Deformable curve and surface finite-elements for free-form shape design. In Proc. of ACM SIGGRAPH 91, 257–266. Google ScholarDigital Library
12. Cortes, C., and Vapnik, V. 1995. Support-vector networks. Machine Learning 20, 3, 273–297. Google ScholarDigital Library
13. DeCarlo, D., and Metaxas, D. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 231–238. Google ScholarDigital Library
14. Desbrun, M., Meyer, M., Schröder, P., and Barr, A. H. 1999. Implicit fairing of irregular meshes using diffusion and curvature flow. In Proc. of ACM SIGGRAPH 99, 317–324. Google ScholarDigital Library
15. Duchon, J. 1977. Spline minimizing rotation-invariant semi-norms in Sobolev spaces. In Constructive Theory of Functions of Several Variables, W. Schempp and K. Zeller, Eds., no. 571 in Lecture Notes in Mathematics. Springer, 85–100.Google Scholar
16. Essa, I. A., and Pentland, A. 1997. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 19, 7, 757–763. Google ScholarDigital Library
17. Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads: Using input from video. In Proc. of Computer Animation 96, 68–79. Google ScholarDigital Library
18. Grinspun, E., Hirani, A. N., Desbrun, M., and Schröder, P. 2003. Discrete shells. In Proc. of ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA), 62–67. Google ScholarDigital Library
19. Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Proc. of ACM SIGGRAPH 98, 55–66. Google ScholarDigital Library
20. Horn, B. K. 1987. Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America 4, 4, 629–642.Google ScholarCross Ref
21. Hyneman, W., Itokazu, H., Williams, L., and Zhao, X. 2005. Human face project. In ACM SIGGRAPH 05 Course Notes. Google ScholarDigital Library
22. Igarashi, T., Nishino, K., and Nayar, S. 2005. The appearance of human skin. Tech. Rep. CUCS-024-05, Department of Computer Science, Columbia University.Google Scholar
23. Jones, A., Gardner, A., Bolas, M., McDowall, I., and Debevec, P. 2006. Performance geometry capture for spatially varying relighting. In 3rd European Conference on Visual Media Production (CVMP 2006).Google Scholar
24. Kobbelt, L., Vorsatz, J., and Seidel, H.-P. 1999. Multiresolution hierarchies on unstructured triangle meshes. Comput. Geom. Theory Appl. 14, 1–3, 5–24. Google ScholarDigital Library
25. Koch, R. M., Gross, M. H., Carls, F. R., Von Büren, D. F., Fankhauser, G., and Parish, Y. 1996. Simulating facial surgery using finite element methods. In Proc. of ACM SIGGRAPH 96, 421–428. Google ScholarDigital Library
26. Lanir, Y. 1987. Skin mechanics. In Handbook of Bioengineering, R. Skalak and S. Chien, Eds. McGraw-Hill, 11.1–11.25.Google Scholar
27. Li, H., Roivainen, P., and Forchheimer, R. 1993. 3-D motion estimation in model-based facial image coding. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 15, 6, 545–555. Google ScholarDigital Library
28. Magnenat-Thalmann, N., Kalra, P., Lévêque, J. L., Bazin, R., Batisse, D., and Queleux, B. 2002. A computational skin model: fold and wrinkle formation. IEEE Trans. on Information Technology in Biomedicine 6, 4, 317–323. Google ScholarDigital Library
29. Meyer, M., Desbrun, M., Schröder, P., and Barr, A. H. 2003. Discrete differential-geometry operators for triangulated 2-manifolds. In Visualization and Mathematics III, H.-C. Hege and K. Polthier, Eds. Springer-Verlag, Heidelberg, 35–57.Google Scholar
30. Nehab, D., Rusinkiewicz, S., Davis, J., and Ramamoorthi, R. 2005. Efficiently combining positions and normals for precise 3d geometry. ACM Transactions on Graphics 24, 3, 536–543. Google ScholarDigital Library
31. Noh, J.-Y, and Neumann, U. 1999. A survey of facial modeling and animation techniques. Tech. Rep. USC-TR-99-705, Univeristy of Southern Californina.Google Scholar
32. Noh, J.-Y., and Neumann, U. 2001. Expression cloning. In Proc. of SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 277–288. Google ScholarDigital Library
33. Pighin, F. H., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3D model-based tracking. In International Conference on Computer Vision (ICCV), 143–150.Google Scholar
34. Shi, L., Yu, Y., Bell, N., and Feng, W.-W. 2006. A fast multigrid algorithm for mesh deformation. ACM Transactions on Graphics 25, 3, 1108–1117. Google ScholarDigital Library
35. Sifakis, E., Neverov, I., and Fedkiw, R. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics 24, 3, 417–425. Google ScholarDigital Library
36. Svoboda, T., Martinec, D., and Pajdla, T. 2005. A convenient multicamera self-calibration for virtual environments. Presence: Teleoper. Virtual Environ. 14, 4, 407–422. Google ScholarDigital Library
37. Terzopoulus, D., and Waters, K. 1993. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 14, 569–579. Google ScholarDigital Library
38. Venkataraman, K., Lodha, S., and Raghavan, R. 2005. A kinematic-variational model for animating skin with wrinkles. Computers & Graphics 29, 5, 756–770. Google ScholarDigital Library
39. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Transactions on Graphics 24, 3, 426–433. Google ScholarDigital Library
40. Wang, Y., Huang, X., Lee, C.-S., Zhang, S., LI, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dynamic 3-D facial expressions. Computer Graphics Forum 23, 3, 677–686.Google ScholarCross Ref
41. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., and Debevec, P. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics 24, 3, 756–764. Google ScholarDigital Library
42. Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., TU, C., McAndless, J., Lee, J., Ngan, A., Jensen, H. W., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Transactions on Graphics 25, 3, 1013–1024. Google ScholarDigital Library
43. Williams, L. 1990. Performance-driven facial animation. In Proc. of ACM SIGGRAPH 90, vol. 24, 235–242. Google ScholarDigital Library
44. Wu, Y., Kalra, P., and Magnenat-Thalmann, N. 1996. Simulation of static and dynamic wrinkles of skin. In Proc. of Computer Animation, 90–97. Google ScholarDigital Library
45. Zhang, S., and Huang, P. 2006. High-resolution, real-time three-dimensional shape measurement. Optical Engineering 45, 12.Google Scholar
46. Zhang, Y., and Sim, T. 2005. Realistic and efficient wrinkle simulation using an anatomy-based face model with adaptive refinement. In Computer Graphics International 2005, 3–10. Google ScholarDigital Library
47. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3, 548–558. Google ScholarDigital Library