“Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition” by Huang, Chai, Tong and Wu

This paper introduces a new approach for acquiring high-fidelity 3D facial performances with realistic dynamic wrinkles and fine-scale facial details. Our approach leverages state-of-the-art motion capture technology and advanced 3D scanning technology for facial performance acquisition. We start the process by recording 3D facial performances of an actor using a marker-based motion capture system and perform facial analysis on the captured data, thereby determining a minimal set of face scans required for accurate facial reconstruction. We introduce a two-step registration process to efficiently build dense consistent surface correspondences across all the face scans. We reconstruct high-fidelity 3D facial performances by combining motion capture data with the minimal set of face scans in the blendshape interpolation framework. We have evaluated the performance of our system on both real and synthetic data. Our results show that the system can capture facial performances that match both the spatial resolution of static face scans and the acquisition speed of motion capture systems.

1. Alexa, M. 2003. Differential coordinates for local mesh morphing and deformation. The Visual Computer. 19(2):105–114.Google ScholarCross Ref
2. Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital emily project: photo-real facial modeling and animation. In ACM SIGGRAPH 2009 Courses, 12:1–12:15. Google Scholar
3. Beeler, T., Bickel, B., Beardsley, P., Sumner, B., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graph. 29, 4, 40:1–40:9. Google ScholarDigital Library
4. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graph. 26, 3, 33:1–33:10. Google ScholarDigital Library
5. Blanz, V., Basso, C., Poggio, T., and Vetter, T. 2003. Reanimating faces in images and video. In Computer Graphics Forum. 22(3):641–650.Google ScholarCross Ref
6. Borshukov, G., Piponi, D., Larsen, O., Lewis, J. P., and Tempelaar-Lietz, C. 2003. Universal capture: image-based facial animation for “the matrix reloaded”. In ACM SIGGRAPH 2003 Sketches & Applications, 1–1. Google Scholar
7. Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 4, 41:1–41:10. Google ScholarDigital Library
8. Cazals, F., and Pouget, M. 2003. Estimating differential quantities using polynomial fitting of osculating jets. In Proceedings of the 2003 Eurographics/ACM SIGGRAPH symposium on Geometry processing, 177–187. Google ScholarDigital Library
9. Chai, J., and Hodgins, J. 2007. Constraint-based motion optimization using a statistical dynamic model. ACM Transactions on Graphics 26, 3, 8:1–8:9. Google ScholarDigital Library
10. Chai, J., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3D facial animation. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 193–206. Google ScholarDigital Library
11. DeCarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. International Journal of Computer Vision. 38(2):99–127. Google ScholarDigital Library
12. Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads using input from video. In Proceedings of Computer Animation Conference. 68–79. Google ScholarDigital Library
13. Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making Faces. In Proceedings of ACM SIGGRAPH 1998, 55–66. Google Scholar
14. James, D. L., and Twigg, C. D. 2005. Skinning mesh animations. ACM Transactions on Graphics 24, 3, 399–407. Google ScholarDigital Library
15. Lau, M., Chai, J., Xu, Y.-Q., and Shum, H. 2009. Face poser: Interactive modeling of 3d facial expressions using facial priors. ACM Transactions on Graphics 29, 1, 3:1–3:17. Google ScholarDigital Library
16. Li, H., Adams, B., Guibas, L. J., and Pauly, M. 2009. Robust single-view geometry and motion reconstruction. ACM Trans. Graph. 28, 5, 175:1–175:10. Google ScholarDigital Library
17. Li, H., Weise, T., and Pauly, M. 2010. Example-based facial rigging. ACM Transactions on Graphics 29, 4, 32:1–32:6. Google ScholarDigital Library
18. Ma, W.-C., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., and Debevec, P. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graph. 27, 5, 121:1–121:10. Google ScholarDigital Library
19. Papenberg, N., Bruhn, A., Brox, T., Didas, S., and Weickert, J. 2006. Highly accurate optic flow computation with theoretically justified warping. Int. J. Comput. Vision 67, 2, 141–158. Google ScholarDigital Library
20. Pighin, F., and Lewis, J. P. 2006. Facial motion retargeting. In ACM SIGGRAPH 2006 Courses, SIGGRAPH ’06. Google Scholar
21. Pighin, F., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3D model-based tracking. In International Conference on Computer Vision. 143–150.Google Scholar
22. Sorkine, O., Cohen-Or, D., Lipman, Y., Alexa, M., Rössl, C., and Seidel, H.-P. 2004. Laplacian surface editing. In SGP ’04: Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, 175–184. Google Scholar
23. Sumner, R. W., Schmid, J., and Pauly, M. Embedded deformation for shape manipulation. ACM Trans. Graph. 26, 3, 80:1–80:7. Google Scholar
24. Vicon Systems, 2011. http://www.vicon.com.Google Scholar
25. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Transactions on Graphics 24, 3, 426–433. Google ScholarDigital Library
26. Williams, L. 1990. Performance driven facial animation. In Proceedings of ACM SIGGRAPH 1990. 24(4):235–242. Google ScholarCross Ref
27. XYZ RGB Systems, 2011. http://www.xyzrgb.com/.Google Scholar
28. Zhang, L., Snavely, N., Curless, B., and Seitz, S. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3, 548–558. Google ScholarDigital Library