“High resolution passive facial performance capture” by Bradley, Heidrich, Popa and Sheffer

We introduce a purely passive facial capture approach that uses only an array of video cameras, but requires no template facial geometry, no special makeup or markers, and no active lighting. We obtain initial geometry using multi-view stereo, and then use a novel approach for automatically tracking texture detail across the frames. As a result, we obtain a high-resolution sequence of compatibly triangulated and parameterized meshes. The resulting sequence can be rendered with dynamically captured textures, while also consistently applying texture changes such as virtual makeup.

1. Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital emily project: photoreal facial modeling and animation. In ACM SIGGRAPH Courses, 1–15. Google ScholarDigital Library
2. Beeler, T., Bickel, B., Sumner, R., Beardsley, P., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH). Google ScholarDigital Library
3. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graphics (Proc. SIGGRAPH), 33. Google ScholarDigital Library
4. Blanz, V., Basso, C., Vetter, T., and Poggio, T. 2003. Reanimating faces in images and video. Computer Graphics Forum (Proc. Eurographics) 22, 3, 641–650.Google ScholarCross Ref
5. Borshukov, G., Piponi, D., Larsen, O., Lewis, J. P., and Tempelaar-Lietz, C. 2003. Universal capture: image-based facial animation for “the matrix reloaded”. In ACM SIGGRAPH Sketches & Applications. Google ScholarDigital Library
6. Bouguet, J.-Y. 1999. Pyramidal implementation of the lucas kanade feature tracker: Description of the algorithm. Tech. rep., Intel Corporation, Microprocessor Research Labs.Google Scholar
7. Bradley, D., and Heidrich, W. 2010. Binocular camera calibration using rectification error. In Canadian Conference on Computer and Robot Vision. Google ScholarDigital Library
8. Bradley, D., Boubekeur, T., and Heidrich, W. 2008. Accurate multi-view reconstruction using robust binocular stereo and surface meshing. In Proc. CVPR.Google Scholar
9. Bradley, D., Popa, T., Sheffer, A., Heidrich, W., and Boubekeur, T. 2008. Markerless garment capture. ACM Trans. Graphics (Proc. SIGGRAPH) 27, 3, 99. Google ScholarDigital Library
10. Bradley, D., Atcheson, B., Ihrke, I., and Heidrich, W. 2009. Synchronization and rolling shutter compensation for consumer video camera arrays. In International Workshop on Projector-Camera Systems (PROCAMS 2009).Google Scholar
11. Canny, J. 1986. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 6, 679–698. Google ScholarDigital Library
12. de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., eidel, H.-P. S., and Thrun, S. 2008. Performance capture from sparse multi-view video. ACM Trans. Graphics (Proc. SIGGRAPH), 98. Google ScholarDigital Library
13. DeCarlo, D., and Metaxas, D. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In CVPR, 231. Google ScholarDigital Library
14. DeCarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. Journal of Computer Vision 38, 2, 99–127. Google ScholarDigital Library
15. Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads using input from video. In Proceedings of Computer Animation, 68. Google ScholarDigital Library
16. Fiala, M., and Shu, C. 2008. Self-identifying patterns for plane-based camera calibration. Machine Vision and Applications 19, 4, 209–216. Google ScholarDigital Library
17. Furukawa, Y., and Ponce, J. 2009. Dense 3d motion capture for human faces. In CVPR.Google Scholar
18. Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In SIGGRAPH ’98: Proceedings of the 25th annual conference on Computer graphics and interactive techniques, 55–66. Google ScholarDigital Library
19. Kadir, T., and Brady, M. 2001. Saliency, scale and image description. Int. J. Comput. Vision 45, 2, 83–105. Google ScholarDigital Library
20. Lévy, B., Petitjean, S., Ray, N., and Maillot, J. 2002. Least squares conformal maps for automatic texture atlas generation. ACM Trans. Graph. 21, 3, 362–371. Google ScholarDigital Library
21. Li, H., Roivainen, P., and Forcheimer, R. 1993. 3-d motion estimation in model-based facial image coding. IEEE Trans. Pattern Anal. Mach. Intell. 15, 6, 545–555. Google ScholarDigital Library
22. Lin, I.-C., and Ouhyoung, M. 2005. Mirror mocap: Automatic and efficient capture of dense 3d facial motion parameters from video. The Visual Computer 21, 6, 355–372.Google ScholarCross Ref
23. Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Eurographics Symposium on Rendering, 183–194. Google ScholarDigital Library
24. Ma, W.-C., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., and Debevec, P. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 27, 5, 121. Google ScholarDigital Library
25. Mohammed, U., Prince, S. J. D., and Kautz, J. 2009. Visiolization: generating novel facial images. ACM Trans. Graph. 28, 3, 57. Google ScholarDigital Library
26. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3, 313–318. Google ScholarDigital Library
27. Pighin, F. H., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3d model-based tracking. In ICCV, 143–150.Google Scholar
28. Shewchuk, J. 1996. Triangle: Engineering a 2D Quality Mesh Generator and Delaunay Triangulator. In Applied Computational Geometry: Towards Geometric Engineering, M. C. Lin and D. Manocha, Eds., vol. 1148 of Lecture Notes in Computer Science. Springer-Verlag, 203–222. Google ScholarDigital Library
29. Wang, Y., Huang, X., Lee, C.-S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dyanmic 3-d facial expressions. In Computer Graphics Forum, 677–686.Google Scholar
30. Williams, L. 1990. Performance-driven facial animation. SIGGRAPH Comput. Graph. 24, 4, 235–242. Google ScholarDigital Library
31. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Trans. Graphics (Proc. SIGGRAPH), 548–558. Google ScholarDigital Library