“High resolution passive facial performance capture” by Bradley, Heidrich, Popa and Sheffer

  • ©Derek Bradley, Wolfgang Heidrich, Tiberiu Popa, and Alla Sheffer




    High resolution passive facial performance capture



    We introduce a purely passive facial capture approach that uses only an array of video cameras, but requires no template facial geometry, no special makeup or markers, and no active lighting. We obtain initial geometry using multi-view stereo, and then use a novel approach for automatically tracking texture detail across the frames. As a result, we obtain a high-resolution sequence of compatibly triangulated and parameterized meshes. The resulting sequence can be rendered with dynamically captured textures, while also consistently applying texture changes such as virtual makeup.


    1. Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital emily project: photoreal facial modeling and animation. In ACM SIGGRAPH Courses, 1–15. Google ScholarDigital Library
    2. Beeler, T., Bickel, B., Sumner, R., Beardsley, P., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH). Google ScholarDigital Library
    3. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graphics (Proc. SIGGRAPH), 33. Google ScholarDigital Library
    4. Blanz, V., Basso, C., Vetter, T., and Poggio, T. 2003. Reanimating faces in images and video. Computer Graphics Forum (Proc. Eurographics) 22, 3, 641–650.Google ScholarCross Ref
    5. Borshukov, G., Piponi, D., Larsen, O., Lewis, J. P., and Tempelaar-Lietz, C. 2003. Universal capture: image-based facial animation for “the matrix reloaded”. In ACM SIGGRAPH Sketches & Applications. Google ScholarDigital Library
    6. Bouguet, J.-Y. 1999. Pyramidal implementation of the lucas kanade feature tracker: Description of the algorithm. Tech. rep., Intel Corporation, Microprocessor Research Labs.Google Scholar
    7. Bradley, D., and Heidrich, W. 2010. Binocular camera calibration using rectification error. In Canadian Conference on Computer and Robot Vision. Google ScholarDigital Library
    8. Bradley, D., Boubekeur, T., and Heidrich, W. 2008. Accurate multi-view reconstruction using robust binocular stereo and surface meshing. In Proc. CVPR.Google Scholar
    9. Bradley, D., Popa, T., Sheffer, A., Heidrich, W., and Boubekeur, T. 2008. Markerless garment capture. ACM Trans. Graphics (Proc. SIGGRAPH) 27, 3, 99. Google ScholarDigital Library
    10. Bradley, D., Atcheson, B., Ihrke, I., and Heidrich, W. 2009. Synchronization and rolling shutter compensation for consumer video camera arrays. In International Workshop on Projector-Camera Systems (PROCAMS 2009).Google Scholar
    11. Canny, J. 1986. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 6, 679–698. Google ScholarDigital Library
    12. de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., eidel, H.-P. S., and Thrun, S. 2008. Performance capture from sparse multi-view video. ACM Trans. Graphics (Proc. SIGGRAPH), 98. Google ScholarDigital Library
    13. DeCarlo, D., and Metaxas, D. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In CVPR, 231. Google ScholarDigital Library
    14. DeCarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. Journal of Computer Vision 38, 2, 99–127. Google ScholarDigital Library
    15. Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads using input from video. In Proceedings of Computer Animation, 68. Google ScholarDigital Library
    16. Fiala, M., and Shu, C. 2008. Self-identifying patterns for plane-based camera calibration. Machine Vision and Applications 19, 4, 209–216. Google ScholarDigital Library
    17. Furukawa, Y., and Ponce, J. 2009. Dense 3d motion capture for human faces. In CVPR.Google Scholar
    18. Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In SIGGRAPH ’98: Proceedings of the 25th annual conference on Computer graphics and interactive techniques, 55–66. Google ScholarDigital Library
    19. Kadir, T., and Brady, M. 2001. Saliency, scale and image description. Int. J. Comput. Vision 45, 2, 83–105. Google ScholarDigital Library
    20. Lévy, B., Petitjean, S., Ray, N., and Maillot, J. 2002. Least squares conformal maps for automatic texture atlas generation. ACM Trans. Graph. 21, 3, 362–371. Google ScholarDigital Library
    21. Li, H., Roivainen, P., and Forcheimer, R. 1993. 3-d motion estimation in model-based facial image coding. IEEE Trans. Pattern Anal. Mach. Intell. 15, 6, 545–555. Google ScholarDigital Library
    22. Lin, I.-C., and Ouhyoung, M. 2005. Mirror mocap: Automatic and efficient capture of dense 3d facial motion parameters from video. The Visual Computer 21, 6, 355–372.Google ScholarCross Ref
    23. Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Eurographics Symposium on Rendering, 183–194. Google ScholarDigital Library
    24. Ma, W.-C., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., and Debevec, P. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 27, 5, 121. Google ScholarDigital Library
    25. Mohammed, U., Prince, S. J. D., and Kautz, J. 2009. Visiolization: generating novel facial images. ACM Trans. Graph. 28, 3, 57. Google ScholarDigital Library
    26. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3, 313–318. Google ScholarDigital Library
    27. Pighin, F. H., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3d model-based tracking. In ICCV, 143–150.Google Scholar
    28. Shewchuk, J. 1996. Triangle: Engineering a 2D Quality Mesh Generator and Delaunay Triangulator. In Applied Computational Geometry: Towards Geometric Engineering, M. C. Lin and D. Manocha, Eds., vol. 1148 of Lecture Notes in Computer Science. Springer-Verlag, 203–222. Google ScholarDigital Library
    29. Wang, Y., Huang, X., Lee, C.-S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dyanmic 3-d facial expressions. In Computer Graphics Forum, 677–686.Google Scholar
    30. Williams, L. 1990. Performance-driven facial animation. SIGGRAPH Comput. Graph. 24, 4, 235–242. Google ScholarDigital Library
    31. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Trans. Graphics (Proc. SIGGRAPH), 548–558. Google ScholarDigital Library

ACM Digital Library Publication: