“Expression flow for 3D-aware face component transfer” by Yang, Wang, Shechtman, Bourdev and Metaxas

We address the problem of correcting an undesirable expression on a face photo by transferring local facial components, such as a smiling mouth, from another face photo of the same person which has the desired expression. Direct copying and blending using existing compositing tools results in semantically unnatural composites, since expression is a global effect and the local component in one expression is often incompatible with the shape and other components of the face in another expression. To solve this problem we present Expression Flow, a 2D flow field which can warp the target face globally in a natural way, so that the warped face is compatible with the new facial component to be copied over. To do this, starting with the two input face photos, we jointly construct a pair of 3D face shapes with the same identity but different expressions. The expression flow is computed by projecting the difference between the two 3D shapes back to 2D. It describes how to warp the target face photo to match the expression of the reference photo. User studies suggest that our system is able to generate face composites with much higher fidelity than existing methods.

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. In Proceedings of ACM SIGGRAPH, vol. 23, 294–302. Google ScholarDigital Library
2. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., and Nayar, S. K. 2008. Face swapping: automatically replacing faces in photographs. In Proceedings of ACM SIGGRAPH, 39:1–39:8. Google ScholarDigital Library
3. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of ACM SIGGRAPH, 187–194. Google Scholar
4. Blanz, V., Basso, C., Poggio, T., and Vetter. 2003. Rean-imating faces in images and video. Computer Graphics Forum 22, 3.Google ScholarCross Ref
5. Blanz, V., Scherbaum, K., Vetter, T., and Seidel, H.-P. 2004. Exchanging faces in images. Computer Graphics Forum 23, 3, 669–676.Google ScholarCross Ref
6. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Analysis and Machine Intelligence 23. Google Scholar
7. Cootes, T., Taylor, C., Cooper, D., and Graham, J. 1995. Active shape models: Their training and application. Computer Vision and Image Understanding 61, 1, 38–59. Google ScholarDigital Library
8. Cootes, T. F., Edwards, G. J., and Taylor, C. J. 2001. Active appearance models. IEEE Trans. Pattern Analysis and Machine Intelligence 23, 681–685. Google ScholarDigital Library
9. Decarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. J. Comput. Vision 38, 99–127. Google ScholarCross Ref
10. Dovgard, R., and Basri, R. 2004. Statistical symmetric shape from shading for 3d structure recovery of faces. In Proceedings of ECCV, 99–113.Google Scholar
11. Faigin, G. 1991. The Artist’s Complete Guide to Facial Expression. Watson-Guptill Publications Inc., New York.Google Scholar
12. Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., and Lischinski, D. 2009. Coordinates for instant image cloning. In Proceedings of ACM SIGGRAPH, vol. 28, 67:1–67:9. Google ScholarDigital Library
13. Fattal, R. 2009. Edge-avoiding wavelets and their applications. In Proceedings of ACM SIGGRAPH, vol. 28. Google ScholarDigital Library
14. Felzenszwalb, P. F., and Huttenlocher, D. P. 2005. Pictorial structures for object recognition. International Journal of Computer Vision 61, 55–79. Google ScholarDigital Library
15. Joshi, N., Matusik, W., Adelson, E. H., and Kriegman, D. J. 2010. Personal photo enhancement using example images. ACM Trans. Graphics 29, 12:1–12:15. Google ScholarDigital Library
16. Leyvand, T., Cohen-Or, D., Dror, G., and Lischinski, D. 2008. Data-driven enhancement of facial attractiveness. In Proceedings of ACM SIGGRAPH, 38:1–38:9. Google ScholarDigital Library
17. Liang, L., Xiao, R., Wen, F., and Sun, J. 2008. Face alignment via component-based discriminative search. In Proceedings of ECCV. Google Scholar
18. Liu, Z., Shan, Y., and Zhang, Z. 2001. Expressive expression mapping with ratio images. In Proceedings of ACM SIGGRAPH, 271–276. Google Scholar
19. Liu, C., Shum, H.-Y., and Freeman, W. T. 2007. Face hallucination: Theory and practice. International Journal of Computer Vision 75, 115–134. Google ScholarDigital Library
20. Lucas, B. D., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proceedings of the 1981 DARPA Image Understanding Workshop, 121–130. Google ScholarDigital Library
21. Milborrow, S., and Nicolls, F. 2008. Locating facial features with an extended active shape model. In Proceedings of ECCV, 504–513. Google Scholar
22. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. In Proceedings of ACM SIGGRAPH, 313–318. Google Scholar
23. Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. H. 1998. Synthesizing realistic facial expressions from photographs. In Proceedings of ACM SIGGRAPH, 75–84. Google Scholar
24. Romdhani, S., and Vetter, T. 2003. Efficient, robust and accurate fitting of a 3d morphable model. In Proceedings of ICCV, 59–66. Google ScholarDigital Library
25. Saragih, J., Lucey, S., and Cohn, J. 2009. Face alignment through subspace constrained mean-shifts. In Proceedings of the 12th ICCV, 1034–1041.Google Scholar
26. Shlizerman, I. K., and Basri, R. 2011. 3d face reconstruction from a single image using a single reference face shape. IEEE Trans. Pattern Analysis and Machine Intelligence 33, 394–405. Google ScholarDigital Library
27. Shlizerman, I. K., Sankar, A., Shechtman, E., and Seitz, S. M. 2010. Being john malkovich. In Proceedings of ECCV, 341–353. Google ScholarDigital Library
28. Singular Inversions Inc. 2009. Facegen modeller manual. In www.facegen.com.Google Scholar
29. Sunkavalli, K., Johnson, M. K., Matusik, W., and Pfister, H. 2010. Multi-scale image harmonization. In Proceedings of ACM SIGGRAPH, vol. 29. Google ScholarDigital Library
30. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. In Proceedings of ACM SIGGRAPH, vol. 24, 426–433. Google ScholarDigital Library
31. Wang, Y., Huang, X., Lee, C.-S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dynamic 3-d facial expressions. In Proceedings of EuroGraphics, 677–686.Google Scholar
32. Williams, L. 1990. Performance-driven facial animation. In Proceedings of ACM SIGGRAPH, vol. 24, 235–242. Google ScholarCross Ref
33. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High-resolution capture for modeling and animation. In Proceedings of ACM SIGGRAPH, 548–558. Google Scholar