“Face swapping: automatically replacing faces in photographs” by Bitouk, Kumar, Dhillon, Belhumeur and Nayar

In this paper, we present a complete system for automatic face replacement in images. Our system uses a large library of face images created automatically by downloading images from the internet, extracting faces using face detection software, and aligning each extracted face to a common coordinate system. This library is constructed off-line, once, and can be efficiently accessed during face replacement. Our replacement algorithm has three main stages. First, given an input image, we detect all faces that are present, align them to the coordinate system used by our face library, and select candidate face images from our face library that are similar to the input face in appearance and pose. Second, we adjust the pose, lighting, and color of the candidate face images to match the appearance of those in the input image, and seamlessly blend in the results. Third, we rank the blended candidate replacements by computing a match distance over the overlap region. Our approach requires no 3D model, is fully automatic, and generates highly plausible results across a wide range of skin tones, lighting conditions, and viewpoints. We show how our approach can be used for a variety of applications including face de-identification and the creation of appealing group photographs from a set of images. We conclude with a user study that validates the high quality of our replacement results, and a discussion on the current limitations of our system.

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive Digital Photomontage. ACM Transactions on Graphics 23, 294–302. Google ScholarDigital Library
2. Avidan, S., and Shamir, A. 2007. Seam carving for contentaware image resizing. ACM Transactions on Graphics 26. Google ScholarDigital Library
3. Basri, R., and Jacobs, D. 2003. Lambertian reflectance and linear subspaces. IEEE TPAMI 25, 218–233. Google ScholarDigital Library
4. Blanz, V., Scherbaum, K., Vetter, T., and Seidel, H.-P. 2004. Exchanging Faces in Images. Computer Graphics Forum 23, 669–676.Google ScholarCross Ref
5. Boyle, M., Edwards, C., and Greenberg, S. 2000. The Effects of Filtered Video on Awareness and Privacy. In ACM Conference on Computer Supported Cooperative Work. Google ScholarDigital Library
6. Cootes, T., Edwards, G., and Taylor, C. 2001. Active Appearance Models. IEEE TPAMI 26, 681–685. Google ScholarDigital Library
7. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the Reflectance Field of a Human Face. In SIGGRAPH 00, 145–156. Google ScholarDigital Library
8. Debevec, P. 1998. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In SIGGRAPH 98, 189–198. Google ScholarDigital Library
9. Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In SIGGRAPH 01, 341–346. Google ScholarDigital Library
10. Fergus, R., Singh, B., Hertzmann, A., Roweis, S., and Freeman, W. 2006. Removing camera shake from a single photograph. SIGGRAPH 06, 787–794. Google ScholarDigital Library
11. Gross, R., Sweeney, L., de la Torre, F., and Baker, S. 2006. Model-Based Face De-Identification. 161–168. Google ScholarDigital Library
12. Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Transactions on Graphics 26, 3. Google ScholarDigital Library
13. Kundur, D., and Hatzinakos, D. 1996. Blind image deconvolution. IEEE Signal Processing Magazine, 3, 43–64.Google ScholarCross Ref
14. Lanitis, I., Draganova, C., and Christodoulou, C. 2004. Comparing different classifiers for automatic age estimation. IEEE Trans. on Systems, Man, and Cybernetics, B 34, 621–628. Google ScholarDigital Library
15. Liu, Z., Shan, Y., and Zhang, Z. 2001. Expressive expression mapping with ratio images. In SIGGRAPH 01: Proc. of the 28th CGIT, 271–276. Google ScholarDigital Library
16. Malik, S., 2003. Digital face replacement in photographs. http://www.cs.toronto.edu/~smalik/2530/project/results.html.Google Scholar
17. Moghaddam, B., and Yang, M.-H. 2002. Learning gender with support faces. IEEE TPAMI 24, 707–711. Google ScholarDigital Library
18. Newton, E., Sweeney, L., and Malin, B. 2005. Preserving Privacy by De-Identifying Face Images. IEEE Trans. on Knowledge and Data Eng., 232–243. Google ScholarDigital Library
19. Omron, 2007. OKAO vision. http://omron.com/rd/vision/01.html.Google Scholar
20. Ramamoorthi, R., and Hanrahan, P. 2001. An Efficient Representation for Irradiance Environment Maps. In SIGGRAPH 01, 497–500. Google ScholarDigital Library
21. Rubner, Y., Tomasi, C., and Guibas, L. J. 2000. The earth mover’s distance as a metric for image retrieval. IJCV, 99–121. Google ScholarDigital Library
22. Wang, H., Raskar, R., Xu, N., and Ahuja, N. 2007. Videoshop: A New Framework for Video Editing in Gradient Domain. Graphical Models 69, 57–70. Google ScholarDigital Library
23. Wang, Y., Liu, Z., Hua, G., Wen, Z., Zhang, Z., and Samaras, D. 2007. Face re-lighting from a single image under harsh lighting conditions. CVPR ’07.Google Scholar
24. Wen, Z., Liu, Z., and Huang., T. S. 2003. Face Relighting with Radiance Environment Maps. In CVPR ’03, 158–165.Google Scholar