“Visio-lization: generating novel facial images” by Mohammed, Prince and Kautz

Our goal is to generate novel realistic images of faces using a model trained from real examples. This model consists of two components: First we consider face images as samples from a texture with spatially varying statistics and describe this texture with a local non-parametric model. Second, we learn a parametric global model of all of the pixel values. To generate realistic faces, we combine the strengths of both approaches and condition the local non-parametric model on the global parametric model. We demonstrate that with appropriate choice of local and global models it is possible to reliably generate new realistic face images that do not correspond to any individual in the training data. We extend the model to cope with considerable intra-class variation (pose and illumination). Finally, we apply our model to editing real facial images: we demonstrate image in-painting, interactive techniques for improving synthesized images and modifying facial expressions.

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Transactions on Graphics (Proc. SIGGRAPH) 23, 3, 294–302. Google ScholarDigital Library
2. Ashikhmin, M. 2001. Synthesizing natural textures. In Proc. ACM Symposium on Interactive 3D Graphics, 217–226. Google ScholarDigital Library
3. Bishop, C. 2006. Pattern Recognition and Machine Learning. Springer. Google ScholarDigital Library
4. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., and Nayar, S. K. 2008. Face swapping: automatically replacing faces in photographs. ACM Trans. Graph. 27, 3, 1–8. Google ScholarDigital Library
5. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of ACM SIGGRAPH 99, 187–194. Google ScholarDigital Library
6. Blanz, V., Albrecht, I., Haber, J., and Seidel, H.-P. 2006. Creating face models from vague mental images. Computer Graphics Forum 25, 3, 645–654.Google ScholarCross Ref
7. Brand, M., and Pletscher, P. 2008. A conditional random field for photo editing. In Proceedings of CVPR, 187–194.Google Scholar
8. Bregler, C., Covell, M., and Slaney, M. 1997. Video rewrite: Driving visual speech with audio. In Proceedings of ACM SIGGRAPH 97, 353–360. Google ScholarDigital Library
9. Dedeoglu, G., Kanade, T., and August, J. 2004. Highzoom video hallucination by exploiting spatio-temporal regularities. In Proceedings of CVPR, 151–158. Google ScholarDigital Library
10. Dempster, A. P., Laird, N. M., and Rubin, D. B. 1977. Maximum likelihood for incomplete data via the EM algorithm. Journal of the Royal Statistical Society 39 (B), 1, 1–38.Google Scholar
11. Diakopoulos, N., Essa, I., and Jain, R. 2004. Content based image synthesis. In CIVR 04, 299–307.Google Scholar
12. Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Proceedings of ACM SIGGRAPH, 341–346. Google ScholarDigital Library
13. Efros, A. A., and Leung, T. K. 1999. Texture synthesis by non-parametric sampling. In Proceedings of ICCV, vol. 2, 1033–1038. Google ScholarDigital Library
14. Ezzat, T., Geiger, G., and Poggio, T. 2002. Trainable videorealistic speech animation. In Proceedings of ACM SIGGRAPH 2002, 388–398. Google ScholarDigital Library
15. Ghahramani, Z., and Hinton, G. E. 1997. The EM algorithm for mixtures of factor analyzers. Technical Report CRG-TR-96-1, Dept. of Computer Science, University of Toronto, Canada.Google Scholar
16. Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Transactions on Graphics (Proc. SIGGRAPH) 26, 3, 4:1–4:7. Google ScholarDigital Library
17. Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics 22, 3, 277–286. Google ScholarDigital Library
18. Lalonde, J., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Transactions on Graphics (Proc. SIGGRAPH) 26, 3, 3:1–3:10. Google ScholarDigital Library
19. Liu, W., Lin, D., and Tang, X. 2005. Hallucinating faces: Tensorpatch super-resolution and coupled residue compensation. In Proceedings of CVPR, 478–484. Google ScholarDigital Library
20. Liu, C., Shum, H., and Freeman, W. 2007. Face hallucination: theory and practice. International Journal of Computer Vision 75, 1, 115–134. Google ScholarDigital Library
21. Messer, K., Matas, J., Kittler, J., Luettin, J., and Maitre, G. 1999. XM2VTSbd: The extended MTVTS database. In Proceedings 2nd Conference on Audio and Videobase Biometric Personal Verification (AVBPA99), 72–77.Google Scholar
22. Nguyen, M., Lalonde, J., Efros, A., and La Torre, F. D. 2008. Image-based shaving. Computer Graphics Forum (Eurographics) 27, 2, 627–635.Google ScholarCross Ref
23. Perez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Transactions on Graphics (Proc. SIGGRAPH) 22, 3, 313–318. Google ScholarDigital Library
24. Prince, S., Elder, J., Warrell, J., and Felisberti, F. 2008. Tied factor analysis for face recognition across large pose differences. IEEE Pattern Recognition and Machine Intelligence 30, 6, 970–984. Google ScholarDigital Library
25. Tenenbaum, J., and Freeman, W. 2000. Separating style and content with bilinear models. Neural Computation 12, 6, 1247–1283. Google ScholarDigital Library
26. Turk, M. A., and Pentland, A. P. 1991. Face recognition using eigenfaces. In Proceedings of CVPR, 586–591.Google Scholar
27. Vlasic, D., Brand, M., Pfister, H., and Popovic, J. 2005. Face transfer with multiliner models. ACM Transactions on Graphics (Proc. SIGGRAPH) 24, 3, 426–433. Google ScholarDigital Library
28. Wei, L., and Levoy, M. 2000. Fast texture synthesis using tree-structured vector quantization. In Proceedings of ACM SIGGRAPH 2000, 479–488. Google ScholarDigital Library
29. Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., McAndless, J., Lee, J., Ngan, A., Jensen, H., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Transactions on Graphics (Proc. SIGGRAPH) 25, 3, 1013–1024. Google ScholarDigital Library