“Post-production facial performance relighting using reflectance transfer” by Peers, Tamura, Matusik and Debevec

We propose a novel post-production facial performance relighting system for human actors. Our system uses just a dataset of view-dependent facial appearances with a neutral expression, captured for a static subject using a Light Stage apparatus. For the actual performance, however, a potentially different actor is captured under known, but static, illumination. During post-production, the reflectance field of the reference dataset actor is transferred onto the dynamic performance, enabling image-based relighting of the entire sequence. Our approach makes post-production relighting more practical and could easily be incorporated in a traditional production pipeline since it does not require additional hardware during principal photography. Additionally, we show that our system is suitable for real-time post-production illumination editing.

1. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of SIGGRAPH 99, Computer Graphics Proceedings, Annual Conference Series, 187–194. Google ScholarDigital Library
2. Blanz, V., Basso, C., Poggio, T., and Vetter, T. 2003. Reanimating faces in images and video. Computer Graphics Forum 22, 3 (Sept.), 641–650.Google ScholarCross Ref
3. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proceedings of the 8th European Conference on Computer Vision, 25–36.Google Scholar
4. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, 145–156. Google ScholarDigital Library
5. Debevec, P. 1998. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, 189–198. Google ScholarDigital Library
6. Einarsson, P., Chabert, C.-F., Jones, A., Ma, W.-C., Lamond, B., Hawkins, T., Bolas, M., Sylwan, S., and Debevec, P. 2006. Relighting human locomotion with flowed reflectance fields. In Rendering Techniques 2006: 17th Eurographics Workshop on Rendering, 183–194. Google ScholarCross Ref
7. Eisemann, E., and Durand, F. 2004. Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics 23, 3, 673–678. Google ScholarDigital Library
8. FUCHS, M., Blanz, V., Lensch, H., and Seidel, H.-P. 2005. Reflectance from images: a model-based approach for human faces. IEEE Transactions on Visualization and Computer Graphics 11, 3 (May-June), 296–305. Google ScholarDigital Library
9. Georghiades, A. S. 2003. Recovering 3-D shape and reflectance from a small number of photographs. In Eurographics Symposium on Rendering: 14th Eurographics Workshop on Rendering, 230–240. Google ScholarDigital Library
10. Hawkins, T., Wenger, A., Tchou, C., Gardner, A., Göransson, F., and Debevec, P. 2004. Animatable facial reflectance fields. In Rendering Techniques 2004: 15th Eurographics Workshop on Rendering, 309–320. Google ScholarCross Ref
11. Jones, M., and Viola, P. 2003. Fast multi-view face detection. Tech. rep., MERL. TR2003–096.Google Scholar
12. Liu, Z., Shan, Y., and Zhang, Z. 2001. Expressive expression mapping with ratio images. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 271–276. Google ScholarDigital Library
13. LUCAS, B., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In International Joint Conference on Artificial Intelligence (IJCAI81), 674–679.Google Scholar
14. Marschner, S. R., and Greenberg, D. P. 1997. Inverse lighting for photography. In Proceedings of the Fifth Color Imaging Conference, Society for Imaging Science and Technology, 262–265.Google Scholar
15. Marschner, S., Westin, S., Lafortune, E., Torrance, K., and Greenberg, D. 1999. Image-based BRDF measurement including human skin. In Rendering Techniques ’99: 10th Eurographics Workshop on Rendering, 139–152. Google ScholarCross Ref
16. Nishino, K., and Nayar, S. K. 2004. Eyes for relighting. ACM Transactions on Graphics 23, 3 (Aug.), 704–711. Google ScholarDigital Library
17. Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., and Toyama, K. 2004. Digital photography with flash and no-flash image pairs. ACM Transactions Graphics 23, 3, 664–672. Google ScholarDigital Library
18. Pighin, F. H., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3D model-based tracking. In International Conference on Computer Vision, 143–150.Google Scholar
19. Riklin-Raviv, T., and Shashua, A. 1999. The quotient image:class based recognition and synthesis under varying illumination conditions. IEEE Transactions on Pattern Analysis and Machine Intelligence 02, 262–265.Google Scholar
20. Sim, T., and Kanade, T. 2002. Illuminating the face. Tech. rep., CMU. CMU-RI-TR-01-31.Google Scholar
21. Stoschek, A. 2000. Image-based re-rendering of faces for continuous pose and illumination directions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 582–587.Google ScholarCross Ref
22. Terzopoulos, D., and Waters, K. 1993. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Transactions on Pattern Analysis and Machine Intelligence 15, 6 (June), 569–579. Google ScholarDigital Library
23. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Transactions on Graphics 24, 3 (Aug.), 426–433. Google ScholarDigital Library
24. Wen, Z., Liu, Z., and Huang, T. 2003. Face relighting with radiance environment maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 157–165.Google Scholar
25. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., and Debevec, P. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics 24, 3 (Aug.), 756–764. Google ScholarDigital Library
26. Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., McAndless, J., Lee, J., Ngan, A., Jensen, H. W., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Transactions on Graphics 25, 3 (July), 1013–1024. Google ScholarDigital Library
27. Williams, L. 1990. Performance-driven facial animation. In Computer Graphics (Proceedings of SIGGRAPH 90), vol. 24, 235–242. Google ScholarDigital Library