“Post-production facial performance relighting using reflectance transfer” by Peers, Tamura, Matusik and Debevec

  • ©Pieter Peers, Naoki Tamura, Wojciech Matusik, and Paul E. Debevec

Conference:


Type(s):


Title:

    Post-production facial performance relighting using reflectance transfer

Presenter(s)/Author(s):


Abstract:


    We propose a novel post-production facial performance relighting system for human actors. Our system uses just a dataset of view-dependent facial appearances with a neutral expression, captured for a static subject using a Light Stage apparatus. For the actual performance, however, a potentially different actor is captured under known, but static, illumination. During post-production, the reflectance field of the reference dataset actor is transferred onto the dynamic performance, enabling image-based relighting of the entire sequence. Our approach makes post-production relighting more practical and could easily be incorporated in a traditional production pipeline since it does not require additional hardware during principal photography. Additionally, we show that our system is suitable for real-time post-production illumination editing.

References:


    1. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of SIGGRAPH 99, Computer Graphics Proceedings, Annual Conference Series, 187–194. Google ScholarDigital Library
    2. Blanz, V., Basso, C., Poggio, T., and Vetter, T. 2003. Reanimating faces in images and video. Computer Graphics Forum 22, 3 (Sept.), 641–650.Google ScholarCross Ref
    3. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proceedings of the 8th European Conference on Computer Vision, 25–36.Google Scholar
    4. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, 145–156. Google ScholarDigital Library
    5. Debevec, P. 1998. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, 189–198. Google ScholarDigital Library
    6. Einarsson, P., Chabert, C.-F., Jones, A., Ma, W.-C., Lamond, B., Hawkins, T., Bolas, M., Sylwan, S., and Debevec, P. 2006. Relighting human locomotion with flowed reflectance fields. In Rendering Techniques 2006: 17th Eurographics Workshop on Rendering, 183–194. Google ScholarCross Ref
    7. Eisemann, E., and Durand, F. 2004. Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics 23, 3, 673–678. Google ScholarDigital Library
    8. FUCHS, M., Blanz, V., Lensch, H., and Seidel, H.-P. 2005. Reflectance from images: a model-based approach for human faces. IEEE Transactions on Visualization and Computer Graphics 11, 3 (May-June), 296–305. Google ScholarDigital Library
    9. Georghiades, A. S. 2003. Recovering 3-D shape and reflectance from a small number of photographs. In Eurographics Symposium on Rendering: 14th Eurographics Workshop on Rendering, 230–240. Google ScholarDigital Library
    10. Hawkins, T., Wenger, A., Tchou, C., Gardner, A., Göransson, F., and Debevec, P. 2004. Animatable facial reflectance fields. In Rendering Techniques 2004: 15th Eurographics Workshop on Rendering, 309–320. Google ScholarCross Ref
    11. Jones, M., and Viola, P. 2003. Fast multi-view face detection. Tech. rep., MERL. TR2003–096.Google Scholar
    12. Liu, Z., Shan, Y., and Zhang, Z. 2001. Expressive expression mapping with ratio images. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 271–276. Google ScholarDigital Library
    13. LUCAS, B., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In International Joint Conference on Artificial Intelligence (IJCAI81), 674–679.Google Scholar
    14. Marschner, S. R., and Greenberg, D. P. 1997. Inverse lighting for photography. In Proceedings of the Fifth Color Imaging Conference, Society for Imaging Science and Technology, 262–265.Google Scholar
    15. Marschner, S., Westin, S., Lafortune, E., Torrance, K., and Greenberg, D. 1999. Image-based BRDF measurement including human skin. In Rendering Techniques ’99: 10th Eurographics Workshop on Rendering, 139–152. Google ScholarCross Ref
    16. Nishino, K., and Nayar, S. K. 2004. Eyes for relighting. ACM Transactions on Graphics 23, 3 (Aug.), 704–711. Google ScholarDigital Library
    17. Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., and Toyama, K. 2004. Digital photography with flash and no-flash image pairs. ACM Transactions Graphics 23, 3, 664–672. Google ScholarDigital Library
    18. Pighin, F. H., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3D model-based tracking. In International Conference on Computer Vision, 143–150.Google Scholar
    19. Riklin-Raviv, T., and Shashua, A. 1999. The quotient image:class based recognition and synthesis under varying illumination conditions. IEEE Transactions on Pattern Analysis and Machine Intelligence 02, 262–265.Google Scholar
    20. Sim, T., and Kanade, T. 2002. Illuminating the face. Tech. rep., CMU. CMU-RI-TR-01-31.Google Scholar
    21. Stoschek, A. 2000. Image-based re-rendering of faces for continuous pose and illumination directions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 582–587.Google ScholarCross Ref
    22. Terzopoulos, D., and Waters, K. 1993. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Transactions on Pattern Analysis and Machine Intelligence 15, 6 (June), 569–579. Google ScholarDigital Library
    23. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Transactions on Graphics 24, 3 (Aug.), 426–433. Google ScholarDigital Library
    24. Wen, Z., Liu, Z., and Huang, T. 2003. Face relighting with radiance environment maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 157–165.Google Scholar
    25. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., and Debevec, P. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics 24, 3 (Aug.), 756–764. Google ScholarDigital Library
    26. Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., McAndless, J., Lee, J., Ngan, A., Jensen, H. W., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Transactions on Graphics 25, 3 (July), 1013–1024. Google ScholarDigital Library
    27. Williams, L. 1990. Performance-driven facial animation. In Computer Graphics (Proceedings of SIGGRAPH 90), vol. 24, 235–242. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: