“Performance relighting and reflectance transformation with time-multiplexed illumination” by Wenger, Gardner, Tchou, Unger, Hawkins, et al. …

  • ©Andreas Wenger, Andrew Gardner, Chris Tchou, Jonas Unger, Tim Hawkins, and Paul E. Debevec




    Performance relighting and reflectance transformation with time-multiplexed illumination



    We present a technique for capturing an actor’s live-action performance in such a way that the lighting and reflectance of the actor can be designed and modified in postproduction. Our approach is to illuminate the subject with a sequence of time-multiplexed basis lighting conditions, and to record these conditions with a high-speed video camera so that many conditions are recorded in the span of the desired output frame interval. We investigate several lighting bases for representing the sphere of incident illumination using a set of discrete LED light sources, and we estimate and compensate for subject motion using optical flow and image warping based on a set of tracking frames inserted into the lighting basis. To composite the illuminated performance into a new background, we include a time-multiplexed matte within the basis. We also show that the acquired data enables time-varying surface normals, albedo, and ambient occlusion to be estimated, which can be used to transform the actor’s reflectance to produce both subtle and stylistic effects.


    1. Black, M. J., and Anandan, P. 1993. A framework for the robust estimation of optical flow. In Fourth International Conf. on Computer Vision, 231–236.Google Scholar
    2. Brostow, G. J., and Essa. I. 2001. Image-based motion blur for stop motion animation. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 561–566. Google ScholarDigital Library
    3. Calahan, S. 2000. Advanced Renderman: Creating CGI for Motion Pictures. Morgan Kaufman Publishers, San Francisco, ch. Storytelling through lighting, a computer perspective, 337382.Google Scholar
    4. Debevec. P., Hawkins, T., Tchou. C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. Proceedings of SIGGRAPH 2000 (July), 145–156. Google ScholarDigital Library
    5. Debevec, P., Wenger, A., Tchou, C., Gardner, A., Waese, J., and Hawkins, T. 2002. A lighting reproduction approach to live-action compositing. ACM Transactions on Graphics 21, 3 (July), 547–556. Google ScholarDigital Library
    6. Fielding, R. 1985. The Technique of Special Effects Cinematography, 4th ed. Hastings House, New York.Google Scholar
    7. Georghiades. A., Belhumeur. P., and Kriegman, D. 1999. Illumination-based image synthesis: Creating novel images of human faces under differing pose and lighting. In IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes, 47–54. Google ScholarDigital Library
    8. Georghiades. A. S. 2003. Recovering 3-d shape and reflectance from a small number of photographs. In Eurographics Symposium on Rendering: 14th Eurographics Workshop on Rendering, 230–240. Google ScholarDigital Library
    9. Gershbein, R., and Hanrahan, P. M. 2000. A fast relighting engine for interactive cinematic lighting design. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, 353–358. Google ScholarDigital Library
    10. Greene, N. 1986. Environment mapping and other application of world projections. IEEE Computer Graphics and Applications 6, 11 (November), 21–29. Google ScholarDigital Library
    11. Guenter, B., Grimm. C., Wood, D., Malvar, H., and Pighin, F. July 1998. Making faces. Proceedings of SIGGRAPH 98, 55–66. Google ScholarDigital Library
    12. Haeberli, P. 1992. Synthetic lighting for photography. Available at http://www.sgi.com/grafica/synth/index.html, January.Google Scholar
    13. Hager, G. D., and Belhumeur, P. N. 1996. Real-time tracking of image regions with changes in geometry and illumination. In Proc. IEEE Conf. on Comp. Vision and Patt. Recog., 403–410. Google ScholarDigital Library
    14. Hall, M. 1998. Combinatorial Theory, 2nd ed. Wiley, New York. Google ScholarDigital Library
    15. Harwit, M., and Sloane, N. J. A. 1979. Hadamard transform optics. Academic Press, New York.Google Scholar
    16. Hawkins, T., Wenger, A., Tchou, C., and Debevec, A. G. F. G. P. 2004. Animatable facial reflectance fields. In Eurographics Symposium on Rendering: 15th Eurographics Workshop on Rendering. Google ScholarDigital Library
    17. Hewlett-Packard Components Group. 1998. Noise sources in cmos image sensors. Tech. rep., Hewlett-Packard.Google Scholar
    18. Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2003. High dynamic range video. ACM Transactions on Graphics 22, 3 (July), 319–325. Google ScholarDigital Library
    19. Landis, H., 2002. Production-ready global illumination. Course Notes for SIGGRAPH 2002 Course 16, RenderMan in Production.Google Scholar
    20. Malzbender, T., Gelb, D., and Wolters, H. 2001. Polynomial texture maps. Proceedings of SIGGRAPH 2001 (August), 519–528. Google ScholarDigital Library
    21. Marks, J., Andalman, B., Beardsley, P. A., Freeman, W., Gibson, S., Hodgins, J. K., Kang. T., Mirtich, B., Pfister, H., Ruml, W., Ryall, K., Seims, J., and Shieber, S. 1997. Design galleries: A general approach to setting parameters for computer graphics and animation. In Proceedings of SIGGRAPH 97, Computer Graphics Proceedings, Annual Conference Series, 389–400. Google ScholarDigital Library
    22. Marschner, S., Guenter, B., and Raghupathy, S. 2000. Modeling and rendering for realistic facial animation. In Rendering Techniques 2000: 11th Eurographics Workshop on Rendering, 231–242. Google ScholarDigital Library
    23. Masselus, V., Peers, P., Dutre, P., and Willems, Y. D. 2004. Smooth reconstruction and compact representation of reflectance functions for image-based relighting. In 15th Eurographics Symposium on Rendering, no. Norrkoping, Sweden. Google ScholarDigital Library
    24. Miller, G. S., and Hoffman, C. R. 1984. Illumination and reflection maps: Simulated objects in simulated and real environments. In SIGGRAPH 84 Course Notes for Advanced Computer Graphics Animation.Google Scholar
    25. Ng, R., Ramamoorthi, R., and Hanrahan, P. 2003. All-frequency shadows using non-linear wavelet lighting approximation. ACM Transactions on Graphics 22, 3 (July). 376–381. Google ScholarDigital Library
    26. Nimeroff, J. S., Simoncelli, E., and Dorsey, J. 1994, Efficient re-rendering of naturally illuminated environments. In Fifth Eurographics Workshop on Rendering, 359–373.Google Scholar
    27. Nishino, K., and Nayar, S. K. 2004. Eyes for relighting. ACM Transactions on Graphics 23, 3 (Aug.), 704–711. Google ScholarDigital Library
    28. Petrović, L., Fujito, B., Williams, L., and Finkelstein, A. 2000. Shadows for cel animation. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings. Annual Conference Series, 511–516. Google ScholarDigital Library
    29. Porter, T., and Duff, T. 1984. Compositing digital images. In Computer Graphics (Proceedings of SIGGRAPH 84), vol. 18, 253–259. Google ScholarDigital Library
    30. Raskar, R., Tan. K.-H., Feris, R., Yu. J., and Turk, M. 2004. Non photorealistic camera: depth edge detection and stylized rendering using multiflash imaging. ACM Transactions on Graphics 23, 3 (Aug.), 679–688. Google ScholarDigital Library
    31. Rusinkiewicz, S., Hall-Holt. O., and Levoy, M. 2002. Real-time 3d model acquisition. ACM Transactions on Graphics 21, 3 (July), 438–446. Google ScholarDigital Library
    32. Schechner, Y. Y., Nayar, S. K., and Belhumeur, P. 2003. A theory of multiplexed illumination. In International Conference on Computer Vision. Google ScholarDigital Library
    33. Sloan, P.-P., Kautz, J., and Snyder, J. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. ACM Transactions on Graphics 21, 3 (July), 527–536. Google ScholarDigital Library
    34. Sloan, N. J. A., 1999. A library of hadamard matrices. http://www.research.att.com/~njas/hadamard/.Google Scholar
    35. Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In Proceedings of SIGGRAPH 96, 259–268. Google ScholarDigital Library
    36. Trumbull, D. 2000. Personal communication. January.Google Scholar
    37. Wen, Z., Liu, Z., and Huang, T. S. 2003. Face relighting with radiance environment maps. In 2003 Conference on Computer Vision and Pattern Recognition (CVPR 2003), 158–165.Google Scholar
    38. Williams, L. 1991. Shading in two dimensions. In Graphics Interface ’91, 143–151.Google Scholar
    39. Woodham, R. J. 1980. Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 1, 139–144.Google ScholarCross Ref
    40. Wu, X. 1991. An efficient antialiasing technique. In SIGGRAPH 91: Proceedings of the 18th annual conference on Computer graphics and interactive techniques, ACM Press, 143–152. Google ScholarDigital Library
    41. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3 (Aug.), 548–558. Google ScholarDigital Library
    42. Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. Proceedings of SIGGRAPH 99 (August), 205–214. Google ScholarDigital Library

ACM Digital Library Publication: