“A lighting reproduction approach to live-action compositing”

We describe a process for compositing a live performance of an actor into a virtual set wherein the actor is consistently illuminated by the virtual environment. The Light Stage used in this work is a two-meter sphere of inward-pointing RGB light emitting diodes focused on the actor, where each light can be set to an arbitrary color and intensity to replicate a real-world or virtual lighting environment. We implement a digital two-camera infrared matting system to composite the actor into the background plate of the environment without affecting the visible-spectrum illumination on the actor. The color reponse of the system is calibrated to produce correct color renditions of the actor as illuminated by the environment. We demonstrate moving-camera composites of actors into real-world environments and virtual sets such that the actor is properly illuminated by the environment into which they are composited.

1. ALEXANDER, D. K., JONES, P. J., AND JENKINS, H. The artificial sky and heliodon facility at Cardiff University. In Proc. International Building Performance Simulation Association (IBPSA) (September 1999). (Abstract).Google Scholar
2. BEN-EZRA, M. Segmentation with invisible keying signal. In Proc. IEEE Conf. on Comp. Vision and Patt. Recog. (June 2000), pp. 568-575.Google Scholar
3. BORGES, C. F. Trichromatic approximation for computer graphics illumination models. In Computer Graphics (Proceedings of SIGGRAPH 91) (July 1991), vol. 25, pp. 101-104. Google Scholar
4. BRINKMANN, R. The Art and Science of Digital Compositing. Morgan Kaufmann, 1999. Google Scholar
5. CHUANG, Y.-Y., ZONGKER, D. E., HINDORFF, J., CURLESS, B., SALESIN, D. H., AND SZELISKI, R. Environment matting extensions: Towards higher accuracy and real-time capture. In Proceedings of SIGGRAPH 2000 (July 2000), pp. 121-130. Google Scholar
6. DEBEVEC, P. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In SIGGRAPH 98 (July 1998). Google Scholar
7. DEBEVEC, P., HAWKINS, T., TCHOU, C., DUIKER, H.-P., SAROKIN, W., AND SAGAR, M. Acquiring the reflectance field of a human face. Proceedings of SIGGRAPH 2000 (July 2000), 145-156. Google Scholar
8. DEBEVEC, P. E., AND MALIK, J. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH 97 (August 1997), pp. 369-378. Google Scholar
9. FAUGERAS, O. Three-Dimensional Computer Vision. MIT Press, 1993. Google Scholar
10. FIELDING, R. The Technique of Special Effects Cinematography, 4th ed. Hastings House, New York, 1985, ch. 11, pp. 290-321.Google Scholar
11. GAT, N. Real-time multi- and hyper-spectral imaging for remote sensing and machine vision: an overview. In Proc. 1998 ASAE Annual International Mtg. (Orlando, Florida, July 1998).Google Scholar
12. KAJIYA, J. T. The rendering equation. In Computer Graphics (Proceedings of SIGGRAPH 86) (Dallas, Texas, August 1986), vol. 20, pp. 143-150. Google Scholar
13. KOUDELKA, M., MAGDA, S., BELHUMEUR, P., AND KRIEGMAN, D. Image-based modeling and rendering of surfaces with arbitrary brdfs. In Proc. IEEE Conf. on Comp. Vision and Patt. Recog. (2001), pp. 568-575.Google Scholar
14. LEVOY, M., AND HANRAHAN, P. Light field rendering. In SIGGRAPH 96 (1996), pp. 31-42. Google Scholar
15. MALZBENDER, T., GELB, D., AND WOLTERS, H. Polynomial texture maps. Proceedings of SIGGRAPH 2001 (August 2001), 519-528. ISBN 1-58113-292-1. Google Scholar
16. MATUSIK, W., BUEHLER, C., RASKAR, R., GORTLER, S. J., AND MCMILLAN, L. Image-based visual hulls. In Proc. SIGGRAPH 2000 (July 2000), pp. 369-374. Google Scholar
17. MILLER, G. S., AND HOFFMAN, C. R. Illumination and reflection maps: Simulated objects in simulated and real environments. In SIGGRAPH 84 Course Notes for Advanced Computer Graphics Animation (July 1984).Google Scholar
18. NIMEROFF, J. S., SIMONCELLI, E., AND DORSEY, J. Efficient re-rendering of naturally illuminated environments. Fifth Eurographics Workshop on Rendering (June 1994), 359-373.Google Scholar
19. PORTER, T., AND DUFF, T. Compositing digital images. In Computer Graphics (Proceedings of SIGGRAPH 84) (Minneapolis, Minnesota, July 1984), vol. 18, pp. 253-259. Google Scholar
20. RAMAMOORTHI, R., AND HANRAHAN, P. A signal-processing framework for inverse rendering. In Proc. SIGGRAPH 2001 (August 2001), pp. 117-128. Google Scholar
21. SMITH, A. R., AND BLINN, J. F. Blue screen matting. In Proceedings of SIGGRAPH 96 (August 1996), pp. 259-268. Google Scholar
22. TCHOU, C., AND DEBEVEC, P. HDR Shop. Available at http://www.debevec.org/HDRShop, 2001.Google Scholar
23. TSAI, R. A versatile camera calibration technique for high accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses. IEEE Journal of Robotics and Automation 3, 4 (August 1987), 323-344.Google Scholar
24. VIDOR, Z. An infrared self-matting process. Society of Motion Picture and Television Engineers 69 (June 1960), 425-427.Google Scholar
25. WINDOWS AND DAYLIGHTING GROUP. The sky simulator for daylighting studies. Tech. Rep. DA 188 LBL, Lawrence Berkeley Laboratories, January 1985.Google Scholar
26. WYSZECKI, G., AND STILES, W. S. Color Science: Concepts and Methods, Quantitative and Formulae, 2nd ed. Wiley, New York, 1982.Google Scholar
27. ZONGKER, D. E., WERNER, D. M., CURLESS, B., AND SALESIN, D. H. Environment matting and compositing. Proceedings of SIGGRAPH 99 (August 1999), 205-214. Google Scholar