“Environment matting and compositing” by Zongker, Werner, Curless and Salesin

This paper introduces a new process, environment matting, which captures not just a foreground object and its traditional opacity matte from a real-world scene, but also a description of how that object refracts and reflects light, which we call an environment matte. The foreground object can then be placed in a new environment, using environment compositing, where it will refract and reflect light from that scene. Objects captured in this way exhibit not only specular but glossy and translucent effects, as well as selective attenuation and scattering of light according to wavelength. Moreover, the environment compositing process, which can be performed largely with texture mapping operations, is fast enough to run at interactive speeds on a desktop PC. We compare our results to photos of the same objects in real scenes. Applications of this work include the relighting of objects for virtual and augmented reality, more realistic 3D clip art, and interactive lighting design.

1. Paul Besl. Active optical range imaging sensors. In Jorge L.C. Sanz, editor, Advances in Machine Vision, chapter 1, pages 1-63. Springer-Verlag, 1989.
2. J. E Blinn and M. E. Newell. Texture and reflection in computer generated images. Communications of the ACM, 19:542-546, 1976.
3. Normand Bri6re and Pierre Poulin. Hierarchical view-dependent structures for interactive scene manipulation. In Proceedings of SIGGRAPH 96, pages 83-90, August 1996.
4. Peter J. Burt. Fast filter transforms for image processing. Computer Graphics and Image P~vcessing, 16:20-51, 1981.
5. Brian Carrihill and Robert Hummel. Experiments with the intensity ratio depth sensor. In Computer Vision, Graphics, and Image P~vcessing, volume 32, pages 337-358, 1985.
6. G. Chazan and N. Kiryati. Pyramidal intensity ratio depth sensor. Technical Report 121, Center for Communication and Information Technologies, Department of Electrical Engineering, Technion, Haifa, Israel, October 1995.
7. Robert L. Cook, Thomas Porter, and Loren Carpenter. Distributed ray tracing. Computer Graphics, 18(3):137-145, July 1984.
8. Franklin C. Crow. Summed-area tables for texture mapping. In P~vceedings of SIGGRAPtt 84, volume 18, pages 207-212, July 1984.
9. Paul E. Debevec and Jitendra Malik. Recovering high dynamic range radiance maps from photographs. In P1vceedings of SIGGRAPH 97, pages 369-378, August 1997.
10. Julie Dorsey, James Arvo, and Donald Greenberg. Interactive design of complex time dependent lighting. IEEE Computer Graphics and Applications, 15(2):26- 36, March 1995.
11. R. Fielding. The Technique of Special Effects Cinematography, pages 220-243. Focal/Hastings House, London, third edition, 1972.
12. Reid Gershbein. Cinematic Lighting in Computer Graphics. PhD thesis, Princeton University, 1999. Expected.
13. Andrew S. Glassner. An overview of ray tracing. In Andrew S. Glassner, editor, An Introduction to Ray Tracing, chapter 1. Academic Press, 1989.
14. Ned Greene. Environment mapping and other applications of world projections. IEEE Computer Graphics and Applications, 6(11), November 1986.
15. Ned Greene and Paul S. Heckbert. Creating raster omnimax images from multiple perspective views using the elliptical weighted average filter. IEEE Computer Graphics and Applications, 6(6):21-27, June 1986.
16. Eli Horn and Nahum Kiryati. Toward optimal structured light patterns. In P1vceedings of the International Conference on Recent Advances in Three- Dimensional Digital Imaging and Modeling, pages 28-35, 1997.
17. Douglas S. Kay and Donald R Greenberg. Transparency for computer synthesized images. In P1vceedings of SIGGRAPH 79, volume 13, pages 158-164, August 1979.
18. Gavin Miller and Marc Mondesir. Rendering hyper-sprites in real time. In G. Drettakis and N. Max, editors, P~vceedings 1998 Emvgraphics Workshop on Rendering, pages 193-198, June 1998.
19. Jeffry S. Nimeroff, Eero Simoncelli, and Julie Dorsey. Efficient re-rendering of naturally illuminated environments. In Fifth Emvgraphics Workshop on Rendering, pages 359-373, June 1994.
20. Thomas Porter and Tom Duff. Compositing digital images. In P~vceedings of SIGGRAPH 84, volume 18, pages 253-259, July 1984.
21. K. Sato and S. Inokuchi. Three-dimensional surface measurement by space encoding range imaging. Journal of Robotic Systems, 2:27-39, 1985.
22. Steven M. Seitz and Charles R. Dyer. View morphing: Synthesizing 3D metamorphoses using image transforms. In P1vceedings of SIGGRAPH 96, pages 21-30, August 1996.
23. Carlo H. S6quin and Eliot K. Smyrl. Parameterized ray tracing. In P~vceedings of SIGGRAPtt 89, volume 23, pages 307-314, July 1989.
24. Jean Paul Serra. Image Analysis and Mathematical Morphology. Academic Press, 1982.
25. Alvy Ray Smith and James F. Blinn. Blue screen matting. In P1vceedings of SIGGRAPtt 96, pages 259-268, August 1996.
26. Douglas Voorhies and Jim Foran. Reflection vector shading hardware. In Andrew Glassner, editor, P1vceedings of SIGGRAPH 94, pages 163-166, July 1994.
27. Gregory J. Ward. Measuring and modeling anisotropic reflection. In Edwin E. Catmull, editor, Computer Graphics (SIGGRAPH ’92 P1vceedings), volume 26, pages 265-272, July 1992.