“A signal-processing framework for inverse rendering” by Ramamoorthi and Hanrahan

Realism in computer-generated images requires accurate input models for lighting, textures and BRDFs. One of the best ways of obtaining high-quality data is through measurements of scene attributes from real photographs by inverse rendering. However, inverse rendering methods have been largely limited to settings with highly controlled lighting. One of the reasons for this is the lack of a coherent mathematical framework for inverse rendering under general illumination conditions. Our main contribution is the introduction of a signal-processing framework which describes the reflected light field as a convolution of the lighting and BRDF, and expresses it mathematically as a product of spherical harmonic coefficients of the BRDF and the lighting. Inverse rendering can then be viewed as deconvolution. We apply this theory to a variety of problems in inverse rendering, explaining a number of previous empirical results. We will show why certain problems are ill-posed or numerically ill-conditioned, and why other problems are more amenable to solution. The theory developed here also leads to new practical representations and algorithms. For instance, we present a method to factor the lighting and BRDF from a small number of views, i.e. to estimate both simultaneously when neither is known.

1. R. Basri and D. Jacobs. Lambertian reflectance and linear subspaces. In International Conference on Computer Vision, 2001.
2. R. Bastos, K. Hoff, W. Wynn, and A. Lastra. Increased photorealism for interactive architectural walkthroughs. In I3D 99, pages 183-190, 1999.
3. B. Cabral, N. Max, and R. Springmeyer. Bidirectional reflection functions from surface bump maps. In SIGGRAPH 87, pages 273-281, 1987.
4. B. Curless and M. Levoy. A volumetric method for building complex models from range images. In SIGGRAPH 96, pages 303-312, 1996.
5. K. Dana, B. Ginneken, S. Nayar, and J. Koenderink. Reflectance and texture of real-world surfaces. ACM Transactions on Graphics, 18(1):1-34, January 1999.
6. P. Debevec, T. Hawkins, C. Tchou, H.P. Duiker, W. Sarokin, and M. Sagar. Acquiring the reflectance field of a human face. In SIGGRAPH 00, pages 145-156.
7. R. Dror, E. Adelson, and A. Willsky. Estimating surface reflectance properties from images under unknown illumination. In SPIE Photonics West: Human Vision and Electronic Imaging VI, 2001.
8. M. D’Zmura. Computational Models of Visual Processing, chapter Shading Ambiguity: Reflectance and Illumination, pages 187-207. MIT Press, 1991.
9. Numerical Algorithms Group. NAG C Library Manual, Mark 5. 1999.
10. K. Ikeuchi and K. Sato. Determining reflectance properties of an object using range and brightness images. PAMI, 13(11):1139-1153, 1991.
11. T. Inui, Y. Tanabe, and Y. Onodera. Group theory and its applications in physics. Springer Verlag, 1990.
12. K. F. Karner, H. Mayer, and M. Gervautz. An image based measurement system for anisotropic reflection. Computer Graphics Forum, 15(3):119-128, 1996.
13. G.J. Klinker, S.A. Shafer, and T. Kanade. The measurement of highlights in color images. IJCV, 2(1):7-32, 1988.
14. J. J. Koenderink and A. J. van Doorn. Phenomenological description of bidirectional surface reflection. JOSA A, 15(11):2903-2912, 1998.
15. E. Land and J. McCann. Lightness and retinex theory. Journal of the Optical Society of America, 61(1):1-11, 1971.
16. M. Levoy, K. Pulli, B. Curless, S. Rusinkiewicz, D. Koller, L. Pereira, M. Ginzton, S. Anderson, J. Davis, J. Ginsberg, J. Shade, and D. Fulk. The digital michelangelo project: 3D scanning of large statues. In SIGGRAPH 00, pages 131-144.
17. R.C. Love. Surface Reflection Model Estimation from Naturally Illuminated Image Sequences. PhD thesis, Leeds, 1997.
18. R. Lu, J. Koenderink, and A. Kappers. Optical properties (bidirectional reflection distribution functions) of velvet. Applied Optics, 37(25):5974-5984, 1998.
19. T.M. MacRobert. Spherical harmonics; an elementary treatise on harmonic functions, with applications. Dover Publications, 1948.
20. S.R. Marschner and D.P. Greenberg. Inverse lighting for photography. In Fifth Color Imaging Conference, pages 262-265, 1997.
21. S.R. Marschner, S.H. Westin, E.P.F. Lafortune, and K.E. Torrance. Image-Based BRDF measurement. Applied Optics, 39(16):2592-2600, 2000.
22. N. McCormick. Inverse radiative transfer problems: a review. Nuclear Science and Engineering, 112:185-198, 1992.
23. G. Miller and C. Hoffman. Illumination and reflection maps: Simulated objects in simulated and real environments. SIGGRAPH 84 Advanced Computer Graphics Animation seminar notes, 1984.
24. F. E. Nicodemus, J. C. Richmond, J. J. Hsia, I. W. Ginsberg, and T. Limperis. Geometric Considerations and Nomenclature for Reflectance. National Bureau of Standards (US), 1977.
25. J. Nimeroff, E. Simoncelli, and J. Dorsey. Efficient re-rendering of naturally illuminated environments. In EGWR 94, pages 359-373.
26. R.W. Preisendorfer. Hydrologic Optics. US Dept Commerce, 1976.
27. R. Ramamoorthi and P. Hanrahan. Analysis of planar light fields from homogeneous convex curved surfaces under distant illumination. In SPIE Photonics West: Human Vision and Electronic Imaging VI, 2001.
28. R. Ramamoorthi and P. Hanrahan. An efficient representation for irradiance environment maps. In SIGGRAPH 01, 2001.
29. R. Ramamoorthi and P. Hanrahan. On the relationship between radiance and irradiance: Determining the illumination from images of a convex lambertian object. To appear, Journal of the Optical Society of America A, 2001.
30. I. Sato, Y. Sato, and K. Ikeuchi. Illumination distribution from brightness in shadows: adaptive estimation of illumination distribution with unknown reflectance properties in shadow regions. In ICCV 99, pages 875 – 882, 1999.
31. Y. Sato and K. Ikeuchi. Reflectance analysis under solar illumination. Technical Report CMU-CS-94-221, Carnegie Mellon University, 1994.
32. Y. Sato, M. D. Wheeler, and K. Ikeuchi. Object shape and reflectance modeling from observation. In SIGGRAPH 97, pages 379-388, 1997.
33. F. X. Sillion, J. Arvo, S. H. Westin, and D. Greenberg. A global illumination solution for general reflectance distributions. In SIGGRAPH 91, pages 187-196.
34. S.R.Marschner. Inverse Rendering for Computer Graphics. PhD thesis, Cornell, 1998.
35. S. Tominaga and N. Tanaka. Estimating reflection parameters from a single color image. IEEE Computer Graphics & Applications, 20(5):58-66, 2000.
36. K. E. Torrance and E. M. Sparrow. Theory for off-specular reflection from roughened surfaces. JOSA, 57(9):1105-1114, 1967.
37. G. J. Ward. Measuring and modeling anisotropic reflection. In SIGGRAPH 92, pages 265-272, 1992.
38. Y. Yu, P. Debevec, J. Malik, and T. Hawkins. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In SIGGRAPH 99, pages 215-224, 1999.
39. Y. Yu and J. Malik. Recovering photometric properties of architectural scenes from photographs. In SIGGRAPH 98, pages 207-218, 1998.
40. Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.