“Recovering photometric properties of architectural scenes from photographs” by Yu and Malik

In this paper, we present a new approach to producing photorealistic computer renderings of real architectural scenes under novel lighting conditions, such as at different times of day, starting from a small set of photographs of the real scene. Traditional texture mapping approaches to image-based modeling and rendering are unable to do this because texture maps are the product of the interaction between lighting and surface reflectance and one cannot deal with novel lighting without dissecting their respective contributions. To obtain this decomposition into lighting and reflectance, our basic approach is to solve a series of optimization problems to find the parameters of appropriate lighting and reflectance models that best explain the measured values in the various photographs of the scene. The lighting models include the radiance distributions from the sun and the sky, as well as the landscape to consider the effect of secondary illumination from the environment. The reflectance models are for the surfaces of the architecture. Photographs are taken for the sun, the sky, the landscape, as well as the architecture at a few different times of day to collect enough data for recovering the various lighting and reflectance models. We can predict novel illumination conditions with the recovered lighting models and use these together with the recovered reflectance values to produce renderings of the scene. Our results show that our goal of generating photorealistic renderings of real architectural scenes under novel lighting conditions has been achieved.

1. BRUNGER, A., AND HOOPER, F. Anisotropic sky radiance model based on narrow field of view measurements of shortwave radiance. Solar Energy 51, 1 (1993), 53-64.
2. DANA, K., VAN GINNEKEN, B., NAYAR, S., AND KOEN- DERINK, J. Reflectance and texture of real-world surfaces. In proceedings of CVPR (1997), pp. 151-157.
3. DEBEVEC, P., AND MALIK, J. Recovering high dynamic range radiance maps from photographs. In Computer Graphics Proceedings, Annual Conference Series (1997), pp. 369- 378.
4. DEBEVEC, P., TAYLOR, C., AND MALIK, J. Modeling and rendering architecture from photographs: A hybrid geometryand image-based approach. In Computer Graphics Proceedings, Annual Conference Series (1996), pp. 11-20.
5. DEBEVEC, P., YU, Y., AND BORSHUKOV, a. Efficient viewdependent image-based rendering with projective texturemapping. UC Berkeley technical report #UCB//CSD-98- 1003.
6. GORTLER, S., GRZESZCZUK, R., SZELISKI, R., AND CO- HEN, M. The lumigraph. In Computer Graphics Proceedings, Annual Conference Series (1996), pp. 43-54.
7. HAMPEL, F., ROUSSEEUW, P., RONCHETTI, E., AND STA- HEL, W. Robust Statistics. John Wiley & Sons, New York, 1986.
8. INEICHEN, P., MOLINEAUX, B., AND PEREZ, e. Sky luminance data validation: Comparison of seven models with four data banks. Solar Energy 52, 4 (1994), 337-346.
9. KLASSEN, R. Modeling the effect of the atmosphere on light. ACM Transactions on Graphics 6, 3 (1987), 215-237.
10. KOENDERINK, J., AND VAN DOORN, A. Illuminance texture due to surface mesostructure. J. Opt. Soc. Am.A 13, 3 (1996), 452-463.
11. LAFORTUNE, E., FOO, S., TORRANCE, K., AND GREEN- BERG, O. Non-linear approximation of reflectance functions. In Computer Graphics Proceedings, Annual Conference Series (1997), pp. 117-126.
12. LEVOY, M., AND HANRAHAN, P. Light field rendering. In Computer Graphics Proceedings, Annual Conference Series (1996), pp. 31-42.
13. LITTLEFAIR, P. A comparison of sky luminance models with measured data from garston, united kingdom. Solar Energy 53, 4 (1994), 315-322.
14. MCMmLAN, L., AND BISHOP, G. Plenoptic modeling: An image-based rendering system. In Computer Graphics Proceedings, Annual Conference Series (1995), pp. 39-46.
15. NISHITA, T., AND NAKAMAE, E. Continuous tone representation of three-dimensional objects illuminated by sky light. Computer Graphics 20, 4 (1986), 125-132.
16. PEREZ, R., SEALS, R., AND MICHALSKY, J. All-weather model for sky luminance distribution-preliminary configuration and validation. Solar Energy 50, 3 (1993), 235-245.
17. PRESS, W., FLANNERY, B., TEUKOLSKY, S., AND VETTER- LING, W. Numerical Recipes in C. Cambridge Univ. Press, New York, 1988.
18. REES, W. Physical Principles of Remote Sensing. Cambridge Univ. Press, 1990.
19. SATO, Y., AND IKEUCHI, K. Reflectance analysis for 3d computer graphics model generation. Graphical Models and Image Processing 58, 5 (1996), 437-451.
20. S ATO, Y., WHEELER, M., AND IKEUCHI, K. Object shape and reflectance modeling from observation. In Computer Graphics Proceedings, Annual Conference Series (1997), pp. 379-388.
21. SZELISKI, e., AND SHUM, U. Creating full view panoramic mosaics and environment maps. In Computer Graphics Proceedings, Annual Conference Series (1997), pp. 251-258.
22. TADAMURA, K., NAKAMAE, E., KANEDA, K., BABA, M., YAMASHITA, H., AND NISHITA, T. Modeling of skylight and rendering of outdoor scenes. Proceedings of EUROGRAPH- ICS’93, Computer Graphics Forum 12, 3 (1993), 189-200.
23. TAKAGI, A., TAKAOKA, H., OSHIMA, T., AND OGATA, Y. Accurate rendering technique based on colorimetric conception. Computer Graphics 24, 4 (1990), 1990.
24. WA~D, G. Measuring and modeling anisotropic reflection. Computer Graphics 26, 2 (1992), 265-272.
25. WOODHAM, R. Photometric method for determining surface orientation from multiple images. In Shape f~vm Shading, B. Horn and M. Brooks, Eds. MIT Press, 1989, pp. 513-532.