“Coherent intrinsic images from photo collections”
Conference:
Type(s):
Title:
- Coherent intrinsic images from photo collections
Session/Category Title: Color and Photos
Presenter(s)/Author(s):
Abstract:
An intrinsic image is a decomposition of a photo into an illumination layer and a reflectance layer, which enables powerful editing such as the alteration of an object’s material independently of its illumination. However, decomposing a single photo is highly under-constrained and existing methods require user assistance or handle only simple scenes. In this paper, we compute intrinsic decompositions using several images of the same scene under different viewpoints and lighting conditions. We use multi-view stereo to automatically reconstruct 3D points and normals from which we derive relationships between reflectance values at different locations, across multiple views and consequently different lighting conditions. We use robust estimation to reliably identify reflectance ratios between pairs of points. From these, we infer constraints for our optimization and enforce a coherent solution across multiple views and illuminations. Our results demonstrate that this constrained optimization yields high-quality and coherent intrinsic decompositions of complex scenes. We illustrate how these decompositions can be used for image-based illumination transfer and transitions between views with consistent lighting.
References:
1. Barrow, H., and Tenenbaum, J. 1978. Recovering intrinsic scene characteristics from images. Computer Vision Systems.
2. Bousseau, A., Paris, S., and Durand, F. 2009. User-assisted intrinsic images. ACM Trans. Graph. 28, 5.
3. Chaurasia, G., Sorkine, O., and Drettakis, G. 2011. Silhouette-aware warping for image-based rendering. Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering) 30, 4.
4. Debevec, P., and et al. 2004. Estimating surface reflectance properties of a complex scene under captured natural illumination. Tech. rep., USC Institute for Creative Technologies.
5. Furukawa, Y., and Ponce, J. 2009. Accurate, dense, and robust multi-view stereopsis. IEEE Trans. PAMI 32, 8, 1362–1376.
6. Garces, E., Munoz, A., Lopez-Moreno, J., and Gutierrez, D. 2012. Intrinsic images by clustering. Computer Graphics Forum. Eurographics Symposium on rendering, EGSR ’12.
7. Garg, R., Du, H., Seitz, S. M., and Snavely, N. 2009. The dimensionality of scene appearance. In IEEE ICCV, 1917–1924.
8. Grosse, R., Johnson, M. K., Adelson, E. H., and Freeman, W. T. 2009. Ground-truth dataset and baseline evaluations for intrinsic image algorithms. In IEEE ICCV.
9. Haber, T., Fuchs, C., Bekaert, P., Seidel, H.-P., Goesele, M., and Lensch, H. 2009. Relighting objects from image collections. In Proc. IEEE CVPR, 627–634.
10. Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM TOG (Proc. SIGGRAPH) 26, 3.
11. Hoiem, D., Efros, A. A., and Hebert, M. 2005. Automatic photo pop-up. ACM TOG (Proc. SIGGRAPH) 24, 3, 577–584.
12. Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., and Stuetzle, W. 1992. Surface reconstruction from unorganized points. SIGGRAPH 26, 71–78.
13. Horn, B. K. 1986. Robot Vision, 1st ed. McGraw-Hill Higher Education.
14. Laffont, P.-Y., Bousseau, A., and Drettakis, G. 2012. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Trans. on Vis. and Comp. Graph..
15. Lee, K. J., Zhao, Q., Tong, X., Gong, M., Izadi, S., Uk Lee, S., Tan, P., and Lin, S. 2012. Estimation of intrinsic image sequences from image+depth video. In Proc. ECCV.
16. Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Trans. PAMI.
17. Liu, X., Wan, L., Qu, Y., Wong, T.-T., Lin, S., Leung, C.-S., and Heng, P.-A. 2008. Intrinsic colorization. ACM, SIGGRAPH Asia ’08, 152:1–152:9.
18. Matsushita, Y., Lin, S., Kang, S., and Shum, H.-Y. 2004. Estimating intrinsic images from image sequences with biased illumination. In Proc. ECCV, vol. 3022, 274–286.
19. Matusik, W., Loper, M., and Pfister, H. 2004. Progressively-refined reflectance functions from natural illumination. In Proc. EGSR, 299–308.
20. Pharr, M., and Humphreys, G. 2010. Physically Based Rendering: From Theory to Implementation, second edition. Morgan Kaufmann Publishers Inc.
21. Preetham, A. J., Shirley, P., and Smits, B. 1999. A practical analytic model for daylight. In SIGGRAPH, 91–100.
22. Roberts, D. A., 2009. Pixelstruct, an opensource tool for visualizing 3d scenes reconstructed from photographs.
23. Shen, L., and Yeo, C. 2011. Intrinsic image decomposition using a local and global sparse representation of reflectance. In Proc. IEEE CVPR.
24. Shen, L., Tan, P., and Lin, S. 2008. Intrinsic image decomposition with non-local texture cues. In Proc. IEEE CVPR.
25. Shen, J., Yang, X., Jia, Y., and Li, X. 2011. Intrinsic images using optimization. In Proc. IEEE CVPR.
26. Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: Exploring photo collections in 3d. ACM TOG (Proc. SIGGRAPH) 25, 3, 835–846.
27. Snavely, N., Garg, R., Seitz, S. M., and Szeliski, R. 2008. Finding paths through the world’s photos. ACM TOG (Proc. SIGGRAPH) 27, 3, 11–21.
28. Sunkavalli, K., Matusik, W., Pfister, H., and Rusinkiewicz, S. 2007. Factored time-lapse video. ACM Transactions on Graphics (Proc. SIGGRAPH) 26, 3.
29. Tappen, M. F., Freeman, W. T., and Adelson, E. H. 2005. Recovering intrinsic images from a single image. IEEE Trans. PAMI 27, 9.
30. Troccoli, A., and Allen, P. 2008. Building illumination coherent 3d models of large-scale outdoor scenes. Int. J. Comput. Vision 78, 2–3, 261–280.
31. Tuite, K., Snavely, N., Hsiao, D.-y., Tabing, N., and Popovic, Z. 2011. Photocity: training experts at large-scale image acquisition through a competitive game. In Proc. SIGCHI’11, 1383–1392.
32. Weiss, Y. 2001. Deriving intrinsic images from image sequences. In IEEE ICCV, vol. 2, 68.
33. Wu, C., Agarwal, S., Curless, B., and Seitz, S. 2011. Multicore bundle adjustment. In Proc. IEEE CVPR, 3057–3064.
34. Yu, Y., and Malik, J. 1998. Recovering photometric properties of architectural scenes from photographs. In SIGGRAPH’98.
35. Yu, Y., Debevec, P., Malik, J., and Hawkins, T. 1999. Inverse global illumination: recovering reflectance models of real scenes from photographs. In SIGGRAPH ’99, 215–224.
36. Zhao, Q., Tan, P., Dai, Q., Shen, L., Wu, E., and Lin, S. 2012. A closed-form solution to retinex with nonlocal texture constraints. IEEE Trans. PAMI 34.
