“Illumination decomposition for material recoloring with consistent interreflections” by Carroll, Ramamoorthi and Agrawala

Changing the color of an object is a basic image editing operation, but a high quality result must also preserve natural shading. A common approach is to first compute reflectance and illumination intrinsic images. Reflectances can then be edited independently, and recomposed with the illumination. However, manipulating only the reflectance color does not account for diffuse interreflections, and can result in inconsistent shading in the edited image. We propose an approach for further decomposing illumination into direct lighting, and indirect diffuse illumination from each material. This decomposition allows us to change indirect illumination from an individual material independently, so it matches the modified reflectance color. To address the underconstrained problem of decomposing illumination into multiple components, we take advantage of its smooth nature, as well as user-provided constraints. We demonstrate our approach on a number of examples, where we consistently edit material colors and the associated interreflections.

1. Bai, J., Chandraker, M., Ng, T.-T., and Ramamoorthi, R. 2010. A dual theory of inverse and forward light transport. In ECCV ’10, 294–307. Google ScholarDigital Library
2. Barrow, H., and Tenenbaum, J. 1978. Recovering intrinsic scene characteristics from images. Computer Vision Systems 27, 9, 3–26.Google Scholar
3. Bioucas-Dias, J. M., and Figueiredo, M. A. T. 2007. A new twist: Two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. on Image Processing 16, 12 (dec.), 2992–3004. Google ScholarDigital Library
4. Bousseau, A., Paris, S., and Durand, F. 2009. User-assisted intrinsic images. ACM Trans. Graph. 28, 130:1–130:10. Google ScholarDigital Library
5. Candes, E., Romberg, J., and Tao, T. 2006. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory 52, 2, 489–509. Google ScholarDigital Library
6. Chuang, Y.-Y., Curless, B., Salesin, D. H., and Szeliski, R. 2001. A bayesian approach to digital matting. In CVPR ’01, vol. 2, 264–271.Google Scholar
7. Cohen, M. F., and Wallace, J. 1993. Radiosity and realistic image synthesis. Academic Press Professional, Inc., San Diego, CA. Google Scholar
8. Donoho, D. 2006. Compressed sensing. IEEE Trans. on Inform. Theory 52, 4, 1289–1306. Google ScholarDigital Library
9. Fang, H., and Hart, J. C. 2004. Textureshop: texture synthesis as a photograph editing tool. ACM Trans. Graph. 23, 354–359. Google ScholarDigital Library
10. Fattal, R., Carroll, R., and Agrawala, M. 2009. Edge-based image coarsening. ACM Trans. Graph. 29, 6:1–6:11. Google ScholarDigital Library
11. Fattal, R. 2008. Single image dehazing. ACM Trans. Graph. 27, 72:1–72:9. Google ScholarDigital Library
12. Finlayson, G., Hordley, S., Lu, C., and Drew, M. 2006. On the removal of shadows from images. IEEE Trans. PAMI 28, 1, 59–68. Google ScholarDigital Library
13. Gutierrez, D., Seron, F. J., Lopez-Moreno, J., Sanchez, M. P., Fandos, J., and Reinhard, E. 2008. Depicting procedural caustics in single images. ACM Trans. Graph. 27, 120:1–120:9. Google ScholarDigital Library
14. Hašan, M., Pellacini, F., and Bala, K. 2006. Direct-to-indirect transfer for cinematic relighting. ACM Trans. Graph. 25, 1089–1097. Google ScholarDigital Library
15. Holland, P. W., and Welsch, R. E. 1977. Robust regression using iteratively reweighted least-squares. Communications in Statistics – Theory and Methods 6 (September), 813–827.Google ScholarCross Ref
16. Horn, B. K. P. 1986. Robot Vision. The MIT Press, March. Google Scholar
17. Hsu, E., Mertens, T., Paris, S., Avidan, S., and Durand, F. 2008. Light mixture estimation for spatially varying white balance. ACM Trans. Graph. 27, 70:1–70:7. Google ScholarDigital Library
18. Joshi, N., Zitnick, C., Szeliski, R., and Kriegman, D. 2009. Image deblurring and denoising using color priors. In CVPR ’09., 1550–1557.Google Scholar
19. Keller, A. 1997. Instant radiosity. In Proc. SIGGRAPH, 49–56. Google Scholar
20. Khan, E. A., Reinhard, E., Fleming, R. W., and Bülthoff, H. H. 2006. Image-based material editing. ACM Trans. Graph. 25, 654–663. Google ScholarDigital Library
21. Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM Trans. Graph. 26. Google Scholar
22. Land, E. H., John, and Mccann, J. 1971. Lightness and retinex theory. Journal of the Optical Society of America, 1–11.Google ScholarCross Ref
23. Levin, A., and Weiss, Y. 2007. User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans. PAMI 29, 9, 1647–1654. Google ScholarDigital Library
24. Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26. Google Scholar
25. Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Trans. PAMI 30 (February), 228–242. Google ScholarDigital Library
26. Mohan, A., Tumblin, J., and Choudhury, P. 2007. Editing soft shadows in a digital photograph. IEEE Comput. Graph. Appl. 27 (March), 23–31. Google ScholarDigital Library
27. Nayar, S. K., Krishnan, G., Grossberg, M. D., and Raskar, R. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM Trans. Graph. 25, 935–944. Google ScholarDigital Library
28. Obert, J., Křivánek, J., Sýkora, D., and Pattanaik, S. 2007. Interactive light transport editing for flexible global illumination. In ACM SIGGRAPH 2007 sketches. Google Scholar
29. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proc. SIGGRAPH, 433–442. Google Scholar
30. Pharr, M., and Humphreys, G. 2004. Physically Based Rendering: From Theory to Implementation. Morgan Kaufmann. Google Scholar
31. Ramamoorthi, R., and Hanrahan, P. 2001. A signal-processing framework for inverse rendering. In Proc. SIGGRAPH, 117–128. Google Scholar
32. Rudin, L. I., Osher, S., and Fatemi, E. 1992. Nonlinear total variation based noise removal algorithms. Phys. D 60 (November), 259–268. Google ScholarDigital Library
33. Schoeneman, C., Dorsey, J., Smits, B., Arvo, J., and Greenberg, D. 1993. Painting with light. In Proc. SIGGRAPH, 143–146. Google Scholar
34. Seitz, S., Matsushita, Y., and Kutulakos, K. 2005. A theory of inverse light transport. In ICCV ’05, vol. 2, 1440–1447. Google Scholar
35. Shen, L., Tan, P., and Lin, S. 2008. Intrinsic image decomposition with non-local texture cues. In CVPR ’08., 1–7.Google Scholar
36. Shor, Y., and Lischinski, D. 2008. The shadow meets the mask: Pyramid-based shadow removal. Comput. Graph. Forum 27, 2, 577–586.Google ScholarCross Ref
37. Sinha, P., and Adelson, E. 1993. Recovering reflectance and illumination in a world of painted polyhedra. In ICCV ’93, 156–163.Google Scholar
38. Tappen, M. F., Russell, B. C., and Freeman, W. T. 2003. Exploiting the sparse derivative prior for super-resolution and image demosaicing. In IEEE Workshop on Stat. and Comput. Theories of Vision.Google Scholar
39. Tappen, M. F., Freeman, W. T., and Adelson, E. H. 2005. Recovering intrinsic images from a single image. IEEE Trans. PAMI 27, 9, 1459–1472. Google ScholarDigital Library
40. Walter, B., Fernandez, S., Arbree, A., Bala, K., Donikian, M., and Greenberg, D. P. 2005. Lightcuts: a scalable approach to illumination. ACM Trans. Graph. 24, 1098–1107. Google ScholarDigital Library
41. Wang, J., and Cohen, M. 2007. Optimized color sampling for robust matting. In CVPR ’07., 1–8.Google Scholar
42. Weiss, Y. 2001. Deriving intrinsic images from image sequences. In ICCV ’01, vol. 2, 68–75.Google Scholar