“Illumination decomposition for material recoloring with consistent interreflections” by Carroll, Ramamoorthi and Agrawala

  • ©Robert Carroll, Ravi Ramamoorthi, and Maneesh Agrawala




    Illumination decomposition for material recoloring with consistent interreflections



    Changing the color of an object is a basic image editing operation, but a high quality result must also preserve natural shading. A common approach is to first compute reflectance and illumination intrinsic images. Reflectances can then be edited independently, and recomposed with the illumination. However, manipulating only the reflectance color does not account for diffuse interreflections, and can result in inconsistent shading in the edited image. We propose an approach for further decomposing illumination into direct lighting, and indirect diffuse illumination from each material. This decomposition allows us to change indirect illumination from an individual material independently, so it matches the modified reflectance color. To address the underconstrained problem of decomposing illumination into multiple components, we take advantage of its smooth nature, as well as user-provided constraints. We demonstrate our approach on a number of examples, where we consistently edit material colors and the associated interreflections.


    1. Bai, J., Chandraker, M., Ng, T.-T., and Ramamoorthi, R. 2010. A dual theory of inverse and forward light transport. In ECCV ’10, 294–307. Google ScholarDigital Library
    2. Barrow, H., and Tenenbaum, J. 1978. Recovering intrinsic scene characteristics from images. Computer Vision Systems 27, 9, 3–26.Google Scholar
    3. Bioucas-Dias, J. M., and Figueiredo, M. A. T. 2007. A new twist: Two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. on Image Processing 16, 12 (dec.), 2992–3004. Google ScholarDigital Library
    4. Bousseau, A., Paris, S., and Durand, F. 2009. User-assisted intrinsic images. ACM Trans. Graph. 28, 130:1–130:10. Google ScholarDigital Library
    5. Candes, E., Romberg, J., and Tao, T. 2006. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory 52, 2, 489–509. Google ScholarDigital Library
    6. Chuang, Y.-Y., Curless, B., Salesin, D. H., and Szeliski, R. 2001. A bayesian approach to digital matting. In CVPR ’01, vol. 2, 264–271.Google Scholar
    7. Cohen, M. F., and Wallace, J. 1993. Radiosity and realistic image synthesis. Academic Press Professional, Inc., San Diego, CA. Google Scholar
    8. Donoho, D. 2006. Compressed sensing. IEEE Trans. on Inform. Theory 52, 4, 1289–1306. Google ScholarDigital Library
    9. Fang, H., and Hart, J. C. 2004. Textureshop: texture synthesis as a photograph editing tool. ACM Trans. Graph. 23, 354–359. Google ScholarDigital Library
    10. Fattal, R., Carroll, R., and Agrawala, M. 2009. Edge-based image coarsening. ACM Trans. Graph. 29, 6:1–6:11. Google ScholarDigital Library
    11. Fattal, R. 2008. Single image dehazing. ACM Trans. Graph. 27, 72:1–72:9. Google ScholarDigital Library
    12. Finlayson, G., Hordley, S., Lu, C., and Drew, M. 2006. On the removal of shadows from images. IEEE Trans. PAMI 28, 1, 59–68. Google ScholarDigital Library
    13. Gutierrez, D., Seron, F. J., Lopez-Moreno, J., Sanchez, M. P., Fandos, J., and Reinhard, E. 2008. Depicting procedural caustics in single images. ACM Trans. Graph. 27, 120:1–120:9. Google ScholarDigital Library
    14. Hašan, M., Pellacini, F., and Bala, K. 2006. Direct-to-indirect transfer for cinematic relighting. ACM Trans. Graph. 25, 1089–1097. Google ScholarDigital Library
    15. Holland, P. W., and Welsch, R. E. 1977. Robust regression using iteratively reweighted least-squares. Communications in Statistics – Theory and Methods 6 (September), 813–827.Google ScholarCross Ref
    16. Horn, B. K. P. 1986. Robot Vision. The MIT Press, March. Google Scholar
    17. Hsu, E., Mertens, T., Paris, S., Avidan, S., and Durand, F. 2008. Light mixture estimation for spatially varying white balance. ACM Trans. Graph. 27, 70:1–70:7. Google ScholarDigital Library
    18. Joshi, N., Zitnick, C., Szeliski, R., and Kriegman, D. 2009. Image deblurring and denoising using color priors. In CVPR ’09., 1550–1557.Google Scholar
    19. Keller, A. 1997. Instant radiosity. In Proc. SIGGRAPH, 49–56. Google Scholar
    20. Khan, E. A., Reinhard, E., Fleming, R. W., and Bülthoff, H. H. 2006. Image-based material editing. ACM Trans. Graph. 25, 654–663. Google ScholarDigital Library
    21. Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM Trans. Graph. 26. Google Scholar
    22. Land, E. H., John, and Mccann, J. 1971. Lightness and retinex theory. Journal of the Optical Society of America, 1–11.Google ScholarCross Ref
    23. Levin, A., and Weiss, Y. 2007. User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans. PAMI 29, 9, 1647–1654. Google ScholarDigital Library
    24. Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26. Google Scholar
    25. Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Trans. PAMI 30 (February), 228–242. Google ScholarDigital Library
    26. Mohan, A., Tumblin, J., and Choudhury, P. 2007. Editing soft shadows in a digital photograph. IEEE Comput. Graph. Appl. 27 (March), 23–31. Google ScholarDigital Library
    27. Nayar, S. K., Krishnan, G., Grossberg, M. D., and Raskar, R. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM Trans. Graph. 25, 935–944. Google ScholarDigital Library
    28. Obert, J., Křivánek, J., Sýkora, D., and Pattanaik, S. 2007. Interactive light transport editing for flexible global illumination. In ACM SIGGRAPH 2007 sketches. Google Scholar
    29. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proc. SIGGRAPH, 433–442. Google Scholar
    30. Pharr, M., and Humphreys, G. 2004. Physically Based Rendering: From Theory to Implementation. Morgan Kaufmann. Google Scholar
    31. Ramamoorthi, R., and Hanrahan, P. 2001. A signal-processing framework for inverse rendering. In Proc. SIGGRAPH, 117–128. Google Scholar
    32. Rudin, L. I., Osher, S., and Fatemi, E. 1992. Nonlinear total variation based noise removal algorithms. Phys. D 60 (November), 259–268. Google ScholarDigital Library
    33. Schoeneman, C., Dorsey, J., Smits, B., Arvo, J., and Greenberg, D. 1993. Painting with light. In Proc. SIGGRAPH, 143–146. Google Scholar
    34. Seitz, S., Matsushita, Y., and Kutulakos, K. 2005. A theory of inverse light transport. In ICCV ’05, vol. 2, 1440–1447. Google Scholar
    35. Shen, L., Tan, P., and Lin, S. 2008. Intrinsic image decomposition with non-local texture cues. In CVPR ’08., 1–7.Google Scholar
    36. Shor, Y., and Lischinski, D. 2008. The shadow meets the mask: Pyramid-based shadow removal. Comput. Graph. Forum 27, 2, 577–586.Google ScholarCross Ref
    37. Sinha, P., and Adelson, E. 1993. Recovering reflectance and illumination in a world of painted polyhedra. In ICCV ’93, 156–163.Google Scholar
    38. Tappen, M. F., Russell, B. C., and Freeman, W. T. 2003. Exploiting the sparse derivative prior for super-resolution and image demosaicing. In IEEE Workshop on Stat. and Comput. Theories of Vision.Google Scholar
    39. Tappen, M. F., Freeman, W. T., and Adelson, E. H. 2005. Recovering intrinsic images from a single image. IEEE Trans. PAMI 27, 9, 1459–1472. Google ScholarDigital Library
    40. Walter, B., Fernandez, S., Arbree, A., Bala, K., Donikian, M., and Greenberg, D. P. 2005. Lightcuts: a scalable approach to illumination. ACM Trans. Graph. 24, 1098–1107. Google ScholarDigital Library
    41. Wang, J., and Cohen, M. 2007. Optimized color sampling for robust matting. In CVPR ’07., 1–8.Google Scholar
    42. Weiss, Y. 2001. Deriving intrinsic images from image sequences. In ICCV ’01, vol. 2, 68–75.Google Scholar

ACM Digital Library Publication:

Overview Page: