“Image-based material editing” by Khan, Reinhard, Fleming and Bülthoff

  • ©Erum Arif Khan, Erik Reinhard, Roland W. Fleming, and Heinrich H. Bülthoff




    Image-based material editing



    Photo editing software allows digital images to be blurred, warped or re-colored at the touch of a button. However, it is not currently possible to change the material appearance of an object except by painstakingly painting over the appropriate pixels. Here we present a method for automatically replacing one material with another, completely different material, starting with only a single high dynamic range image as input. Our approach exploits the fact that human vision is surprisingly tolerant of certain (sometimes enormous) physical inaccuracies, while being sensitive to others. By adjusting our simulations to be careful about those aspects to which the human visual system is sensitive, we are for the first time able to demonstrate significant material changes on the basis of a single photograph as input.


    1. Adelson, E. 2001. On seeing stuff: The perception of materials by humans. In Proc. of the SPIE, Human Vision and Electronic Imaging VI, vol. 4299, 1–12.Google ScholarCross Ref
    2. Belhumeur, P., Kriegman, D., and Yuille, A. 1999. The bas-relief ambiguity. International Journal of Computer Vision 1, 33–44. Google ScholarDigital Library
    3. Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. 2000. Image inpainting. In SIGGRAPH ’00: Proceedings of the 27th annual conference on computer graphics and interactive techniques, 417–424. Google ScholarDigital Library
    4. Blake, A., and Bülthoff, H. H. 1990. Does the brain know the physics of specular reflection? Nature 343, 6254, 165–168.Google Scholar
    5. Blake, A., and Bülthoff, H. H. 1991. Shape from specularities: Computation and psychophysics. Philosophical Transactions of the Royal Society (London) Series B 331, 237–252.Google ScholarCross Ref
    6. Chuang, Y., Zongker, D. E., Hindorff, J., Curless, B., Salesin, D., and Szeliski, R. 2000. Environment matting extensions: Towards higher accuracy and real-time capture. In SIGGRAPH ’00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 121–130. Google ScholarDigital Library
    7. Debevec, P. E., and Malik, J. 1997. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH 97 Conference Proceedings, 369–378. Google ScholarDigital Library
    8. Debevec, P. E. 1998. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with illumination and high dynamic range photography. In SIGGRAPH 98 Conference Proceedings, 45–50. Google ScholarDigital Library
    9. Debevec, P. 2002. A tutorial on image-based lighting. IEEE Computer Graphics and Applications 22, 2, 26–34. Google ScholarDigital Library
    10. Drew, M. S. 1994. Robust specularity detection from a single multi-illuminant color image. Computer Vision, Graphics, and Image Processing: Image Understanding 59, 3, 320–327. Google ScholarDigital Library
    11. Drori, I., Cohen-Or, D., and Yeshurun, H. 2003. Fragment-based image completion. ACM Transactions on Graphics 22, 3, 303–312. Google ScholarDigital Library
    12. Durand, F., and Dorsey, J. 2002. Fast bilateral filtering for the display of high-dynamic-range images. ACM Transactions on Graphics 21, 3, 257–266. Google ScholarDigital Library
    13. Eisemann, E., and Durand, F. 2004. Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics 23, 3, 673–678. Google ScholarDigital Library
    14. Fang, H., and Hart, J. C. 2004. Textureshop: Texture synthesis as a photographic editing tool. ACM Transactions on Graphics 23, 3, 354–358. Google ScholarDigital Library
    15. Fleming, R. W., and Bülthoff, H. H. 2005. Low-level image cues in the perception of translucent materials. ACM Trans. on Applied Perception 2, 3, 346–382. Google ScholarDigital Library
    16. Freeman, W. T. 1994. The generic viewpoint assumption in a framework for visual perception. Nature 368, 542–545.Google ScholarCross Ref
    17. Freeman, W. T. 1996. Exploiting the generic viewpoint assumption. International Journal of Computer Vision 20, 3, 243–261. Google ScholarDigital Library
    18. Hart, J. C., Francis, G. K., and Kauffman, L. H. 1994. Visualizing quaternion rotation. ACM Transactions on Graphics 13, 3, 256–276. Google ScholarDigital Library
    19. Igarashi, T., Matsuoka, S., and Tanaka, H. 1999. Teddy: a sketching interface for 3d freeform design. In SIGGRAPH ’99: Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 409–416. Google ScholarDigital Library
    20. Itu, 1990. International Telecommunication Union ITU-R Recommendation BT.709, Basic Parameter Values for the HDTV Standard for the Studio and for International Programme Exchange. Geneva. Formerly CCIR Rec. 709.Google Scholar
    21. Kajiya, J. T. 1986. The rendering equation. In SIGGRAPH ’86: Proc. of the 13th annual conference on Computer graphics and interactive techniques, 143–150. Google ScholarDigital Library
    22. Kang, S. 1998. Depth-painting for image based rendering applications. Tech. rep., Microsoft.Google Scholar
    23. Klinker, G. J., Shafer, S. A., and Kanade, T. 1988. The measurement of highlights in color images. International Journal of Computer Vision 2, 1, 7–32.Google ScholarCross Ref
    24. Koenderink, J. J., and van Doorn, A. J. 1979. The internal representation of solid shape with respect to vision. Biological Cybernetics 32, 211–216.Google ScholarDigital Library
    25. Koenderink, J. J., van Doorn, A. J., Kappers, A., and Todd, J. 2001. Ambiguity and the ‘mental eye’ in pictorial relief. Perception 30, 4, 431–448.Google ScholarCross Ref
    26. Langer, M. S., and Bülthoff, H. H. 2000. Depth discrimination from shading under diffuse lighting. Perception 29, 6, 649–660.Google ScholarCross Ref
    27. Langer, M. S., and Zucker, S. W. 1994. Shape-from-shading on a cloudy day. Journal of the Optical Society of America A 11, 467–478.Google ScholarCross Ref
    28. Lawson, R., Bülthoff, H. H., and Dumbell, S. 2003. Interactions between view changes and shape changes in picture – picture matching. Perception 32, 1465–1498.Google ScholarCross Ref
    29. Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. ACM Transactions on Graphics 23, 3, 303–308. Google ScholarDigital Library
    30. Matusik, W., Pfister, H., Brand, M., and McMillan, L. 2003. A data-driven reflectance model. ACM Transactions on Graphics 22, 3, 759–769. Google ScholarDigital Library
    31. McGuire, M., Matusik, W., Pfister, H., Hughes, J. F., and Durand, F. 2005. Defocus video matting. ACM Transactions on Graphics 24, 3, 567–576. Google ScholarDigital Library
    32. Metzger, W. 1975. Gesetze des Sehens (Laws of Seeing). Waldemar Kramer, Frankfurt.Google Scholar
    33. Motoyoshi, I., Nishida, S., and Adelson, E. H. 2005. Luminance re-mapping for the control of apparent material. In Proceedings of the 2nd ACM SIGGRAPH Symposium on Applied Perception in Graphics and Visualization, 165. Google ScholarDigital Library
    34. Naka, K. I., and Rushton, W. A. H. 1966. S-potentials from luminosity units in the retina of fish (cyprinidae). Journal of Physiology 185, 587–599.Google ScholarCross Ref
    35. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In SIGGRAPH ’01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 433–442. Google ScholarDigital Library
    36. Oliveira, M. M. 2002. Image-based modeling and rendering techniques: A survey. RITA – Revista de Informática Teórica e Aplicada IX (October), 37–66.Google Scholar
    37. Ostromoukhov, V., Donohue, C., and Jodoin, P. 2004. Fast hierarchical importance sampling with blue noise properties. ACM Transactions on Graphics 23, 3, 488–495. Google ScholarDigital Library
    38. Ostrovsky, Y., Cavanagh, P., and Sinha, P. 2005. Perceiving illumination inconsistencies. Perception 34, 1301–1314.Google ScholarCross Ref
    39. te Pas, S. F., and Pont, S. C. 2005. A comparison of material and illumination discrimination performance for real rough, real smooth and computer generated smooth spheres. In APGV ’05: Proceedings of the 2nd ACM Symposium on Appied Perception in Graphics and Visualization, 75–81. Google ScholarDigital Library
    40. te Pas, S. F., and Pont, S. C. 2005. Estimations of light source direction depend critically on material brdfs. Perception 34, 212. Supplement ECVP05, A Coruña.Google Scholar
    41. Ramamoorthi, R., and Hanrahan, P. 2001. A signal-processing framework for reflection. ACM Transactions on Graphics 23, 4, 1004–1042. Google ScholarDigital Library
    42. Reinhard, E., and Khan, E. A. 2005. Depth-of-field-based alpha-matte-extraction. In APGV ’05: Proceedings of the 2nd ACM Symposium on Appied Perception in Graphics and Visualization, 95–102. Google ScholarDigital Library
    43. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21 (September/October), 34–41. Google ScholarDigital Library
    44. Reinhard, E., Stark, M., Shirley, P., and Ferwerda, J. 2002. Photographic tone reproduction for digital images. ACM Trans. on Graphics 21, 3, 267–276. Google ScholarDigital Library
    45. Reinhard, E., Shirley, P., Ashikhmin, M., and Troscianko, T. 2004. Second order image statistics for computer graphics. In APGV ’04: Proc. of the 1st ACM Symposium on Applied Perception in Graphics and Visualization (APGV), 99–106. Google ScholarDigital Library
    46. Reinhard, E., Khan, E. A., Akyüz, A. O., Fleming, R. W., and Bülthoff, H. H. 2005. Image-based material editing. ACM SIGGRAPH Computer Animation Festival/Video Review, 152. Google ScholarDigital Library
    47. Reinhard, E., Ward, G., Pattanaik, S., and Debevec, P. 2005. High Dynamic Range Imaging: Acquisition, Display and Image-Based Lighting. Morgan Kaufmann Publishers, San Francisco. Google ScholarDigital Library
    48. Rother, C., Kolmogorov, V., and Blake, A. 2004. “grabcut” — interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics 23, 3, 309–314. Google ScholarDigital Library
    49. Ruderman, D. L. 1997. The statistics of natural images. Network: Computation in Neural Systems 5, 4, 517–548.Google ScholarCross Ref
    50. van der Schaaf, A. 1998. Natural image statistics and visual processing. PhD thesis, Rijksuniversiteit Groningen, The Netherlands.Google Scholar
    51. Simons, D. J., and Levin, D. T. 1997. Change blindness. Trends in Cognitive Science 1, 261–267.Google ScholarCross Ref
    52. Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In SIGGRAPH ’96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 259–268. Google ScholarDigital Library
    53. Sun, J., Yuan, L., Jia, J., and Shum, H.-Y. 2005. Image completion with structure propagation. ACM Transactions on Graphics 24, 3, 861–868. Google ScholarDigital Library
    54. Tarr, M. J., Kersten, D., and Bülthoff, H. H. 1999. Why the visual recognition system might encode the effects of illumination. Vision Research 39, 2259–2275.Google Scholar
    55. Todd, J. T., and Mingolla, E. 1983. Perception of surface curvature and direction of illuminant from patterns of shading. Journal of Experimental Psychology: Human Perception and Performance 9, 583–595.Google ScholarCross Ref
    56. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proc. IEEE International Conference on Computer Vision, 836–846. Google ScholarDigital Library
    57. Torralba, A., and Oliva, A. 2003. Statistics of natural image categories. Network: Computation in Neural Systems 14, 391–412.Google ScholarCross Ref
    58. Zelinka, S., Fang, H., Garland, M., and Hart, J. C. 2005. Interactive material replacement in photographs. In GI ’05: Proceedings of the 2005 conference on Graphics interface, 227–232. Google ScholarDigital Library
    59. Zhang, R., Tsai, P., Cryer, J., and Shah, M. 1999. Shape from shading: a survey. IEEE Trans. on Pattern Analysis and Machine Intelligence 21, 8, 690–706. Google ScholarDigital Library
    60. Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. In SIGGRAPH ’99: Proc. of the 26th annual conference on Computer graphics and interactive techniques, 205–214. Google ScholarDigital Library

ACM Digital Library Publication: