“Image-based material editing” by Khan, Reinhard, Fleming and Bülthoff

Photo editing software allows digital images to be blurred, warped or re-colored at the touch of a button. However, it is not currently possible to change the material appearance of an object except by painstakingly painting over the appropriate pixels. Here we present a method for automatically replacing one material with another, completely different material, starting with only a single high dynamic range image as input. Our approach exploits the fact that human vision is surprisingly tolerant of certain (sometimes enormous) physical inaccuracies, while being sensitive to others. By adjusting our simulations to be careful about those aspects to which the human visual system is sensitive, we are for the first time able to demonstrate significant material changes on the basis of a single photograph as input.

1. Adelson, E. 2001. On seeing stuff: The perception of materials by humans. In Proc. of the SPIE, Human Vision and Electronic Imaging VI, vol. 4299, 1–12.Google ScholarCross Ref
2. Belhumeur, P., Kriegman, D., and Yuille, A. 1999. The bas-relief ambiguity. International Journal of Computer Vision 1, 33–44. Google ScholarDigital Library
3. Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. 2000. Image inpainting. In SIGGRAPH ’00: Proceedings of the 27th annual conference on computer graphics and interactive techniques, 417–424. Google ScholarDigital Library
4. Blake, A., and Bülthoff, H. H. 1990. Does the brain know the physics of specular reflection? Nature 343, 6254, 165–168.Google Scholar
5. Blake, A., and Bülthoff, H. H. 1991. Shape from specularities: Computation and psychophysics. Philosophical Transactions of the Royal Society (London) Series B 331, 237–252.Google ScholarCross Ref
6. Chuang, Y., Zongker, D. E., Hindorff, J., Curless, B., Salesin, D., and Szeliski, R. 2000. Environment matting extensions: Towards higher accuracy and real-time capture. In SIGGRAPH ’00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 121–130. Google ScholarDigital Library
7. Debevec, P. E., and Malik, J. 1997. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH 97 Conference Proceedings, 369–378. Google ScholarDigital Library
8. Debevec, P. E. 1998. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with illumination and high dynamic range photography. In SIGGRAPH 98 Conference Proceedings, 45–50. Google ScholarDigital Library
9. Debevec, P. 2002. A tutorial on image-based lighting. IEEE Computer Graphics and Applications 22, 2, 26–34. Google ScholarDigital Library
10. Drew, M. S. 1994. Robust specularity detection from a single multi-illuminant color image. Computer Vision, Graphics, and Image Processing: Image Understanding 59, 3, 320–327. Google ScholarDigital Library
11. Drori, I., Cohen-Or, D., and Yeshurun, H. 2003. Fragment-based image completion. ACM Transactions on Graphics 22, 3, 303–312. Google ScholarDigital Library
12. Durand, F., and Dorsey, J. 2002. Fast bilateral filtering for the display of high-dynamic-range images. ACM Transactions on Graphics 21, 3, 257–266. Google ScholarDigital Library
13. Eisemann, E., and Durand, F. 2004. Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics 23, 3, 673–678. Google ScholarDigital Library
14. Fang, H., and Hart, J. C. 2004. Textureshop: Texture synthesis as a photographic editing tool. ACM Transactions on Graphics 23, 3, 354–358. Google ScholarDigital Library
15. Fleming, R. W., and Bülthoff, H. H. 2005. Low-level image cues in the perception of translucent materials. ACM Trans. on Applied Perception 2, 3, 346–382. Google ScholarDigital Library
16. Freeman, W. T. 1994. The generic viewpoint assumption in a framework for visual perception. Nature 368, 542–545.Google ScholarCross Ref
17. Freeman, W. T. 1996. Exploiting the generic viewpoint assumption. International Journal of Computer Vision 20, 3, 243–261. Google ScholarDigital Library
18. Hart, J. C., Francis, G. K., and Kauffman, L. H. 1994. Visualizing quaternion rotation. ACM Transactions on Graphics 13, 3, 256–276. Google ScholarDigital Library
19. Igarashi, T., Matsuoka, S., and Tanaka, H. 1999. Teddy: a sketching interface for 3d freeform design. In SIGGRAPH ’99: Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 409–416. Google ScholarDigital Library
20. Itu, 1990. International Telecommunication Union ITU-R Recommendation BT.709, Basic Parameter Values for the HDTV Standard for the Studio and for International Programme Exchange. Geneva. Formerly CCIR Rec. 709.Google Scholar
21. Kajiya, J. T. 1986. The rendering equation. In SIGGRAPH ’86: Proc. of the 13th annual conference on Computer graphics and interactive techniques, 143–150. Google ScholarDigital Library
22. Kang, S. 1998. Depth-painting for image based rendering applications. Tech. rep., Microsoft.Google Scholar
23. Klinker, G. J., Shafer, S. A., and Kanade, T. 1988. The measurement of highlights in color images. International Journal of Computer Vision 2, 1, 7–32.Google ScholarCross Ref
24. Koenderink, J. J., and van Doorn, A. J. 1979. The internal representation of solid shape with respect to vision. Biological Cybernetics 32, 211–216.Google ScholarDigital Library
25. Koenderink, J. J., van Doorn, A. J., Kappers, A., and Todd, J. 2001. Ambiguity and the ‘mental eye’ in pictorial relief. Perception 30, 4, 431–448.Google ScholarCross Ref
26. Langer, M. S., and Bülthoff, H. H. 2000. Depth discrimination from shading under diffuse lighting. Perception 29, 6, 649–660.Google ScholarCross Ref
27. Langer, M. S., and Zucker, S. W. 1994. Shape-from-shading on a cloudy day. Journal of the Optical Society of America A 11, 467–478.Google ScholarCross Ref
28. Lawson, R., Bülthoff, H. H., and Dumbell, S. 2003. Interactions between view changes and shape changes in picture – picture matching. Perception 32, 1465–1498.Google ScholarCross Ref
29. Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. ACM Transactions on Graphics 23, 3, 303–308. Google ScholarDigital Library
30. Matusik, W., Pfister, H., Brand, M., and McMillan, L. 2003. A data-driven reflectance model. ACM Transactions on Graphics 22, 3, 759–769. Google ScholarDigital Library
31. McGuire, M., Matusik, W., Pfister, H., Hughes, J. F., and Durand, F. 2005. Defocus video matting. ACM Transactions on Graphics 24, 3, 567–576. Google ScholarDigital Library
32. Metzger, W. 1975. Gesetze des Sehens (Laws of Seeing). Waldemar Kramer, Frankfurt.Google Scholar
33. Motoyoshi, I., Nishida, S., and Adelson, E. H. 2005. Luminance re-mapping for the control of apparent material. In Proceedings of the 2nd ACM SIGGRAPH Symposium on Applied Perception in Graphics and Visualization, 165. Google ScholarDigital Library
34. Naka, K. I., and Rushton, W. A. H. 1966. S-potentials from luminosity units in the retina of fish (cyprinidae). Journal of Physiology 185, 587–599.Google ScholarCross Ref
35. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In SIGGRAPH ’01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 433–442. Google ScholarDigital Library
36. Oliveira, M. M. 2002. Image-based modeling and rendering techniques: A survey. RITA – Revista de Informática Teórica e Aplicada IX (October), 37–66.Google Scholar
37. Ostromoukhov, V., Donohue, C., and Jodoin, P. 2004. Fast hierarchical importance sampling with blue noise properties. ACM Transactions on Graphics 23, 3, 488–495. Google ScholarDigital Library
38. Ostrovsky, Y., Cavanagh, P., and Sinha, P. 2005. Perceiving illumination inconsistencies. Perception 34, 1301–1314.Google ScholarCross Ref
39. te Pas, S. F., and Pont, S. C. 2005. A comparison of material and illumination discrimination performance for real rough, real smooth and computer generated smooth spheres. In APGV ’05: Proceedings of the 2nd ACM Symposium on Appied Perception in Graphics and Visualization, 75–81. Google ScholarDigital Library
40. te Pas, S. F., and Pont, S. C. 2005. Estimations of light source direction depend critically on material brdfs. Perception 34, 212. Supplement ECVP05, A Coruña.Google Scholar
41. Ramamoorthi, R., and Hanrahan, P. 2001. A signal-processing framework for reflection. ACM Transactions on Graphics 23, 4, 1004–1042. Google ScholarDigital Library
42. Reinhard, E., and Khan, E. A. 2005. Depth-of-field-based alpha-matte-extraction. In APGV ’05: Proceedings of the 2nd ACM Symposium on Appied Perception in Graphics and Visualization, 95–102. Google ScholarDigital Library
43. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21 (September/October), 34–41. Google ScholarDigital Library
44. Reinhard, E., Stark, M., Shirley, P., and Ferwerda, J. 2002. Photographic tone reproduction for digital images. ACM Trans. on Graphics 21, 3, 267–276. Google ScholarDigital Library
45. Reinhard, E., Shirley, P., Ashikhmin, M., and Troscianko, T. 2004. Second order image statistics for computer graphics. In APGV ’04: Proc. of the 1st ACM Symposium on Applied Perception in Graphics and Visualization (APGV), 99–106. Google ScholarDigital Library
46. Reinhard, E., Khan, E. A., Akyüz, A. O., Fleming, R. W., and Bülthoff, H. H. 2005. Image-based material editing. ACM SIGGRAPH Computer Animation Festival/Video Review, 152. Google ScholarDigital Library
47. Reinhard, E., Ward, G., Pattanaik, S., and Debevec, P. 2005. High Dynamic Range Imaging: Acquisition, Display and Image-Based Lighting. Morgan Kaufmann Publishers, San Francisco. Google ScholarDigital Library
48. Rother, C., Kolmogorov, V., and Blake, A. 2004. “grabcut” — interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics 23, 3, 309–314. Google ScholarDigital Library
49. Ruderman, D. L. 1997. The statistics of natural images. Network: Computation in Neural Systems 5, 4, 517–548.Google ScholarCross Ref
50. van der Schaaf, A. 1998. Natural image statistics and visual processing. PhD thesis, Rijksuniversiteit Groningen, The Netherlands.Google Scholar
51. Simons, D. J., and Levin, D. T. 1997. Change blindness. Trends in Cognitive Science 1, 261–267.Google ScholarCross Ref
52. Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In SIGGRAPH ’96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 259–268. Google ScholarDigital Library
53. Sun, J., Yuan, L., Jia, J., and Shum, H.-Y. 2005. Image completion with structure propagation. ACM Transactions on Graphics 24, 3, 861–868. Google ScholarDigital Library
54. Tarr, M. J., Kersten, D., and Bülthoff, H. H. 1999. Why the visual recognition system might encode the effects of illumination. Vision Research 39, 2259–2275.Google Scholar
55. Todd, J. T., and Mingolla, E. 1983. Perception of surface curvature and direction of illuminant from patterns of shading. Journal of Experimental Psychology: Human Perception and Performance 9, 583–595.Google ScholarCross Ref
56. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proc. IEEE International Conference on Computer Vision, 836–846. Google ScholarDigital Library
57. Torralba, A., and Oliva, A. 2003. Statistics of natural image categories. Network: Computation in Neural Systems 14, 391–412.Google ScholarCross Ref
58. Zelinka, S., Fang, H., Garland, M., and Hart, J. C. 2005. Interactive material replacement in photographs. In GI ’05: Proceedings of the 2005 conference on Graphics interface, 227–232. Google ScholarDigital Library
59. Zhang, R., Tsai, P., Cryer, J., and Shah, M. 1999. Shape from shading: a survey. IEEE Trans. on Pattern Analysis and Machine Intelligence 21, 8, 690–706. Google ScholarDigital Library
60. Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. In SIGGRAPH ’99: Proc. of the 26th annual conference on Computer graphics and interactive techniques, 205–214. Google ScholarDigital Library