“The effect of shape and illumination on material perception: model and applications” by Serrano, Chen, Wang, Piovarči, Seidel, et al. …

Material appearance hinges on material reflectance properties but also surface geometry and illumination. The unlimited number of potential combinations between these factors makes understanding and predicting material appearance a very challenging task. In this work, we collect a large-scale dataset of perceptual ratings of appearance attributes with more than 215,680 responses for 42,120 distinct combinations of material, shape, and illumination. The goal of this dataset is twofold. First, we analyze for the first time the effects of illumination and geometry in material perception across such a large collection of varied appearances. We connect our findings to those of the literature, discussing how previous knowledge generalizes across very diverse materials, shapes, and illuminations. Second, we use the collected dataset to train a deep learning architecture for predicting perceptual attributes that correlate with human judgments. We demonstrate the consistent and robust behavior of our predictor in various challenging scenarios, which, for the first time, enables estimating perceived material attributes from general 2D images. Since our predictor relies on the final appearance in an image, it can compare appearance properties across different geometries and illumination conditions. Finally, we demonstrate several applications that use our predictor, including appearance reproduction using 3D printing, BRDF editing by integrating our predictor in a differentiable renderer, illumination design, or material recommendations for scene design.

1. Wendy J Adams, Gizem Kucukoglu, Michael S Landy, and Rafał K Mantiuk. 2018. Naturally glossy: Gloss perception, illumination statistics, and tone mapping. Journal of Vision 18, 13 (2018), 4–4.Google ScholarCross Ref
2. Alan Agresti. 2003. Categorical data analysis. Vol. 482. John Wiley & Sons.Google Scholar
3. Barton L Anderson. 2011. Visual perception of materials and surfaces. Current Biology 21, 24 (2011), R978–R983.Google ScholarCross Ref
4. Barton L Anderson and Juno Kim. 2009. Image statistics do not explain the perception of gloss and lightness. Journal of Vision 9, 11 (2009), 10–10.Google ScholarCross Ref
5. Teun Baar, Sepideh Samadzadegan, Hans Brettel, Philipp Urban, and Maria V Ortiz Segovia. 2014. Printing gloss effects in a 2.5 D system. In Measuring, Modeling, and Reproducing Material Appearance, Vol. 9018. International Society for Optics and Photonics, 90180M.Google Scholar
6. Jacob Beck and Slava Prazdny. 1981. Highlights and the perception of glossiness. Perception & Psychophysics 30, 4 (1981), 407–410.Google ScholarCross Ref
7. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material Recognition in the Wild with the Materials in Context Database. Computer Vision and Pattern Recognition (2015).Google Scholar
8. Julia Berzhanskaya, Gurumurthy Swaminathan, Jacob Beck, and Ennio Mingolla. 2005. Remote Effects of Highlights on Gloss Perception. Perception 34, 5 (2005), 565–575.Google ScholarCross Ref
9. J Bieron and P Peers. 2020. An adaptive brdf fitting metric. In Computer Graphics Forum, Vol. 39. 59–74.Google ScholarCross Ref
10. Adrien Bousseau, Emmanuelle Chapoulie, Ravi Ramamoorthi, and Maneesh Agrawala. 2011. Optimizing environment maps for material depiction. In Computer Graphics Forum, Vol. 30. 1171–1180.Google ScholarDigital Library
11. Adrien Bousseau, James P O’shea, Frédo Durand, Ravi Ramamoorthi, and Maneesh Agrawala. 2013. Gloss perception in painterly and cartoon rendering. ACM Trans. Graph. 32, 2 (2013), 1–13.Google ScholarDigital Library
12. Paul Brossier, Juan Pablo Bello, and Mark D Plumbley. 2004. Real-time temporal segmentation of note objects in music signals. In Proceedings of ICMC 2004, the 30th Annual International Computer Music Conference.Google Scholar
13. Alice C Chadwick and RW Kentridge. 2015. The perception of gloss: A review. Vision Research 109 (2015), 221–235.Google ScholarCross Ref
14. Rune Haubo B Christensen. 2018. Cumulative link models for ordinal regression with the R package ordinal. Submitted in J. Stat. Software (2018).Google Scholar
15. Douglas W Cunningham and Christian Wallraven. 2011. Experimental design: From user studies to psychophysics. CRC Press.Google Scholar
16. CW Dawson, Nick J Mount, Robert J Abrahart, and John Louis. 2014. Sensitivity analysis for comparison, validation and physical legitimacy of neural network-based hydrological models. Journal of Hydroinformatics 16, 2 (2014), 407–424.Google ScholarCross Ref
17. Johanna Delanoy, Manuel Lagunas, Ignacio Galve, Diego Gutierrez, Ana Serrano, Roland Fleming, and Belen Masia. 2020. The Role of Objective and Subjective Measures in Material Similarity Learning. In ACM SIGGRAPH Posters. Article 51, 2 pages.Google Scholar
18. Katja Doerschner, Laurence T. Maloney, and Huseyin Boyaci. 2010. Perceived glossiness in high dynamic range scenes. Journal of Vision 10, 9, Article 11 (2010).Google ScholarCross Ref
19. Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016. Toward a learning science for complex crowdsourcing tasks. In Human Factors in Computing Systems. 2623–2634.Google Scholar
20. Ron O. Dror, Alan S. Willsky, and Edward H. Adelson. 2004. Statistical characterization of real-world illumination. Journal of Vision 4 (2004), 821–837.Google ScholarCross Ref
21. Jonathan Dupuy and Wenzel Jakob. 2018. An adaptive parameterization for efficient material acquisition and rendering. ACM Trans. Graph. 37, 6 (2018), 1–14.Google ScholarDigital Library
22. Frédo Durand and Julie Dorsey. 2002. Fast bilateral filtering for the display of high-dynamic-range images. In Proc. ACM SIGGRAPH. 257–266.Google ScholarDigital Library
23. Willemijn Elkhuizen, Tessa Essers, Yu Song, Jo Geraedts, Clemens Weijkamp, Joris Dik, and Sylvia Pont. 2019. Gloss, Color, and Topography Scanning for Reproducing a Painting’s Appearance Using 3D Printing. Journal on Computing and Cultural Heritage (JOCCH) 12, 4 (2019), 1–22.Google Scholar
24. Franz Faul. 2019. The influence of Fresnel effects on gloss perception. Journal of Vision 19, 13 (2019), 1–39.Google ScholarCross Ref
25. Jiří Filip. 2015. Analyzing and predicting anisotropic effects of BRDFs. In Proc. ACM Symposium on Applied Perception. 25–32.Google ScholarDigital Library
26. J. Filip and R. Vávra. 2014. Template-Based Sampling of Anisotropic BRDFs. Computer Graphics Forum 33, 7 (2014), 91–99.Google ScholarDigital Library
27. Roland W Fleming. 2014. Visual perception of materials and their properties. Vision Research 94 (2014), 62–75.Google ScholarCross Ref
28. Roland W Fleming. 2017. Material perception. Annual Review of Vision Science 3 (2017), 365–388.Google ScholarCross Ref
29. Roland W. Fleming and Heinrich H. Bülthoff. 2005. Low-Level Image Cues in the Perception of Translucent Materials. ACM Trans. Appl. Percept. 2, 3 (2005), 346–382.Google ScholarDigital Library
30. Roland W. Fleming, Ron O. Dror, and Edward H. Adelson. 2003. Real-world illumination and the perception of surface reflectance properties. Journal of Vision 3, 5 (2003), 3–3.Google ScholarCross Ref
31. Roland W Fleming, Shin’ya Nishida, and Karl R Gegenfurtner. 2015. Perception of material properties. Vision Research 115 (2015), 157–62.Google ScholarCross Ref
32. Roland W Fleming and Katherine R Storrs. 2019. Learning to see stuff. Current Opinion in Behavioral Sciences 30 (2019), 100–108.Google ScholarCross Ref
33. Adria Fores, James Ferwerda, and Jinwei Gu. 2012. Toward a perceptually based metric for BRDF modeling. In Proc. Color and Imaging Conference. 142–148.Google Scholar
34. A. Gilchrist, Kossyfidis C., Bonato F., and et al. 1999. An anchoring theory of lightness perception. Psychol Rev. 106, 4 (1999), 795–834.Google ScholarCross Ref
35. Ioannis Gkioulekas, Bruce Walter, Edward H Adelson, Kavita Bala, and Todd Zickler. 2015. On the appearance of translucent edges. In Computer Vision and Pattern Recognition. 5528–5536.Google Scholar
36. Dar’ya Guarnera, Giuseppe Claudio Guarnera, Matteo Toscani, Mashhuda Glencross, Baihua Li, Jon Yngve Hardeberg, and Karl R Gegenfurtner. 2018. Perceptually validated cross-renderer analytical BRDF parameter remapping. IEEE Transactions on Visualization and Computer Graphics 26, 6 (2018), 2258–2272.Google ScholarCross Ref
37. Sabrina Hansmann-Roth and Pascal Mamassian. 2017. A Glossy Simultaneous Contrast: Conjoint Measurements of Gloss and Lightness. i-Perception 8, 1 (2017).Google Scholar
38. Vlastimil Havran, Jiri Filip, and Karol Myszkowski. 2016. Perceptually motivated BRDF comparison using single image. In Computer Graphics Forum, Vol. 35. 1–12.Google ScholarDigital Library
39. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition. 770–778.Google Scholar
40. Jeffrey Heer and Michael Bostock. 2010. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In Human Factors in Computing Systems. 203–212.Google Scholar
41. Bingyang Hu, Jie Guo, Yanjun Chen, Mengtian Li, and Yanwen Guo. 2020. DeepBRDF: A Deep Representation for Manipulating Measured BRDF. In Computer Graphics Forum, Vol. 39. 157–166.Google ScholarCross Ref
42. R.S. Hunter. 1937. Methods of Determining Gloss. Part of Journal of Research of the National Bureau of Standards 18 (1937), 19–39.Google Scholar
43. R.S. Hunter and R.W. Harold. 1987. The measurement of appearance (2nd ed.). Wiley, New York.Google Scholar
44. Juno Kim, Phillip J. Marlow, and Barton L. Anderson. 2012. The dark side of gloss. Nature Neuroscience 15, 11 (2012), 1590–1595.Google ScholarCross Ref
45. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980 (2014).Google Scholar
46. Manuel Lagunas, Sandra Malpica, Ana Serrano, Elena Garces, Diego Gutierrez, and Belen Masia. 2019. A Similarity Measure for Material Appearance. ACM Trans. Graph. 38, 4 (2019).Google ScholarDigital Library
47. Manuel Lagunas, Ana Serrano, Diego Gutierrez, and Belen Masia. 2021. The joint role of geometry and illumination on material recognition. Journal of Vision 21, 2 (2021). Google ScholarCross Ref
48. Y-X. Landy, M.S. Landy, and L.T. Maloney. 2008. Conjoint measurement of gloss and surface texture. Psychological Science 19, 2 (2008), 196–204.Google ScholarCross Ref
49. Guillaume Lavoué, Nicolas Bonneel, Jean-Philippe Farrugia, and Cyril Soler. 2021. Perceptual quality of BRDF approximations: dataset and metrics. Computer Graphics Forum 40, 2 (2021).Google Scholar
50. Frédéric B Leloup, Michael R Pointer, Philip Dutré, and Peter Hanselaer. 2010. Geometry of illumination, luminance contrast, and gloss perception. J. Opt. Soc. Am. A 27, 9 (2010), 2046–2054.Google ScholarCross Ref
51. A Luongo, V Falster, MB Doest, MM Ribo, ER Eiriksson, DB Pedersen, and JR Frisvad. 2019. Microstructure Control in 3D Printing with Digital Light Processing. In Computer Graphics Forum, Vol. 39. 347–359.Google ScholarCross Ref
52. Phillip J Marlow and Barton L Anderson. 2013. Generative constraints on image cues for perceived gloss. Journal of Vision 13, 14 (2013), 2–2.Google ScholarCross Ref
53. Phillip J Marlow, Juno Kim, and Barton L Anderson. 2012. The perception and misperception of specular surface reflectance. Current Biology 22, 20 (2012), 1909–1913.Google ScholarCross Ref
54. Wojciech Matusik. 2003. A data-driven reflectance model. Ph.D. Dissertation. Massachusetts Institute of Technology.Google ScholarDigital Library
55. Wojciech Matusik, Boris Ajdin, Jinwei Gu, Jason Lawrence, Hendrik P. A. Lensch, Fabio Pellacini, and Szymon Rusinkiewicz. 2009. Printing Spatially-Varying Reflectance. ACM Trans. Graph. 28, 5 (2009), 1–9.Google ScholarDigital Library
56. Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-Driven Reflectance Model. ACM Trans. Graph. 22, 3 (2003), 759–769.Google ScholarDigital Library
57. Peter McCullagh. 1980. Regression models for ordinal data. Journal of the Royal Statistical Society: Series B (Methodological) 42, 2 (1980), 109–127.Google ScholarCross Ref
58. Isamu Motoyoshi and Hiroaki Matoba. 2012. Variability in constancy of the perceived surface reflectance across different illumination statistics. Vision Research 53, 1 (2012), 30–39.Google ScholarCross Ref
59. Isamu Motoyoshi, Shin’ya Nishida, Lavanya Sharan, and Edward H Adelson. 2007. Image statistics and the perception of surface qualities. Nature 447, 7141 (2007), 206–209.Google Scholar
60. Addy Ngan, Frédo Durand, and Wojciech Matusik. 2006. Image-driven Navigation of Analytical BRDF Models. Rendering Techniques (2006), 399–407.Google Scholar
61. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A retargetable forward and inverse renderer. ACM Trans. Graph. 38, 6 (2019), 1–17.Google ScholarDigital Library
62. Shin’ya Nishida and Mikio Shinya. 1998. Use of image-based information in judgments of surface-reflectance properties. J. Opt. Soc. Am. A 15, 12 (1998), 2951–2965.Google ScholarCross Ref
63. Jum C Nunnally. 1994. Psychometric theory 3E. Tata McGraw-hill education.Google Scholar
64. Maria Olkkonen and David H Brainard. 2010. Perceived glossiness and lightness under real-world illumination. Journal of Vision 10, 9 (2010), 5–5.Google ScholarCross Ref
65. Fabio Pellacini, James A Ferwerda, and Donald P Greenberg. 2000. Toward a psychophysically-based light reflection model for image synthesis. In Proc. ACM SIGGRAPH. 55–64.Google ScholarDigital Library
66. Thiago Pereira and Szymon Rusinkiewicz. 2012. Gamut mapping spatially varying reflectance with an improved BRDF similarity metric. In Computer Graphics Forum, Vol. 31. 1557–1566.Google ScholarDigital Library
67. Michal Piovarči, Michael Wessely, Michał Jagielski, Marc Alexa, Wojciech Matusik, and Piotr Didyk. 2017. Directional screens. In Proceedings of the 1st Annual ACM Symposium on Computational Fabrication. ACM, 1.Google ScholarDigital Library
68. Michal Piovarči, Michael Foshey, Vahid Babaei, Szymon Rusinkiewicz, Wojciech Matusik, and Piotr Didyk. 2020. Towards Spatially Varying Gloss Reproduction for 3D Printing. ACM Trans. Graph. 39, 6, Article 206 (2020).Google ScholarDigital Library
69. Sylvia C Pont and Susan F te Pas. 2006. Material — Illumination Ambiguities and the Perception of Solid Objects. Perception 35, 10 (2006), 1331–1350.Google ScholarCross Ref
70. Ganesh Ramanarayanan, James Ferwerda, Bruce Walter, and Kavita Bala. 2007. Visual equivalence: towards a new standard for image fidelity. ACM Trans. Graph. 26, 3 (2007), 76–es.Google ScholarDigital Library
71. Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. 2002. Photographic tone reproduction for digital images. In Proc. ACM SIGGRAPH. 267–276.Google ScholarDigital Library
72. Olivier Rouiller, Bernd Bickel, Jan Kautz, Wojciech Matusik, and Marc Alexa. 2013. 3D-printing spatially varying BRDFs. IEEE Computer Graphics and Applications 33, 6 (2013), 48–57.Google ScholarDigital Library
73. Sepideh Samadzadegan, Teun Baar, Philipp Urban, Maria V Ortiz Segovia, and Jana Blahová. 2015. Controlling colour-printed gloss by varnish-halftones. In Measuring, Modeling, and Reproducing Material Appearance 2015, Vol. 9398. International Society for Optics and Photonics, 93980V.Google Scholar
74. A. C. Schmid, P. Barla, and K. Doerschner. 2020. Material category determined by specular reflection structure mediates the processing of image features for perceived gloss. bioRxiv (2020).Google Scholar
75. Gabriel Schwartz and Ko Nishino. 2019. Recognizing material properties from images. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 8 (2019), 1981–1995.Google ScholarDigital Library
76. Ana Serrano, Diego Gutierrez, Karol Myszkowski, Hans-Peter Seidel, and Belen Masia. 2016. An intuitive control space for material appearance. ACM Trans. Graph. 35, 6 (2016).Google ScholarDigital Library
77. Lavanya Sharan, Yuanzhen Li, Isamu Motoyoshi, Shin’ya Nishida, and Edward H Adelson. 2008. Image statistics for surface reflectance perception. J. Opt. Soc. Am. A 25, 4 (2008), 846–865.Google ScholarCross Ref
78. Lavanya Sharan, Ruth Rosenholtz, and Edward H Adelson. 2014. Accuracy and speed of material categorization in real-world images. Journal of Vision 14, 9 (2014), 12–12.Google ScholarCross Ref
79. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014).Google Scholar
80. Katherine R Storrs and Roland W Fleming. 2020. Unsupervised Learning Predicts Human Perception and Misperception of Specular Surface Reflectance. bioRxiv (2020).Google Scholar
81. Tiancheng Sun, Henrik Wann Jensen, and Ravi Ramamoorthi. 2018. Connecting measured BRDFs to analytic brdfs by data-driven diffuse-specular separation. ACM Trans. Graph. 37, 6 (2018), 1–15.Google ScholarDigital Library
82. Tiancheng Sun, Ana Serrano, Diego Gutierrez, and Belen Masia. 2017. Attribute-preserving gamut mapping of measured BRDFs. In Computer Graphics Forum, Vol. 36. 47–54.Google ScholarDigital Library
83. Alejandro Sztrajman, Jaroslav Křivánek, Alexander Wilkie, and Tim Weyrich. 2019. Image-based Remapping of Spatially-Varying Material Appearance. Journal of Computer Graphics Techniques (JCGT) 8, 4 (2019), 1–30.Google Scholar
84. James T. Todd, J. Farley Norman, and Ennio Mingolla. 2004. Lightness Constancy in the Presence of Specular Highlights. Psychological Science 15, 1 (2004), 33–39.Google ScholarCross Ref
85. Matteo Toscani, Dar’ya Guarnera, Giuseppe Claudio Guarnera, Jon Yngve Hardeberg, and Karl R Gegenfurtner. 2020. Three perceptual dimensions for specular and diffuse reflection. ACM Transactions on Applied Perception (TAP) 17, 2 (2020), 1–26.Google ScholarDigital Library
86. Matteo Toscani and Matteo Valsecchi. 2019. Lightness Discrimination Depends More on Bright Rather Than Shaded Regions of Three-Dimensional Objects. i-Perception 10, 6 (2019), 1–10.Google Scholar
87. Matteo Toscani, Matteo Valsecchi, and Karl R Gegenfurtner. 2017. Lightness perception for matte and glossy complex shapes. Vision Research 131 (2017), 82–95.Google ScholarCross Ref
88. TS Trowbridge and Karl P Reitz. 1975. Average irregularity representation of a rough surface for ray reflection. J. Opt. Soc. Am. A 65, 5 (1975), 531–536.Google ScholarCross Ref
89. Peter Vangorp, Pascal Barla, and Roland W Fleming. 2017. The perception of hazy gloss. Journal of Vision 17, 5 (2017), 19–19.Google ScholarCross Ref
90. Peter Vangorp and Philip Dutré. 2008. Shape-dependent gloss correction. In Proc. Applied Perception in Graphics and Visualization. 123–130.Google ScholarDigital Library
91. Peter Vangorp, Jurgen Laurijssen, and Philip Dutré. 2007. The influence of shape on the perception of material reflectance. In Proc. ACM SIGGRAPH. 77:1–77:9.Google Scholar
92. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces. Rendering Techniques (2007), 195–206.Google ScholarDigital Library
93. Gregory J. Ward. 1992. Measuring and Modeling Anisotropic Reflection. Proc. ACM SIGGRAPH 26, 2 (1992), 265–272.Google ScholarDigital Library
94. Peter Welinder, Steve Branson, Pietro Perona, and Serge Belongie. 2010. The multidimensional wisdom of crowds. Advances in Neural Information Processing Systems 23 (2010), 2424–2432.Google ScholarDigital Library
95. G. Wendt, F. Faul, V. Ekroll, and R. Mausfeld. 2010. Disparity, motion, and color information improve gloss constancy performance. Journal of Vision 10, 9 (2010).Google ScholarCross Ref
96. Tim Weyrich, Pieter Peers, Wojciech Matusik, and Szymon Rusinkiewicz. 2009. Fabricating microgeometry for custom surface reflectance. ACM Trans. Graph. 28, 3 (2009), 32.Google ScholarDigital Library
97. Christiane B Wiebel, Matteo Toscani, and Karl R Gegenfurtner. 2015. Statistical correlates of perceived gloss in natural images. Vision Research 115 (2015), 175–187.Google ScholarCross Ref
98. Josh Wills, Sameer Agarwal, David Kriegman, and Serge Belongie. 2009. Toward a perceptual space for gloss. ACM Trans. Graph. 28, 4 (2009), 1–15.Google ScholarDigital Library
99. Haoyu Xu, Zhenqi Han, Songlin Feng, Han Zhou, and Yuchun Fang. 2018. Foreign object debris material recognition based on convolutional neural networks. Eurasip Journal on Image and Video Processing 2018, 1 (2018), 1–10.Google ScholarCross Ref
100. Fan Zhang, Huib de Ridder, Pascal Barla, and Sylvia Pont. 2020a. Effects of light map orientation and shape on the visual perception of canonical materials. Journal of Vision 20, 4, Article 13 (2020), 18 pages.Google ScholarCross Ref
101. Fan Zhang, Huib de Ridder, Pascal Barla, and Sylvia Pont. 2020b. A systematic approach to testing and predicting light-material interactions. Journal of Vision 19, 4, Article 11 (2020), 22 pages.Google Scholar
102. Fan Zhang, Huib de Ridder, and Sylvia Pont. 2015. The influence of lighting on visual perception of material qualities. In Human Vision and Electronic Imaging, Vol. SPIE 9394. 239–248.Google Scholar
103. Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian Material Synthesis. ACM Trans. Graph. 37, 4, Article 76 (2018).Google ScholarDigital Library