“The effect of shape and illumination on material perception: model and applications” by Serrano, Chen, Wang, Piovarči, Seidel, et al. …

  • ©Ana Serrano, Bin Chen, Chao Wang, Michal Piovarči, Hans-Peter Seidel, Piotr Didyk, and Karol Myszkowski




    The effect of shape and illumination on material perception: model and applications



    Material appearance hinges on material reflectance properties but also surface geometry and illumination. The unlimited number of potential combinations between these factors makes understanding and predicting material appearance a very challenging task. In this work, we collect a large-scale dataset of perceptual ratings of appearance attributes with more than 215,680 responses for 42,120 distinct combinations of material, shape, and illumination. The goal of this dataset is twofold. First, we analyze for the first time the effects of illumination and geometry in material perception across such a large collection of varied appearances. We connect our findings to those of the literature, discussing how previous knowledge generalizes across very diverse materials, shapes, and illuminations. Second, we use the collected dataset to train a deep learning architecture for predicting perceptual attributes that correlate with human judgments. We demonstrate the consistent and robust behavior of our predictor in various challenging scenarios, which, for the first time, enables estimating perceived material attributes from general 2D images. Since our predictor relies on the final appearance in an image, it can compare appearance properties across different geometries and illumination conditions. Finally, we demonstrate several applications that use our predictor, including appearance reproduction using 3D printing, BRDF editing by integrating our predictor in a differentiable renderer, illumination design, or material recommendations for scene design.


    1. Wendy J Adams, Gizem Kucukoglu, Michael S Landy, and Rafał K Mantiuk. 2018. Naturally glossy: Gloss perception, illumination statistics, and tone mapping. Journal of Vision 18, 13 (2018), 4–4.Google ScholarCross Ref
    2. Alan Agresti. 2003. Categorical data analysis. Vol. 482. John Wiley & Sons.Google Scholar
    3. Barton L Anderson. 2011. Visual perception of materials and surfaces. Current Biology 21, 24 (2011), R978–R983.Google ScholarCross Ref
    4. Barton L Anderson and Juno Kim. 2009. Image statistics do not explain the perception of gloss and lightness. Journal of Vision 9, 11 (2009), 10–10.Google ScholarCross Ref
    5. Teun Baar, Sepideh Samadzadegan, Hans Brettel, Philipp Urban, and Maria V Ortiz Segovia. 2014. Printing gloss effects in a 2.5 D system. In Measuring, Modeling, and Reproducing Material Appearance, Vol. 9018. International Society for Optics and Photonics, 90180M.Google Scholar
    6. Jacob Beck and Slava Prazdny. 1981. Highlights and the perception of glossiness. Perception & Psychophysics 30, 4 (1981), 407–410.Google ScholarCross Ref
    7. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material Recognition in the Wild with the Materials in Context Database. Computer Vision and Pattern Recognition (2015).Google Scholar
    8. Julia Berzhanskaya, Gurumurthy Swaminathan, Jacob Beck, and Ennio Mingolla. 2005. Remote Effects of Highlights on Gloss Perception. Perception 34, 5 (2005), 565–575.Google ScholarCross Ref
    9. J Bieron and P Peers. 2020. An adaptive brdf fitting metric. In Computer Graphics Forum, Vol. 39. 59–74.Google ScholarCross Ref
    10. Adrien Bousseau, Emmanuelle Chapoulie, Ravi Ramamoorthi, and Maneesh Agrawala. 2011. Optimizing environment maps for material depiction. In Computer Graphics Forum, Vol. 30. 1171–1180.Google ScholarDigital Library
    11. Adrien Bousseau, James P O’shea, Frédo Durand, Ravi Ramamoorthi, and Maneesh Agrawala. 2013. Gloss perception in painterly and cartoon rendering. ACM Trans. Graph. 32, 2 (2013), 1–13.Google ScholarDigital Library
    12. Paul Brossier, Juan Pablo Bello, and Mark D Plumbley. 2004. Real-time temporal segmentation of note objects in music signals. In Proceedings of ICMC 2004, the 30th Annual International Computer Music Conference.Google Scholar
    13. Alice C Chadwick and RW Kentridge. 2015. The perception of gloss: A review. Vision Research 109 (2015), 221–235.Google ScholarCross Ref
    14. Rune Haubo B Christensen. 2018. Cumulative link models for ordinal regression with the R package ordinal. Submitted in J. Stat. Software (2018).Google Scholar
    15. Douglas W Cunningham and Christian Wallraven. 2011. Experimental design: From user studies to psychophysics. CRC Press.Google Scholar
    16. CW Dawson, Nick J Mount, Robert J Abrahart, and John Louis. 2014. Sensitivity analysis for comparison, validation and physical legitimacy of neural network-based hydrological models. Journal of Hydroinformatics 16, 2 (2014), 407–424.Google ScholarCross Ref
    17. Johanna Delanoy, Manuel Lagunas, Ignacio Galve, Diego Gutierrez, Ana Serrano, Roland Fleming, and Belen Masia. 2020. The Role of Objective and Subjective Measures in Material Similarity Learning. In ACM SIGGRAPH Posters. Article 51, 2 pages.Google Scholar
    18. Katja Doerschner, Laurence T. Maloney, and Huseyin Boyaci. 2010. Perceived glossiness in high dynamic range scenes. Journal of Vision 10, 9, Article 11 (2010).Google ScholarCross Ref
    19. Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016. Toward a learning science for complex crowdsourcing tasks. In Human Factors in Computing Systems. 2623–2634.Google Scholar
    20. Ron O. Dror, Alan S. Willsky, and Edward H. Adelson. 2004. Statistical characterization of real-world illumination. Journal of Vision 4 (2004), 821–837.Google ScholarCross Ref
    21. Jonathan Dupuy and Wenzel Jakob. 2018. An adaptive parameterization for efficient material acquisition and rendering. ACM Trans. Graph. 37, 6 (2018), 1–14.Google ScholarDigital Library
    22. Frédo Durand and Julie Dorsey. 2002. Fast bilateral filtering for the display of high-dynamic-range images. In Proc. ACM SIGGRAPH. 257–266.Google ScholarDigital Library
    23. Willemijn Elkhuizen, Tessa Essers, Yu Song, Jo Geraedts, Clemens Weijkamp, Joris Dik, and Sylvia Pont. 2019. Gloss, Color, and Topography Scanning for Reproducing a Painting’s Appearance Using 3D Printing. Journal on Computing and Cultural Heritage (JOCCH) 12, 4 (2019), 1–22.Google Scholar
    24. Franz Faul. 2019. The influence of Fresnel effects on gloss perception. Journal of Vision 19, 13 (2019), 1–39.Google ScholarCross Ref
    25. Jiří Filip. 2015. Analyzing and predicting anisotropic effects of BRDFs. In Proc. ACM Symposium on Applied Perception. 25–32.Google ScholarDigital Library
    26. J. Filip and R. Vávra. 2014. Template-Based Sampling of Anisotropic BRDFs. Computer Graphics Forum 33, 7 (2014), 91–99.Google ScholarDigital Library
    27. Roland W Fleming. 2014. Visual perception of materials and their properties. Vision Research 94 (2014), 62–75.Google ScholarCross Ref
    28. Roland W Fleming. 2017. Material perception. Annual Review of Vision Science 3 (2017), 365–388.Google ScholarCross Ref
    29. Roland W. Fleming and Heinrich H. Bülthoff. 2005. Low-Level Image Cues in the Perception of Translucent Materials. ACM Trans. Appl. Percept. 2, 3 (2005), 346–382.Google ScholarDigital Library
    30. Roland W. Fleming, Ron O. Dror, and Edward H. Adelson. 2003. Real-world illumination and the perception of surface reflectance properties. Journal of Vision 3, 5 (2003), 3–3.Google ScholarCross Ref
    31. Roland W Fleming, Shin’ya Nishida, and Karl R Gegenfurtner. 2015. Perception of material properties. Vision Research 115 (2015), 157–62.Google ScholarCross Ref
    32. Roland W Fleming and Katherine R Storrs. 2019. Learning to see stuff. Current Opinion in Behavioral Sciences 30 (2019), 100–108.Google ScholarCross Ref
    33. Adria Fores, James Ferwerda, and Jinwei Gu. 2012. Toward a perceptually based metric for BRDF modeling. In Proc. Color and Imaging Conference. 142–148.Google Scholar
    34. A. Gilchrist, Kossyfidis C., Bonato F., and et al. 1999. An anchoring theory of lightness perception. Psychol Rev. 106, 4 (1999), 795–834.Google ScholarCross Ref
    35. Ioannis Gkioulekas, Bruce Walter, Edward H Adelson, Kavita Bala, and Todd Zickler. 2015. On the appearance of translucent edges. In Computer Vision and Pattern Recognition. 5528–5536.Google Scholar
    36. Dar’ya Guarnera, Giuseppe Claudio Guarnera, Matteo Toscani, Mashhuda Glencross, Baihua Li, Jon Yngve Hardeberg, and Karl R Gegenfurtner. 2018. Perceptually validated cross-renderer analytical BRDF parameter remapping. IEEE Transactions on Visualization and Computer Graphics 26, 6 (2018), 2258–2272.Google ScholarCross Ref
    37. Sabrina Hansmann-Roth and Pascal Mamassian. 2017. A Glossy Simultaneous Contrast: Conjoint Measurements of Gloss and Lightness. i-Perception 8, 1 (2017).Google Scholar
    38. Vlastimil Havran, Jiri Filip, and Karol Myszkowski. 2016. Perceptually motivated BRDF comparison using single image. In Computer Graphics Forum, Vol. 35. 1–12.Google ScholarDigital Library
    39. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition. 770–778.Google Scholar
    40. Jeffrey Heer and Michael Bostock. 2010. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In Human Factors in Computing Systems. 203–212.Google Scholar
    41. Bingyang Hu, Jie Guo, Yanjun Chen, Mengtian Li, and Yanwen Guo. 2020. DeepBRDF: A Deep Representation for Manipulating Measured BRDF. In Computer Graphics Forum, Vol. 39. 157–166.Google ScholarCross Ref
    42. R.S. Hunter. 1937. Methods of Determining Gloss. Part of Journal of Research of the National Bureau of Standards 18 (1937), 19–39.Google Scholar
    43. R.S. Hunter and R.W. Harold. 1987. The measurement of appearance (2nd ed.). Wiley, New York.Google Scholar
    44. Juno Kim, Phillip J. Marlow, and Barton L. Anderson. 2012. The dark side of gloss. Nature Neuroscience 15, 11 (2012), 1590–1595.Google ScholarCross Ref
    45. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980 (2014).Google Scholar
    46. Manuel Lagunas, Sandra Malpica, Ana Serrano, Elena Garces, Diego Gutierrez, and Belen Masia. 2019. A Similarity Measure for Material Appearance. ACM Trans. Graph. 38, 4 (2019).Google ScholarDigital Library
    47. Manuel Lagunas, Ana Serrano, Diego Gutierrez, and Belen Masia. 2021. The joint role of geometry and illumination on material recognition. Journal of Vision 21, 2 (2021). Google ScholarCross Ref
    48. Y-X. Landy, M.S. Landy, and L.T. Maloney. 2008. Conjoint measurement of gloss and surface texture. Psychological Science 19, 2 (2008), 196–204.Google ScholarCross Ref
    49. Guillaume Lavoué, Nicolas Bonneel, Jean-Philippe Farrugia, and Cyril Soler. 2021. Perceptual quality of BRDF approximations: dataset and metrics. Computer Graphics Forum 40, 2 (2021).Google Scholar
    50. Frédéric B Leloup, Michael R Pointer, Philip Dutré, and Peter Hanselaer. 2010. Geometry of illumination, luminance contrast, and gloss perception. J. Opt. Soc. Am. A 27, 9 (2010), 2046–2054.Google ScholarCross Ref
    51. A Luongo, V Falster, MB Doest, MM Ribo, ER Eiriksson, DB Pedersen, and JR Frisvad. 2019. Microstructure Control in 3D Printing with Digital Light Processing. In Computer Graphics Forum, Vol. 39. 347–359.Google ScholarCross Ref
    52. Phillip J Marlow and Barton L Anderson. 2013. Generative constraints on image cues for perceived gloss. Journal of Vision 13, 14 (2013), 2–2.Google ScholarCross Ref
    53. Phillip J Marlow, Juno Kim, and Barton L Anderson. 2012. The perception and misperception of specular surface reflectance. Current Biology 22, 20 (2012), 1909–1913.Google ScholarCross Ref
    54. Wojciech Matusik. 2003. A data-driven reflectance model. Ph.D. Dissertation. Massachusetts Institute of Technology.Google ScholarDigital Library
    55. Wojciech Matusik, Boris Ajdin, Jinwei Gu, Jason Lawrence, Hendrik P. A. Lensch, Fabio Pellacini, and Szymon Rusinkiewicz. 2009. Printing Spatially-Varying Reflectance. ACM Trans. Graph. 28, 5 (2009), 1–9.Google ScholarDigital Library
    56. Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-Driven Reflectance Model. ACM Trans. Graph. 22, 3 (2003), 759–769.Google ScholarDigital Library
    57. Peter McCullagh. 1980. Regression models for ordinal data. Journal of the Royal Statistical Society: Series B (Methodological) 42, 2 (1980), 109–127.Google ScholarCross Ref
    58. Isamu Motoyoshi and Hiroaki Matoba. 2012. Variability in constancy of the perceived surface reflectance across different illumination statistics. Vision Research 53, 1 (2012), 30–39.Google ScholarCross Ref
    59. Isamu Motoyoshi, Shin’ya Nishida, Lavanya Sharan, and Edward H Adelson. 2007. Image statistics and the perception of surface qualities. Nature 447, 7141 (2007), 206–209.Google Scholar
    60. Addy Ngan, Frédo Durand, and Wojciech Matusik. 2006. Image-driven Navigation of Analytical BRDF Models. Rendering Techniques (2006), 399–407.Google Scholar
    61. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A retargetable forward and inverse renderer. ACM Trans. Graph. 38, 6 (2019), 1–17.Google ScholarDigital Library
    62. Shin’ya Nishida and Mikio Shinya. 1998. Use of image-based information in judgments of surface-reflectance properties. J. Opt. Soc. Am. A 15, 12 (1998), 2951–2965.Google ScholarCross Ref
    63. Jum C Nunnally. 1994. Psychometric theory 3E. Tata McGraw-hill education.Google Scholar
    64. Maria Olkkonen and David H Brainard. 2010. Perceived glossiness and lightness under real-world illumination. Journal of Vision 10, 9 (2010), 5–5.Google ScholarCross Ref
    65. Fabio Pellacini, James A Ferwerda, and Donald P Greenberg. 2000. Toward a psychophysically-based light reflection model for image synthesis. In Proc. ACM SIGGRAPH. 55–64.Google ScholarDigital Library
    66. Thiago Pereira and Szymon Rusinkiewicz. 2012. Gamut mapping spatially varying reflectance with an improved BRDF similarity metric. In Computer Graphics Forum, Vol. 31. 1557–1566.Google ScholarDigital Library
    67. Michal Piovarči, Michael Wessely, Michał Jagielski, Marc Alexa, Wojciech Matusik, and Piotr Didyk. 2017. Directional screens. In Proceedings of the 1st Annual ACM Symposium on Computational Fabrication. ACM, 1.Google ScholarDigital Library
    68. Michal Piovarči, Michael Foshey, Vahid Babaei, Szymon Rusinkiewicz, Wojciech Matusik, and Piotr Didyk. 2020. Towards Spatially Varying Gloss Reproduction for 3D Printing. ACM Trans. Graph. 39, 6, Article 206 (2020).Google ScholarDigital Library
    69. Sylvia C Pont and Susan F te Pas. 2006. Material — Illumination Ambiguities and the Perception of Solid Objects. Perception 35, 10 (2006), 1331–1350.Google ScholarCross Ref
    70. Ganesh Ramanarayanan, James Ferwerda, Bruce Walter, and Kavita Bala. 2007. Visual equivalence: towards a new standard for image fidelity. ACM Trans. Graph. 26, 3 (2007), 76–es.Google ScholarDigital Library
    71. Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. 2002. Photographic tone reproduction for digital images. In Proc. ACM SIGGRAPH. 267–276.Google ScholarDigital Library
    72. Olivier Rouiller, Bernd Bickel, Jan Kautz, Wojciech Matusik, and Marc Alexa. 2013. 3D-printing spatially varying BRDFs. IEEE Computer Graphics and Applications 33, 6 (2013), 48–57.Google ScholarDigital Library
    73. Sepideh Samadzadegan, Teun Baar, Philipp Urban, Maria V Ortiz Segovia, and Jana Blahová. 2015. Controlling colour-printed gloss by varnish-halftones. In Measuring, Modeling, and Reproducing Material Appearance 2015, Vol. 9398. International Society for Optics and Photonics, 93980V.Google Scholar
    74. A. C. Schmid, P. Barla, and K. Doerschner. 2020. Material category determined by specular reflection structure mediates the processing of image features for perceived gloss. bioRxiv (2020).Google Scholar
    75. Gabriel Schwartz and Ko Nishino. 2019. Recognizing material properties from images. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 8 (2019), 1981–1995.Google ScholarDigital Library
    76. Ana Serrano, Diego Gutierrez, Karol Myszkowski, Hans-Peter Seidel, and Belen Masia. 2016. An intuitive control space for material appearance. ACM Trans. Graph. 35, 6 (2016).Google ScholarDigital Library
    77. Lavanya Sharan, Yuanzhen Li, Isamu Motoyoshi, Shin’ya Nishida, and Edward H Adelson. 2008. Image statistics for surface reflectance perception. J. Opt. Soc. Am. A 25, 4 (2008), 846–865.Google ScholarCross Ref
    78. Lavanya Sharan, Ruth Rosenholtz, and Edward H Adelson. 2014. Accuracy and speed of material categorization in real-world images. Journal of Vision 14, 9 (2014), 12–12.Google ScholarCross Ref
    79. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014).Google Scholar
    80. Katherine R Storrs and Roland W Fleming. 2020. Unsupervised Learning Predicts Human Perception and Misperception of Specular Surface Reflectance. bioRxiv (2020).Google Scholar
    81. Tiancheng Sun, Henrik Wann Jensen, and Ravi Ramamoorthi. 2018. Connecting measured BRDFs to analytic brdfs by data-driven diffuse-specular separation. ACM Trans. Graph. 37, 6 (2018), 1–15.Google ScholarDigital Library
    82. Tiancheng Sun, Ana Serrano, Diego Gutierrez, and Belen Masia. 2017. Attribute-preserving gamut mapping of measured BRDFs. In Computer Graphics Forum, Vol. 36. 47–54.Google ScholarDigital Library
    83. Alejandro Sztrajman, Jaroslav Křivánek, Alexander Wilkie, and Tim Weyrich. 2019. Image-based Remapping of Spatially-Varying Material Appearance. Journal of Computer Graphics Techniques (JCGT) 8, 4 (2019), 1–30.Google Scholar
    84. James T. Todd, J. Farley Norman, and Ennio Mingolla. 2004. Lightness Constancy in the Presence of Specular Highlights. Psychological Science 15, 1 (2004), 33–39.Google ScholarCross Ref
    85. Matteo Toscani, Dar’ya Guarnera, Giuseppe Claudio Guarnera, Jon Yngve Hardeberg, and Karl R Gegenfurtner. 2020. Three perceptual dimensions for specular and diffuse reflection. ACM Transactions on Applied Perception (TAP) 17, 2 (2020), 1–26.Google ScholarDigital Library
    86. Matteo Toscani and Matteo Valsecchi. 2019. Lightness Discrimination Depends More on Bright Rather Than Shaded Regions of Three-Dimensional Objects. i-Perception 10, 6 (2019), 1–10.Google Scholar
    87. Matteo Toscani, Matteo Valsecchi, and Karl R Gegenfurtner. 2017. Lightness perception for matte and glossy complex shapes. Vision Research 131 (2017), 82–95.Google ScholarCross Ref
    88. TS Trowbridge and Karl P Reitz. 1975. Average irregularity representation of a rough surface for ray reflection. J. Opt. Soc. Am. A 65, 5 (1975), 531–536.Google ScholarCross Ref
    89. Peter Vangorp, Pascal Barla, and Roland W Fleming. 2017. The perception of hazy gloss. Journal of Vision 17, 5 (2017), 19–19.Google ScholarCross Ref
    90. Peter Vangorp and Philip Dutré. 2008. Shape-dependent gloss correction. In Proc. Applied Perception in Graphics and Visualization. 123–130.Google ScholarDigital Library
    91. Peter Vangorp, Jurgen Laurijssen, and Philip Dutré. 2007. The influence of shape on the perception of material reflectance. In Proc. ACM SIGGRAPH. 77:1–77:9.Google Scholar
    92. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces. Rendering Techniques (2007), 195–206.Google ScholarDigital Library
    93. Gregory J. Ward. 1992. Measuring and Modeling Anisotropic Reflection. Proc. ACM SIGGRAPH 26, 2 (1992), 265–272.Google ScholarDigital Library
    94. Peter Welinder, Steve Branson, Pietro Perona, and Serge Belongie. 2010. The multidimensional wisdom of crowds. Advances in Neural Information Processing Systems 23 (2010), 2424–2432.Google ScholarDigital Library
    95. G. Wendt, F. Faul, V. Ekroll, and R. Mausfeld. 2010. Disparity, motion, and color information improve gloss constancy performance. Journal of Vision 10, 9 (2010).Google ScholarCross Ref
    96. Tim Weyrich, Pieter Peers, Wojciech Matusik, and Szymon Rusinkiewicz. 2009. Fabricating microgeometry for custom surface reflectance. ACM Trans. Graph. 28, 3 (2009), 32.Google ScholarDigital Library
    97. Christiane B Wiebel, Matteo Toscani, and Karl R Gegenfurtner. 2015. Statistical correlates of perceived gloss in natural images. Vision Research 115 (2015), 175–187.Google ScholarCross Ref
    98. Josh Wills, Sameer Agarwal, David Kriegman, and Serge Belongie. 2009. Toward a perceptual space for gloss. ACM Trans. Graph. 28, 4 (2009), 1–15.Google ScholarDigital Library
    99. Haoyu Xu, Zhenqi Han, Songlin Feng, Han Zhou, and Yuchun Fang. 2018. Foreign object debris material recognition based on convolutional neural networks. Eurasip Journal on Image and Video Processing 2018, 1 (2018), 1–10.Google ScholarCross Ref
    100. Fan Zhang, Huib de Ridder, Pascal Barla, and Sylvia Pont. 2020a. Effects of light map orientation and shape on the visual perception of canonical materials. Journal of Vision 20, 4, Article 13 (2020), 18 pages.Google ScholarCross Ref
    101. Fan Zhang, Huib de Ridder, Pascal Barla, and Sylvia Pont. 2020b. A systematic approach to testing and predicting light-material interactions. Journal of Vision 19, 4, Article 11 (2020), 22 pages.Google Scholar
    102. Fan Zhang, Huib de Ridder, and Sylvia Pont. 2015. The influence of lighting on visual perception of material qualities. In Human Vision and Electronic Imaging, Vol. SPIE 9394. 239–248.Google Scholar
    103. Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian Material Synthesis. ACM Trans. Graph. 37, 4, Article 76 (2018).Google ScholarDigital Library

ACM Digital Library Publication: