“A similarity measure for material appearance” by Lagunas, Malpica, Serrano, Garces, Gutierrez, et al. …
Conference:
Type:
Title:
- A similarity measure for material appearance
Session/Category Title: Acquiring, Perceiving and Rendering Material Appearance
Presenter(s)/Author(s):
Abstract:
We present a model to measure the similarity in appearance between different materials, which correlates with human similarity judgments. We first create a database of 9,000 rendered images depicting objects with varying materials, shape and illumination. We then gather data on perceived similarity from crowdsourced experiments; our analysis of over 114,840 answers suggests that indeed a shared perception of appearance similarity exists. We feed this data to a deep learning architecture with a novel loss function, which learns a feature space for materials that correlates with such perceived appearance similarity. Our evaluation shows that our model outperforms existing metrics. Last, we demonstrate several applications enabled by our metric, including appearance-based search for material suggestions, database visualization, clustering and summarization, and gamut mapping.
References:
1. Edward H Adelson. 2001. On seeing stuff: the perception of materials by humans and machines. In Human vision and electronic imaging VI, Vol. 4299. International Society for Optics and Photonics, 1–13.Google Scholar
2. Sameer Agarwal, Josh Wills, Lawrence Cayton, Gert Lanckriet, David Kriegman, and Serge Belongie. 2007. Generalized non-metric multidimensional scaling. In Conference on artifical intelligence and statistics. 11–18.Google Scholar
3. Barton L Anderson. 2011. Visual perception of materials and surfaces. Current biology 21, 24 (2011), R978–R983.Google Scholar
4. Amit Banerjee and Rajesh N Dave. 2004. Validating clusters using the Hopkins statistic. In 2004 IEEE International conference on fuzzy systems (IEEE Cat. No. 04CH37542), Vol. 1. IEEE, 149–153.Google ScholarCross Ref
5. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2013. Opensurfaces: A richly annotated catalog of surface appearance. ACM Transactions on graphics (TOG) 32, 4 (2013), 111. Google ScholarDigital Library
6. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material recognition in the wild with the materials in context database. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 3479–3487.Google ScholarCross Ref
7. A. C. Chadwick and R. W. Kentridge. 2015. The perception of gloss: A review. Vision research 109 (2015), 221 — 235.Google Scholar
8. Kang Chen, Kun Xu, Yizhou Yu, Tian-Yi Wang, and Shi-Min Hu. 2015. Magic decorator: automatic material suggestion for indoor digital scenes. ACM Transactions on graphics (TOG) 34, 6 (2015), 232. Google ScholarDigital Library
9. De Cheng, Yihong Gong, Sanping Zhou, Jinjun Wang, and Nanning Zheng. 2016. Person re-identification by multi-channel parts-based cnn with improved triplet loss function. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 1335–1344.Google ScholarCross Ref
10. D. W. Cunningham, C. Wallraven, R. W. Fleming, and W. Strasser. 2007. Perceptual Reparameterization of Material Properties. In Proceedings of the third Eurographics conference on computational aesthetics in graphics, visualization and imaging (Computational Aesthetics’07). 89–96. Google ScholarDigital Library
11. Kristin J. Dana, Bram van Ginneken, Shree K. Nayar, and Jan J. Koenderink. 1999. Reflectance and Texture of Real-world Surfaces. ACM Transactions on graphics (TOG) 18, 1 (Jan. 1999), 1–34. Google ScholarDigital Library
12. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). Ieee, 248–255.Google ScholarCross Ref
13. Katja Doerschner, Roland W. Fleming, Ozgur Yilmaz, Paul R. Schrater, Bruce Hartung, and Daniel Kersten. 2011. Visual Motion and the Perception of Surface Material. Current biology 21, 23 (2011), 2010 — 2016.Google Scholar
14. Julie Dorsey, Holly Rushmeier, and François Sillion. 2010. Digital modeling of material appearance. Elsevier. Google ScholarDigital Library
15. Jonathan Dupuy and Wenzel Jakob. 2018. An Adaptive Parameterization for Efficient Material Acquisition and Rendering. ACM Transactions on graphics (TOG). Google ScholarDigital Library
16. Jirí Filip and Radomír Vávra. 2014. Template-based sampling of anisotropic BRDFs. In Computer graphics forum, Vol. 33. Wiley Online Library, 91–99. Google ScholarDigital Library
17. Roland W. Fleming. 2014. Visual perception of materials and their properties. Vision research 94 (2014), 62 — 75.Google Scholar
18. Roland W Fleming. 2017. Material perception. Annual review of vision science 3 (2017), 365–388.Google Scholar
19. Roland W Fleming, Ron O Dror, and Edward H Adelson. 2003. Real-world illumination and the perception of surface reflectance properties. Journal of vision 3, 5 (2003), 3–3.Google ScholarCross Ref
20. Roland W. Fleming, Shin’ya Nishida, and Karl R. Gegenfurtner. 2015. Perception of material properties. Vision research 115 (2015), 157 — 162.Google Scholar
21. Adria Fores, James Ferwerda, and Jinwei Gu. 2012. Toward a perceptually based metric for BRDF modeling. In Color and imaging conference, Vol. 2012. Society for Imaging Science and Technology, 142–148.Google Scholar
22. Jiyang Gao and Ram Nevatia. 2018. Revisiting Temporal Modeling for Video-based Person ReID. arXiv preprint arXiv:1805.02104 (2018).Google Scholar
23. Elena Garces, Aseem Agarwala, Diego Gutierrez, and Aaron Hertzmann. 2014. A Similarity Measure for Illustration Style. ACM transactions on graphics (TOG, proc. SIGGRAPH) 33, 4 (2014). Google ScholarDigital Library
24. Stamatios Georgoulis, Vincent Vanweddingen, Marc Proesmans, and Luc Van Gool. 2017. Material Classification under Natural Illumination Using Reflectance Maps. In IEEE Winter conference on applications of computer vision (WACV). IEEE, 244–253.Google ScholarCross Ref
25. Ioannis Gkioulekas, Bruce Walter, Edward H Adelson, Kavita Bala, and Todd Zickler. 2015. On the appearance of translucent edges. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 5528–5536.Google ScholarCross Ref
26. Ioannis Gkioulekas, Bei Xiao, Shuang Zhao, Edward H Adelson, Todd Zickler, and Kavita Bala. 2013. Understanding the role of phase function in translucent appearance. ACM Transactions on graphics (TOG) 32, 5 (2013), 147. Google ScholarDigital Library
27. Vlastimil Havran, Jiri Filip, and Karol Myszkowski. 2016. Perceptually Motivated BRDF Comparison using Single Image. Computer graphics forum (2016). Google ScholarDigital Library
28. Eric Hayman, Barbara Caputo, Mario Fritz, and Jan-Olof Eklundh. 2004. On the significance of real-world conditions for material classification. In European conference on computer vision (ECCV). Springer, 253–266.Google ScholarCross Ref
29. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 770–778.Google ScholarCross Ref
30. Yun-Xian Ho, Michael S Landy, and Laurence T Maloney. 2006. How direction of illumination affects visually perceived surface roughness. Journal of vision 6, 5 (2006), 8–8.Google ScholarCross Ref
31. Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely Connected Convolutional Networks.. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Vol. 1. 3.Google ScholarCross Ref
32. Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.Google Scholar
33. Adrian Jarabo, Hongzhi Wu, Julie Dorsey, Holly Rushmeier, and Diego Gutierrez. 2014. Effects of Approximate Filtering on the Appearance of Bidirectional Texture Functions. IEEE Transactions on visualization and computer graphics 20, 6 (2014). Google ScholarDigital Library
34. Maurice Kendall and Jean D. Gibbons. 1990. Rank Correlation Methods (5 ed.). A Charles Griffin Title.Google Scholar
35. M G Kendall and B Babington-Smith. 1940. On the Method of Paired Comparisons. Biometrica 31 (1940), 324–345.Google ScholarCross Ref
36. Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, and Jan Kautz. 2017. A lightweight approach for on-the-fly reflectance estimation. In Proceedings of the international conference on computer vision (ICCV). 20–28.Google ScholarCross Ref
37. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
38. Jaroslav Křivánek, James A. Ferwerda, and Kavita Bala. 2010. Effects of global illumination approximations on material appearance. ACM transactions on graphics (TOG, proc. SIGGRAPH) 29, 4, Article 112 (July 2010), 112:1–112:10 pages. Google ScholarDigital Library
39. Manuel Lagunas, Elena Garces, and Diego Gutierrez. 2018. Learning icons appearance similarity. Multimedia tools and applications (2018), 1–19. Google ScholarDigital Library
40. Thomas Leung and Jitendra Malik. 2001. Representing and recognizing the visual appearance of materials using three-dimensional textons. International journal of computer vision 43, 1 (2001), 29–44. Google ScholarDigital Library
41. Stephen Lombardi and Ko Nishino. 2012. Reflectance and natural illumination from a single image. In European conference on computer vision (ECCV). Springer, 582–595. Google ScholarDigital Library
42. Feng Lu, Xiaowu Chen, Imari Sato, and Yoichi Sato. 2018. SymPS: BRDF symmetry guided photometric stereo for shape and light source estimation. IEEE transactions on pattern analysis and machine intelligence 40, 1 (2018), 221–234.Google Scholar
43. Zhaoliang Lun, Evangelos Kalogerakis, and Alla Sheffer. 2015. Elements of style: learning perceptual shape style similarity. ACM Transactions on graphics (TOG) 34, 4 (2015), 84. Google ScholarDigital Library
44. Laurence T Maloney and David H Brainard. 2010. Color and material perception: Achievements and challenges. Journal of vision 10, 9 (2010), 19–19.Google ScholarCross Ref
45. Rafał Mantiuk, Scott Daly, and Louis Kerofsky. 2008. Display adaptive tone mapping. In ACM Transactions on graphics (TOG), Vol. 27. ACM, 68. Google ScholarDigital Library
46. Wojciech Matusik, Boris Ajdin, Jinwei Gu, Jason Lawrence, Hendrik P. A. Lensch, Fabio Pellacini, and Szymon Rusinkiewicz. 2009. Printing Spatially-Varying Reflectance. ACM transactions on graphics (TOG, proc. SIGGRAPH Asia) 28, 5 (Dec. 2009). Google ScholarDigital Library
47. Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-Driven Reflectance Model. ACM Transactions on graphics (TOG) 22, 3 (July 2003), 759–769. Google ScholarDigital Library
48. Brian McFee and Gert Lanckriet. 2011. Learning Multi-modal Similarity. Journal of machine learning research 12 (2011), 491–523. Google ScholarDigital Library
49. Leland McInnes and John Healy. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018).Google Scholar
50. M Mylo, M Giesel, Q Zaidi, M Hullin, and R Klein. 2017. Appearance bending: A perceptual editing paradigm for data-driven material models. Vision, modeling and visualization. The Eurographics association (2017). Google ScholarDigital Library
51. Addy Ngan, Frédo Durand, and Wojciech Matusik. 2005. Experimental Analysis of BRDF Models. In Eurographics Symposium on Rendering (2005). The Eurographics association. Google ScholarDigital Library
52. Addy Ngan, Frédo Durand, and Wojciech Matusik. 2006. Image-driven Navigation of Analytical BRDF Models.. In Rendering Techniques. 399–407. Google ScholarDigital Library
53. Fabio Pellacini, James A. Ferwerda, and Donald P. Greenberg. 2000. Toward a Psychophysically-based Light Reflection Model for Image Synthesis. In Proceedings of the 27th annual conference on computer graphics and interactive techniques (SIGGRAPH ’00). 55–64. Google ScholarDigital Library
54. Thiago Pereira and Szymon Rusinkiewicz. 2012. Gamut mapping spatially varying reflectance with an improved BRDF similarity metric. In Computer graphics forum, Vol. 31. Wiley Online Library, 1557–1566. Google ScholarDigital Library
55. Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. 2019. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237 (2019).Google Scholar
56. Michael Rubinstein, Diego Gutierrez, Olga Sorkine, and Ariel Shamir. 2010. A Comparative Study of Image Retargeting. ACM transactions on graphics (TOG, proc. SIGGRAPH Asia) 29, 6 (2010), 160:1–160:10. Google ScholarDigital Library
57. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 3 (2015), 211–252. Google ScholarDigital Library
58. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 815–823.Google ScholarCross Ref
59. M. Schultz and T. Joachims. 2003. Learning a Distance Metric from Relative Comparisons. In Advances in neural information processing systems. Google ScholarDigital Library
60. Gabriel Schwartz and Ko Nishino. 2016. Material Recognition from Local Appearance in Global Context. arXiv preprint arXiv:1611.09394 (2016).Google Scholar
61. Gabriel Schwartz and Ko Nishino. 2018. Recognizing Material Properties from Images. arXiv preprint arXiv:1801.03127 (2018).Google Scholar
62. Ana Serrano, Diego Gutierrez, Karol Myszkowski, Hans-Peter Seidel, and Belen Masia. 2016. An Intuitive Control Space for Material Appearance. ACM Transactions on graphics (TOG) 35, 6, Article 186 (Nov. 2016), 186:1–186:12 pages. Google ScholarDigital Library
63. Lavanya Sharan, Ruth Rosenholtz, and Edward Adelson. 2009. Material perception: What can you see in a brief glance? Journal of vision 9, 8 (2009), 784–784.Google ScholarCross Ref
64. Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014. CNN features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) workshops. 806–813. Google ScholarDigital Library
65. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
66. Cyril Soler, Kartic Subr, and Derek Nowrouzezahrai. 2018. A Versatile Parameterization for Measured Material Manifolds. In Computer graphics forum, Vol. 37. Wiley Online Library, 135–144.Google Scholar
67. Tiancheng Sun, Henrik Wann Jensen, and Ravi Ramamoorthi. 2018. Connecting Measured BRDFs to Analytic BRDFs by Data-driven Diffuse-specular Separation. In ACM transactions on graphics (TOG, proc. SIGGRAPH Asia). Article 273, 273:1–273:15 pages. Google ScholarDigital Library
68. Tiancheng Sun, Ana Serrano, Diego Gutierrez, and Belen Masia. 2017. Attribute-preserving Gamut Mapping of Measured BRDFs. Computer graphics forum 36, 4 (July 2017). Google ScholarDigital Library
69. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2818–2826.Google ScholarCross Ref
70. Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir, and Adam Tauman Kalai. 2011. Adaptively Learning the Crowd Kernel. In Proceedings of the 28th international conference on machine learning (ICML’11). 673–680. Google ScholarDigital Library
71. William Thompson, Roland Fleming, Sarah Creem-Regehr, and Jeanine Kelly Stefanucci. 2011. Visual Perception from a Computer Graphics Perspective (1st ed.). A. K. Peters, Ltd., Natick, MA, USA.Google ScholarDigital Library
72. Jan Jaap R Van Assen, Pascal Barla, and Roland W Fleming. 2018. Visual features in the perception of liquids. Current biology 28, 3 (2018), 452–458.Google Scholar
73. Laurens Van Der Maaten and Kilian Weinberger. 2012. Stochastic triplet embedding. In Machine learning for signal processing (MLSP), 2012 IEEE International Workshop on. IEEE, 1–6.Google Scholar
74. Peter Vangorp, Pascal Barla, and Roland W Fleming. 2017. The perception of hazy gloss. Journal of vision 17, 5 (2017), 19–19.Google ScholarCross Ref
75. Peter Vangorp, Jurgen Laurijssen, and Philip Dutré. 2007. The Influence of Shape on the Perception of Material Reflectance. ACM Transactions on graphics (TOG) 26, 3, Article 77 (July 2007). Google ScholarDigital Library
76. Raquel Vidaurre, Dan Casas, Elena Garces, and Jorge Lopez-Moreno. 2019. BRDF Estimation of Complex Materials with Nested Learning. In IEEE Winter conference on applications of computer vision (WACV).Google Scholar
77. Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612. Google ScholarDigital Library
78. Peter Welinder, Steve Branson, Pietro Perona, and Serge J Belongie. 2010. The multidimensional wisdom of crowds. In Advances in neural information processing systems. 2424–2432. Google ScholarDigital Library
79. Josh Wills, Sameer Agarwal, David Kriegman, and Serge Belongie. 2009. Toward a Perceptual Space for Gloss. ACM Transactions on graphics (TOG) 28, 4, Article 103 (Sept. 2009), 103:1–103:15 pages. Google ScholarDigital Library
80. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks?. In Advances in neural information processing systems. 3320–3328. Google ScholarDigital Library
81. Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society, 586–595.Google ScholarCross Ref
82. Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian Material Synthesis. ACM Transactions on graphics (TOG) 37, 4, Article 76 (July 2018), 76:1–76:14 pages. Google ScholarDigital Library
