“Photo-to-shape material transfer for diverse structures” by Hu, Su, Chen, Kaick and Huang

We introduce a method for assigning photorealistic relightable materials to 3D shapes in an automatic manner. Our method takes as input a photo exemplar of a real object and a 3D object with segmentation, and uses the exemplar to guide the assignment of materials to the parts of the shape, so that the appearance of the resulting shape is as similar as possible to the exemplar. To accomplish this goal, our method combines an image translation neural network with a material assignment neural network. The image translation network translates the color from the exemplar to a projection of the 3D shape and the part segmentation from the projection to the exemplar. Then, the material prediction network assigns materials from a collection of realistic materials to the projected parts, based on the translated images and perceptual similarity of the materials.One key idea of our method is to use the translation network to establish a correspondence between the exemplar and shape projection, which allows us to transfer materials between objects with diverse structures. Another key idea of our method is to use the two pairs of (color, segmentation) images provided by the image translation to guide the material assignment, which enables us to ensure the consistency in the assignment. We demonstrate that our method allows us to assign materials to shapes so that their appearances better resemble the input exemplars, improving the quality of the results over the state-of-the-art method, and allowing us to automatically create thousands of shapes with high-quality photorealistic materials. Code and data for this paper are available at https://github.com/XiangyuSu611/TMT.

1. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2013. OpenSurfaces: A Richly Annotated Catalog of Surface Appearance. ACM Trans. on Graphics (Proc. SIGGRAPH) 32, 4 (2013), 111:1–17.Google ScholarDigital Library
2. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material Recognition in the Wild With the Materials in Context Database. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.Google ScholarCross Ref
3. Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV). 801–818.Google ScholarDigital Library
4. Qifeng Chen and Vladlen Koltun. 2017. Photographic Image Synthesis With Cascaded Refinement Networks. In Proc. Int. Conf. on Computer Vision.Google ScholarCross Ref
5. Zhiqin Chen and Hao Zhang. 2019. Learning Implicit Fields for Generative Shape Modeling. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 5939–5948.Google ScholarCross Ref
6. Julian Chibane, Aymen Mir, and Gerard Pons-Moll. 2020. Neural Unsigned Distance Fields for Implicit Function Learning. In Advances in Neural Information Processing Systems (NeurIPS).Google Scholar
7. Johanna Delanoy, Mathieu Aubry, Phillip Isola, Alexei A. Efros, and Adrien Bousseau. 2018. 3D Sketching Using Multi-View Deep Volumetric Prediction. In Proc. ACM Comput. Graph. Interact. Tech. 21:1–15.Google ScholarDigital Library
8. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.Google ScholarCross Ref
9. D. Guarnera, G. C. Guarnera, A. Ghosh, C. Denk, and M. Glencross. 2016. BRDF Representation and Acquisition. In Proc. Eurographics State of the Art Reports. Eurographics Association, 625–650.Google Scholar
10. Xian-Feng Han, Hamid Laga, and Mohammed Bennamoun. 2021. Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era. IEEE Trans. Pattern Analysis & Machine Intelligence 43, 5 (2021), 1578–1604.Google ScholarCross Ref
11. Vlastimil Havran, Jiri Filip, and Karol Myszkowski. 2016. Perceptually motivated BRDF comparison using single image. In Computer graphics forum, Vol. 35. Wiley Online Library, 1–12.Google Scholar
12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.Google ScholarCross Ref
13. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017).Google Scholar
14. Elad Hoffer and Nir Ailon. 2015. Deep metric learning using triplet network. In International workshop on similarity-based pattern recognition. Springer, 84–92.Google ScholarCross Ref
15. Tao Hu, Geng Lin, Zhizhong Han, and Matthias Zwicker. 2021. Learning to Generate Dense Point Clouds With Textures on Multiple Categories. In Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2170–2179.Google ScholarCross Ref
16. Yiwei Hu, Julie Dorsey, and Holly Rushmeier. 2019. A Novel Framework for Inverse Procedural Texture Modeling. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 38, 6 (2019), 186:1–14.Google Scholar
17. Hui Huang, Ke Xie, Lin Ma, Dani Lischinski, Minglun Gong, Xin Tong, and Daniel Cohen-Or. 2018b. Appearance Modeling via Proxy-to-Image Alignment. ACM Trans. on Graphics 37, 1 (2018), 10:1–15.Google ScholarDigital Library
18. Qixing Huang, Hai Wang, and Vladlen Koltun. 2015. Single-view reconstruction via joint analysis of image and shape collections. ACM Transactions on Graphics (TOG) 34, 4 (2015), 1–10.Google ScholarDigital Library
19. Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018a. Multimodal Unsupervised Image-to-image Translation. In Proc. Euro. Conf. on Computer Vision.Google ScholarDigital Library
20. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-To-Image Translation With Conditional Adversarial Networks. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.Google Scholar
21. Arjun Jain, Thorsten Thormählen, Tobias Ritschel, and Hans-Peter Seidel. 2012. Material memex: Automatic material suggestions for 3d objects. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1–8.Google ScholarDigital Library
22. Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang. 2017. SurfaceNet: An End-To-End 3D Neural Network for Multiview Stereopsis. In Proc. Int. Conf. on Computer Vision. 2307–2315.Google ScholarCross Ref
23. Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 2017. 3D Shape Segmentation with Projective Convolutional Networks. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.Google ScholarCross Ref
24. Manuel Lagunas, Sandra Malpica, Ana Serrano, Elena Garces, Diego Gutierrez, and Belen Masia. 2019a. A Similarity Measure for Material Appearance. ACM Trans. on Graphics (Proc. SIGGRAPH) 38, 4 (2019), 135:1–12.Google ScholarDigital Library
25. Manuel Lagunas, Sandra Malpica, Ana Serrano, Elena Garces, Diego Gutierrez, and Belen Masia. 2019b. A similarity measure for material appearance. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–12.Google ScholarDigital Library
26. Juncheng Liu, Zhouhui Lian, and Jianguo Xiao. 2017. Auto-Colorization of 3D Models from Images. In SIGGRAPH Asia 2017 Technical Briefs. 15:1–4.Google Scholar
27. Aymen Mir, Thiemo Alldieck, and Gerard Pons-Moll. 2020. Learning to Transfer Texture from Clothing Images to 3D Humans. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. IEEE.Google ScholarCross Ref
28. Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. 2019. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 909–918.Google ScholarCross Ref
29. Chuong H Nguyen, Tobias Ritschel, Karol Myszkowski, Elmar Eisemann, and Hans-Peter Seidel. 2012. 3D material style transfer. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 431–438.Google Scholar
30. Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. 2019. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4531–4540.Google ScholarCross Ref
31. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 165–174.Google ScholarCross Ref
32. Keunhong Park, Konstantinos Rematas, Ali Farhadi, and Steven M Seitz. 2018. PhotoShape: photorealistic materials for large-scale shape collections. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 37, 6 (2018), 1–12.Google ScholarDigital Library
33. Quang-Hieu Pham, Thanh Nguyen, Binh-Son Hua, Gemma Roig, and Sai-Kit Yeung. 2019. JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.Google ScholarCross Ref
34. Amit Raj, Cusuh Ham, Connelly Barnes, Vladimir Kim, Jingwan Lu, and James Hays. 2019. Learning to Generate Textures on 3D Meshes. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.Google Scholar
35. Konstantinos Rematas, Chuong H Nguyen, Tobias Ritschel, Mario Fritz, and Tinne Tuytelaars. 2016. Novel views of objects from a single image. IEEE transactions on pattern analysis and machine intelligence 39, 8 (2016), 1576–1590.Google Scholar
36. Thorsten-Walther Schmidt, Fabio Pellacini, Derek Nowrouzezahrai, Wojciech Jarosz, and Carsten Dachsbacher. 2016. State of the Art in Artistic Editing of Appearance, Lighting and Material. Computer Graphics Forum 35, 1 (2016), 216–233.Google ScholarDigital Library
37. Ana Serrano, Diego Gutierrez, Karol Myszkowski, Hans-Peter Seidel, and Belen Masia. 2016. An Intuitive Control Space for Material Appearance. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 35, 6 (2016), 186:1–12.Google Scholar
38. Hao Su, Qixing Huang, Niloy J Mitra, Yangyan Li, and Leonidas Guibas. 2014. Estimating image depth using shape collections. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–11.Google ScholarDigital Library
39. Tiancheng Sun, Ana Serrano, Diego Gutierrez, and Belen Masia. 2017. Attribute-preserving gamut mapping of measured BRDFs. In Computer graphics forum, Vol. 36. Wiley Online Library, 47–54.Google Scholar
40. Tuanfeng Y Wang, Hao Su, Qixing Huang, Jingwei Huang, Leonidas J Guibas, and Niloy J Mitra. 2016. Unsupervised texture transfer from images to model collections. ACM Trans. on Graphics (Proc. SIGGRAPH Asia) 35, 6 (2016), 177–1.Google Scholar
41. Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, William T Freeman, and Joshua B Tenenbaum. 2017. MarrNet: 3D Shape Reconstruction via 2.5D Sketches. In Proc. Conf. on Neural Information Processing Systems.Google Scholar
42. Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. 2019. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. arXiv preprint arXiv:1905.10711 (2019).Google Scholar
43. Bo Zhang, Mingming He, Jing Liao, Pedro V Sander, Lu Yuan, Amine Bermak, and Dong Chen. 2019. Deep exemplar-based video colorization. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 8052–8061.Google ScholarCross Ref
44. Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. 2020. Cross-Domain Correspondence Learning for Exemplar-Based Image Translation. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. IEEE.Google ScholarCross Ref
45. Jie Zhu, Yanwen Guo, and Han Ma. 2018. A Data-Driven Approach for Furniture and Indoor Scene Colorization. IEEE Trans. Visualization & Computer Graphics 24, 9 (2018), 2473–2486.Google ScholarCross Ref