“PhotoShape: photorealistic materials for large-scale shape collections”
Conference:
Type(s):
Title:
- PhotoShape: photorealistic materials for large-scale shape collections
Session/Category Title: Fun in geometry & fabrication
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Existing online 3D shape repositories contain thousands of 3D models but lack photorealistic appearance. We present an approach to automatically assign high-quality, realistic appearance models to large scale 3D shape collections. The key idea is to jointly leverage three types of online data – shape collections, material collections, and photo collections, using the photos as reference to guide assignment of materials to shapes. By generating a large number of synthetic renderings, we train a convolutional neural network to classify materials in real photos, and employ 3D-2D alignment techniques to transfer materials to different parts of each shape model. Our system produces photorealistic, relightable, 3D shapes (PhotoShapes).
References:
1. Adobe Stock. 2018. Adobe Stock: Stock photos, royalty-free images, graphics, vectors and videos. https://stock.adobe.com/. (2018).Google Scholar
2. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2013. Practical SVBRDF Capture in the Frequency Domain. ACM Trans. Graph. 32, 4 (2013). Google ScholarDigital Library
3. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF Capture for Stationary Materials. ACM Trans. Graph. 34, 4 (2015). Google ScholarDigital Library
4. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2013. OpenSurfaces: A Richly Annotated Catalog of Surface Appearance. ACM Trans. on Graphics 32, 4 (2013). Google ScholarDigital Library
5. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material Recognition in the Wild with the Materials in Context Database. In CVPR.Google Scholar
6. Adam Brady, Jason Lawrence, Pieter Peers, and Westley Weimer. 2014. genBRDF: discovering new analytic BRDFs with genetic programming. ACM Transactions on Graphics 33, 4 (2014). Google ScholarDigital Library
7. Brent Burley and Walt Disney Animation Studios. {n. d.}. Physically-based shading at disney.Google Scholar
8. Manmohan Chandraker. 2014. On Shape and Material Recovery from Motion. In ECCV. Springer International Publishing, Cham.Google Scholar
9. Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 {cs.GR}. Stanford University — Princeton University — Toyota Technological Institute at Chicago.Google Scholar
10. Kang Chen, Kun Xu, Yizhou Yu, Tian-Yi Wang, and Shi-Min Hu. 2015. Magic Decorator: Automatic Material Suggestion for Indoor Digital Scenes. ACM Trans. Graph. 34, 6 (2015). Google ScholarDigital Library
11. Qifeng Chen and Vladlen Koltun. 2017. Photographic Image Synthesis with Cascaded Refinement Networks. In ICCV.Google Scholar
12. M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. 2014. Describing Textures in the Wild. In CVPR. Google ScholarDigital Library
13. M. Cimpoi, S. Maji, and A. Vedaldi. 2015. Deep filter banks for texture recognition and segmentation. In CVPR.Google Scholar
14. Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. 1996. Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-based Approach. In SIGGRAPH. Google ScholarDigital Library
15. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR.Google Scholar
16. Olga Diamanti, Connelly Barnes, Sylvain Paris, Eli Shechtman, and Olga Sorkine-Hornung. 2015. Synthesis of Complex Image Appearance from Limited Exemplars. ACM Transactions on Graphics 34, 2 (2015). Google ScholarDigital Library
17. Yue Dong, Guojun Chen, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Appearance-from-motion: Recovering Spatially Varying Surface Reflectance Under Unknown Lighting. ACM Trans. Graph. 33, 6 (2014). Google ScholarDigital Library
18. Alexei A. Efros and William T. Freeman. 2001. Image Quilting for Texture Synthesis and Transfer. In SIGGRAPH. Google ScholarDigital Library
19. Alexei A. Efros and Thomas K. Leung. 1999. Texture Synthesis by Non-Parametric Sampling. In ICCV. Google ScholarDigital Library
20. Pedro F. Felzenszwalb, Ross B. Girshick, David McAllester, and Deva Ramanan. 2010. Object Detection with Discriminatively Trained Part-Based Models. IEEE TPAMI 32, 9 (2010). Google ScholarDigital Library
21. L. A. Gatys, A. S. Ecker, and M. Bethge. 2015. Texture Synthesis Using Convolutional Neural Networks. In NIPS. Google ScholarDigital Library
22. Stamatios Georgoulis, Konstantinos Rematas, Tobias Ritschel, Efstratios Gavves, Mario Fritz, Luc Van Gool, and Tinne Tuytelaars. 2017a. Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning. IEEE TPAMI (2017).Google Scholar
23. Stamatios Georgoulis, Vincent Vanweddingen, Marc Proesmans, and Luc Van Gool. 2017b. Material Classification under Natural Illumination Using Reflectance Maps. In WACV.Google Scholar
24. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.Google Scholar
25. Herman-Miller. 2018. Herman Miller 3D Models. https://www.hermanmiller.com/resources/models/3d-models/. (2018).Google Scholar
26. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In SIGGRAPH. Google ScholarDigital Library
27. Hui Huang, Ke Xie, Lin Ma, Dani Lischinski, Minglun Gong, Xin Tong, and Daniel Cohen-Or. 2018. Appearance Modeling via Proxy-to-Image Alignment. ACM Trans. Graph. 37, 1 (2018). Google ScholarDigital Library
28. Qixing Huang, Hai Wang, and Vladlen Koltun. 2015. Single-view Reconstruction via Joint Analysis of Image and Shape Collections. ACM Trans. Graph. 34, 4 (2015). Google ScholarDigital Library
29. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In CVPR.Google Scholar
30. Hamid Izadinia, Qi Shan, and Steven M Seitz. 2017. IM2CAD. In CVPR.Google Scholar
31. Arjun Jain, Thorsten Thormählen, Tobias Ritschel, and Hans-Peter Seidel. 2012. Material Memex: Automatic Material Suggestions for 3D Objects. ACM Trans. Graph. (Proc. SIGGRAPH Asia 2012) 31, 5 (2012). Google ScholarDigital Library
32. Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. In CVPR.Google Scholar
33. Natasha Kholgade, Tomas Simon, Alexei Efros, and Yaser Sheikh. 2014. 3D Object Manipulation in a Single Photograph using Stock 3D Models. ACM Trans. Graph. 33, 4 (2014). Google ScholarDigital Library
34. Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, and Jan Kautz. 2017. A Lightweight Approach for On-the-Fly Reflectance Estimation. In ICCV.Google Scholar
35. Johannes Kopf, Chi-Wing Fu, Daniel Cohen-Or, Oliver Deussen, Dani Lischinski, and Tien-Tsin Wong. 2007. Solid Texture Synthesis from 2D Exemplars. ACM Trans. Graph. 26, 3 (2007). Google ScholarDigital Library
36. Philipp Krähenbühl and Vladlen Koltun. 2011. Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. In NIPS. Google ScholarDigital Library
37. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling Surface Appearance from a Single Photograph Using Self-augmented Convolutional Neural Networks. ACM Trans. Graph. 36, 4 (2017). Google ScholarDigital Library
38. C. Liu, J. Yuen, and A. Torralba. 2011. SIFT Flow: Dense Correspondence across Scenes and Its Applications. IEEE TPAMI 33, 5 (2011). Google ScholarDigital Library
39. Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. 2017. Material Editing using a Physically Based Rendering Network. In ICCV.Google Scholar
40. Stephen Lombardi and Ko Nishino. 2015. Reflectance and Illumination Recovery in the Wild. IEEE TPAMI (2015). Google ScholarDigital Library
41. Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-Driven Reflectance Model. ACM Trans. Graph. 22, 3 (2003). Google ScholarDigital Library
42. Geoffrey Oxholm and Ko Nishino. 2016. Shape and Reflectance Estimation in the Wild. IEEE TPAMI (2016). Google ScholarDigital Library
43. Poliigon. 2018. A library of textures, materials and HDR’s for artists that want photorealism. https://www.poliigon.com/. (2018).Google Scholar
44. Konstantinos Rematas, Chuong Nguyen, Tobias Ritschel, Mario Fritz, and Tinne Tuytelaars. 2017. Novel Views of Objects from a Single Image. TPAMI (2017).Google Scholar
45. Konstantinos Rematas, Tobias Ritschel, Mario Fritz, Efstratios Gavves, and Tinne Tuytelaars. 2016. Deep Reflectance Maps. In CVPR.Google Scholar
46. Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. 2016. Playing for data: Ground truth from computer games. In ECCV. Springer.Google Scholar
47. Gabriel Schwartz and Ko Nishino. 2015. Automatically discovering local visual material attributes. CVPR (2015).Google Scholar
48. Omry Sendik and Daniel Cohen-Or. 2017. Deep Correlations for Texture Synthesis. ACM Trans. Graph. 36, 4 (2017).Google Scholar
49. Lavanya Sharan, Ce Liu, Ruth Rosenholtz, and Edward H. Adelson. 2013. Recognizing materials using perceptually inspired features. International Journal of Computer Vision 108, 3 (2013).Google Scholar
50. Lavanya Sharan, Ruth Rosenholtz, and Edward H. Adelson. 2014. Accuracy and speed of material categorization in real-world images. Journal of Vision 14, 10 (2014).Google ScholarCross Ref
51. Hao Su, Charles R. Qi, Yangyan Li, and Leonidas J. Guibas. 2015. Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views. In ICCV. Google ScholarDigital Library
52. Bruno Vallet and Bruno LÃl’vy. 2009. What you seam is what you get. Technical Report. INRIA – ALICE Project Team.Google Scholar
53. Vray-materials.de. 2018. Vray-materials.de – Your ultimate V-Ray material resource. https://www.vray-materials.de. (2018).Google Scholar
54. Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet Models for Refraction Through Rough Surfaces. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR’07). Google ScholarDigital Library
55. Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei Efros, and Ravi Ramamoorthi. 2016b. A 4D light-field dataset and CNN architectures for material recognition. In ECCV.Google Scholar
56. Tuanfeng Y. Wang, Hao Su, Qixing Huang, Jingwei Huang, Leonidas Guibas, and Niloy J. Mitra. 2016a. Unsupervised Texture Transfer from Images to Model Collections. ACM Trans. Graph. 35, 6, Article 177 (Nov. 2016). Google ScholarDigital Library
57. Yunhai Wang, Minglun Gong, Tianhua Wang, Daniel Cohen-Or, Hao Zhang, and Baoquan Chen. 2013. Projective Analysis for 3D Shape Segmentation. ACM Trans. Graph. 32, 6 (2013). Google ScholarDigital Library
58. Mitra NJ Wang TY, Ritschel T. 2017. Joint Material and Illumination Estimation from Photo Sets in the Wild. Eurographics (2017).Google Scholar
59. Jia Xue, Hang Zhang, Kristin J. Dana, and Ko Nishino. 2017. Differential Angular Imaging for Material Recognition. CVPR (2017).Google Scholar
60. zbyg. 2018. HDRi Pack. https://zbyg.deviantart.com/art/HDRi-Pack-2-103458406. (2018).Google Scholar
61. Hang Zhang, Kristin Dana, and Ko Nishino. 2015. Reflectance Hashing for Material Recognition. In CVPR.Google Scholar
62. Zhiming Zhou, Guojun Chen, Yue Dong, David Wipf, Yong Yu, John Snyder, and Xin Tong. 2016. Sparse-as-possible SVBRDF Acquisition. ACM Trans. Graph. 35, 6 (2016). Google ScholarDigital Library
