“Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images” by Gao, Li, Dong, Peers, Xu, et al. …
Conference:
Type:
Title:
- Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images
Session/Category Title: Acquiring, Perceiving and Rendering Material Appearance
Presenter(s)/Author(s):
Abstract:
In this paper we present a unified deep inverse rendering framework for estimating the spatially-varying appearance properties of a planar exemplar from an arbitrary number of input photographs, ranging from just a single photograph to many photographs. The precision of the estimated appearance scales from plausible when the input photographs fails to capture all the reflectance information, to accurate for large input sets. A key distinguishing feature of our framework is that it directly optimizes for the appearance parameters in a latent embedded space of spatially-varying appearance, such that no handcrafted heuristics are needed to regularize the optimization. This latent embedding is learned through a fully convolutional auto-encoder that has been designed to regularize the optimization. Our framework not only supports an arbitrary number of input photographs, but also at high resolution. We demonstrate and evaluate our deep inverse rendering solution on a wide variety of publicly available datasets.
References:
1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/Google Scholar
2. Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance Modeling by Neural Texture Synthesis. ACM Trans. Graph. 35, 4, Article 65 (July 2016). Google ScholarDigital Library
3. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF Capture for Stationary Materials. ACM Trans. Graph. 34, 4, Article 110 (July 2015). Google ScholarDigital Library
4. Dan A. Calian, Jean-François Lalonde, Paulo Gotardo, Tomas Simon, Iain Matthews, and Kenny Mitchell. 2018. From Faces to Outdoor Light Probes. Computer Graphics Forum 37, 2 (2018), 51–61.Google ScholarCross Ref
5. Inchang Choi, Daniel S. Jeon, Giljoo Nam, Diego Gutierrez, and Min H. Kim. 2017. High-Quality Hyperspectral Reconstruction Using a Spectral Prior. ACM Trans. Graph. 36, 6, Article 218 (2017). Google ScholarDigital Library
6. Valentin Deschaintre, Miika Aittala, Frédo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-Image SVBRDF Capture with a Rendering-Aware Deep Network. ACM Trans. Graph. 37, 128 (aug 2018). Google ScholarDigital Library
7. Yue Dong, Guojun Chen, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Appearance-from-motion: Recovering Spatially Varying Surface Reflectance Under Unknown Lighting. ACM Trans. Graph. 33, 6, Article 193 (2014). Google ScholarDigital Library
8. Julie Dorsey, Holly Rushmeier, and Franois Sillion. 2008. Digital Modeling of Material Appearance. Morgan Kaufmann Publishers Inc. Google ScholarDigital Library
9. Leon Gatys, Alexander S Ecker, and Matthias Bethge. 2015. Texture synthesis using convolutional neural networks. In NIPS. 262–270. Google ScholarDigital Library
10. Geoffrey Hinton and Ruslan Salakhutdinov. 2006. Reducing the Dimensionality of Data with Neural Networks. Science 313, 5786 (2006), 504 — 507.Google Scholar
11. Zhuo Hui, Kalyan Sunkavalli, Joon-Young Lee, Sunil Hadap, and Aswin Sankaranarayanan. 2017. Reflectance Capture using Univariate Sampling of BRDFs. In ICCV.Google Scholar
12. Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML. 448–456. Google ScholarDigital Library
13. Kaizhang Kang, Zimin Chen, Jiaping Wang, Kun Zhou, and Hongzhi Wu. 2018. Efficient Reflectance Capture Using an Autoencoder. ACM Trans. Graph. 37, 4, Article 127 (July 2018). Google ScholarDigital Library
14. Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, and Jan Kautz. 2017. A Lightweight Approach for On-the-Fly Reflectance Estimation. In ICCV.Google Scholar
15. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.Google Scholar
16. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling Surface Appearance from a Single Photograph Using Self-augmented Convolutional Neural Networks. ACM Trans. Graph. 36, 4, Article 45 (July 2017), 11 pages. Google ScholarDigital Library
17. Zhengqin Li, Kalyan Sunkavalli, and Manmohan Krishna Chandraker. 2018a. Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image. ECCV.Google Scholar
18. Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018b. Learning to Reconstruct Shape and Spatially-varying Reflectance from a Single Image. ACM Trans. Graph. 37, 6 (2018), 126. Google ScholarDigital Library
19. Gianpaolo Palma, Marco Callieri, Matteo Dellepiane, and Roberto Scopigno. 2012. A Statistical Method for SVBRDF Approximation from Video Sequences in General Lighting Conditions. Comput. Graph. Forum 31, 4 (2012), 1491–1500. Google ScholarDigital Library
20. Jérémy Riviere, Pieter Peers, and Abhijeet Ghosh. 2016. Mobile Surface Reflectometry. Comput. Graph. Forum 35, 1 (2016), 191–202. Google ScholarDigital Library
21. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing High-Dimensional Data Using t-SNE. (2008), 2579–2605.Google Scholar
22. Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces. In Rendering Techniques. 195–206. Google ScholarDigital Library
23. Michael Weinmann and Richard Klein. 2015. Advances in Geometry and Reflectance Acquisition. In ACM SIGGRAPH Asia, Course Notes. Google ScholarDigital Library
24. Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. 2016. Recovering Shape and Spatially-Varying Surface Reflectance under Unknown Illumination. ACM Trans. Graph. 35, 6 (December 2016). Google ScholarDigital Library
25. Zexiang Xu, Jannik Boll Nielsen, Jiyang Yu, Henrik Wann Jensen, and Ravi Ramamoorthi. 2016. Minimal BRDF Sampling for Two-shot Near-field Reflectance Acquisition. ACM Trans. Graph. 35, 6, Article 188 (Nov. 2016). Google ScholarDigital Library
26. Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. 2018. Deep image-based relighting from optimal sparse samples. ACM Trans. Graph. 37, 4 (2018), 126. Google ScholarDigital Library
27. Wenjie Ye, Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2018. Single Photograph Surface Appearance Modeling with Self-Augmented CNNs and Inexact Supervision. Comput. Graph. Forum 37, 7 (Oct 2018).Google Scholar
28. Zhiming Zhou, Guojun Chen, Yue Dong, David Wipf, Yong Yu, John Snyder, and Xin Tong. 2016. Sparse-as-Possible SVBRDF Acquisition. ACM Trans. Graph. 35 (November 2016). Google ScholarDigital Library
