“DeepBasis: Hand-Held Single-Image SVBRDF Capture via Two-Level Basis Material Model” by Wang, Zhang, Gao and Zhang
Conference:
Type(s):
Title:
- DeepBasis: Hand-Held Single-Image SVBRDF Capture via Two-Level Basis Material Model
Session/Category Title: Visualizing the Future
Presenter(s)/Author(s):
Abstract:
Recovering spatial-varying bi-directional reflectance distribution function (SVBRDF) from a single hand-held captured image has been a meaningful but challenging task in computer graphics. Benefiting from the learned data priors, some previous methods can utilize the potential material correlations between image pixels to serve for SVBRDF estimation. To further reduce the ambiguity from single-image estimation, it is necessary to integrate additional explicit material correlations. Given the flexible expressive ability of basis material assumption, we propose DeepBasis, a deep-learning-based method integrated with this assumption. It jointly predicts basis materials and their blending weights. Then the estimated SVBRDF is their linear combination. To facilitate the extraction of data priors, we introduce a two-level basis model to keep the sufficient representative while using a fixed number of basis materials. Moreover, considering the absence of ground-truth basis materials and weights during network training, we propose a variance-consistency loss and adopt a joint prediction strategy, thereby enabling the existing SVBRDF dataset available for training. Additionally, due to the hand-held capture setting, the exact lighting directions are unknown. We model the lighting direction estimation as a sampling problem and propose an optimization-based algorithm to find the optimal estimation. Quantitative evaluation and qualitative analysis demonstrate that DeepBasis can produce a higher quality SVBRDF estimation than previous methods. All source codes will be publicly released.
References:
[1]
Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance Modeling by Neural Texture Synthesis. ACM Trans. Graph. 35, 4, Article 65 (jul 2016), 13 pages. https://doi.org/10.1145/2897824.2925917
[2]
Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-Shot SVBRDF Capture for Stationary Materials. ACM Trans. Graph. 34, 4, Article 110 (jul 2015), 13 pages. https://doi.org/10.1145/2766967
[3]
Neil Alldrin, Todd Zickler, and David Kriegman. 2008. Photometric stereo with non-parametric and spatially-varying reflectance. In 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1–8.
[4]
Guojun Chen, Yue Dong, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Reflectance Scanning: Estimating Shading Frame and BRDF with Generalized Linear Light Sources. ACM Trans. Graph. 33, 4, Article 117 (jul 2014), 11 pages. https://doi.org/10.1145/2601097.2601180
[5]
Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. 2022. Simple baselines for image restoration. arXiv preprint arXiv:2204.04676 (2022).
[6]
Xiaojie Chu, Liangyu Chen, and Wenqing Yu. 2022. NAFSSR: Stereo Image Super-Resolution Using NAFNet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 1239–1248.
[7]
Robert L. Cook and Kenneth E. Torrance. 1981. A Reflectance Model for Computer Graphics(SIGGRAPH ’81). Association for Computing Machinery, New York, NY, USA, 307–316. https://doi.org/10.1145/800224.806819
[8]
Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-Image SVBRDF Capture with a Rendering-Aware Deep Network. ACM Trans. Graph. 37, 4, Article 128 (jul 2018), 15 pages. https://doi.org/10.1145/3197517.3201378
[9]
Valentin Deschaintre, Miika Aittala, Frédo Durand, George Drettakis, and Adrien Bousseau. 2019. Flexible svbrdf capture with a multi-image deep network. In Computer graphics forum, Vol. 38. Wiley Online Library, 1–13.
[10]
Valentin Deschaintre, George Drettakis, and Adrien Bousseau. 2020. Guided fine-tuning for large-scale material transfer. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 91–105.
[11]
Yue Dong. 2019. Deep appearance modeling: A survey. Visual Informatics 3, 2 (2019), 59–68.
[12]
Yue Dong, Jiaping Wang, Xin Tong, John Snyder, Yanxiang Lan, Moshe Ben-Ezra, and Baining Guo. 2010. Manifold Bootstrapping for SVBRDF Capture. ACM Trans. Graph. 29, 4, Article 98 (jul 2010), 10 pages. https://doi.org/10.1145/1778765.1778835
[13]
Jiahui Fan, Beibei Wang, Milos Hasan, Jian Yang, and Ling-Qi Yan. 2023. Neural Biplane Representation for BTF Rendering and Acquisition. In ACM SIGGRAPH 2023 Conference Proceedings. 1–11.
[14]
Duan Gao, Xiao Li, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. 2019. Deep Inverse Rendering for High-Resolution SVBRDF Estimation from an Arbitrary Number of Images. 38, 4, Article 134 (jul 2019), 15 pages. https://doi.org/10.1145/3306346.3323042
[15]
Dan B Goldman, Brian Curless, Aaron Hertzmann, and Steven M. Seitz. 2010. Shape and Spatially-Varying BRDFs from Photometric Stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 6 (2010), 1060–1071. https://doi.org/10.1109/TPAMI.2009.102
[16]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT press.
[17]
Darya Guarnera, Giuseppe Claudio Guarnera, Abhijeet Ghosh, Cornelia Denk, and Mashhuda Glencross. 2016. BRDF representation and acquisition. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 625–650.
[18]
Jie Guo, Shuichang Lai, Chengzhi Tao, Yuelong Cai, Lei Wang, Yanwen Guo, and Ling-Qi Yan. 2021. Highlight-Aware Two-Stream Network for Single-Image SVBRDF Acquisition. ACM Trans. Graph. 40, 4, Article 123 (jul 2021), 14 pages. https://doi.org/10.1145/3450626.3459854
[19]
Yu Guo, Cameron Smith, Miloš Hašan, Kalyan Sunkavalli, and Shuang Zhao. 2020. MaterialGAN: Reflectance Capture Using a Generative SVBRDF Model. 39, 6, Article 254 (nov 2020), 13 pages. https://doi.org/10.1145/3414685.3417779
[20]
Philipp Henzler, Valentin Deschaintre, Niloy J. Mitra, and Tobias Ritschel. 2021. Generative Modelling of BRDF Textures from Flash Images. 40, 6, Article 284 (dec 2021), 13 pages. https://doi.org/10.1145/3478513.3480507
[21]
Yong Hwi Kim and Kwan H Lee. 2022. Data Driven SVBRDF Estimation Using Deep Embedded Clustering. Electronics 11, 19 (2022), 3239.
[22]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[23]
Jason Lawrence, Aner Ben-Artzi, Christopher DeCoro, Wojciech Matusik, Hanspeter Pfister, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2006. Inverse shade trees for non-parametric material representation and editing. ACM Transactions on Graphics (TOG) 25, 3 (2006), 735–745.
[24]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature 521, 7553 (2015), 436–444.
[25]
Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. 2003. Image-Based Reconstruction of Spatial Appearance and Geometric Detail. 22, 2 (apr 2003), 234–257. https://doi.org/10.1145/636886.636891
[26]
Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling Surface Appearance from a Single Photograph Using Self-Augmented Convolutional Neural Networks. ACM Trans. Graph. 36, 4, Article 45 (jul 2017), 11 pages. https://doi.org/10.1145/3072959.3073641
[27]
Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker. 2018a. Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image. In Computer Vision – ECCV 2018, Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer International Publishing, Cham, 74–90.
[28]
Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018b. Learning to Reconstruct Shape and Spatially-Varying Reflectance from a Single Image. 37, 6, Article 269 (dec 2018), 11 pages. https://doi.org/10.1145/3272127.3275055
[29]
Ilya Loshchilov and Frank Hutter. 2016. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016).
[30]
Rosalie Martin, Arthur Roullier, Romain Rouffet, Adrien Kaiser, and Tamy Boubekeur. 2022. MaterIA: Single Image High-Resolution Material Capture in the Wild. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 163–177.
[31]
Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003a. A Data-Driven Reflectance Model. ACM Trans. Graph. 22, 3 (jul 2003), 759–769. https://doi.org/10.1145/882262.882343
[32]
Wojciech Matusik, Hanspeter Pfister, Matthew Brand, and Leonard McMillan. 2003b. Efficient Isotropic BRDF Measurement(EGRW ’03). Eurographics Association, Goslar, DEU, 241–247.
[33]
Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. 2018. Practical svbrdf acquisition of 3d objects with unstructured flash photography. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1–12.
[34]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
[35]
Peiran Ren, Jiaping Wang, John Snyder, Xin Tong, and Baining Guo. 2011. Pocket Reflectometry. 30, 4, Article 45 (jul 2011), 10 pages. https://doi.org/10.1145/2010324.1964940
[36]
Roland Ruiters, Christopher Schwartz, and Reinhard Klein. 2013. Example-based Interpolation and Synthesis of Bidirectional Texture Functions. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 361–370.
[37]
Giuseppe Vecchio, Simone Palazzo, and Concetto Spampinato. 2021. SurfaceNet: Adversarial SVBRDF Estimation from a Single Image. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12840–12848.
[38]
Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces(EGSR’07). Eurographics Association, Goslar, DEU, 195–206.
[39]
Chun-Po Wang, Noah Snavely, and Steve Marschner. 2011. Estimating Dual-Scale Properties of Glossy Surfaces from Step-Edge Lighting(SA ’11). Association for Computing Machinery, New York, NY, USA, Article 172, 12 pages. https://doi.org/10.1145/2024156.2024206
[40]
Tao Wen, Beibei Wang, Lei Zhang, Jie Guo, and Nicolas Holzschuch. 2022. SVBRDF Recovery from a Single Image with Highlights Using a Pre-trained Generative Adversarial Network. In Computer Graphics Forum. Wiley Online Library.
[41]
Wenjie Ye, Yue Dong, Pieter Peers, and Baining Guo. 2021. Deep Reflectance Scanning: Recovering Spatially-varying Material Appearance from a Flash-lit Video Sequence. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 409–427.
[42]
Wenjie Ye, Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2018. Single image surface appearance modeling with self-augmented cnns and inexact supervision. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 201–211.
[43]
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[44]
Yezi Zhao, Beibei Wang, Yanning Xu, Zheng Zeng, Lu Wang, and Nicolas Holzschuch. 2020. Joint SVBRDF Recovery and Synthesis From a Single Image using an Unsupervised Generative Adversarial Network. In EGSR (DL). 53–66.
[45]
Xilong Zhou, Milos Hasan, Valentin Deschaintre, Paul Guerrero, Kalyan Sunkavalli, and Nima Khademi Kalantari. 2022. Tilegen: Tileable, controllable material generation and capture. In SIGGRAPH Asia 2022 Conference Papers. 1–9.
[46]
Xilong Zhou and Nima Khademi Kalantari. 2021. Adversarial Single-Image SVBRDF Estimation with Hybrid Training. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 315–325.
[47]
Xilong Zhou and Nima Khademi Kalantari. 2022. Look-Ahead Training with Learned Reflectance Loss for Single-Image SVBRDF Estimation. 41, 6, Article 266 (nov 2022), 12 pages. https://doi.org/10.1145/3550454.3555495
[48]
Zhiming Zhou, Guojun Chen, Yue Dong, David Wipf, Yong Yu, John Snyder, and Xin Tong. 2016. Sparse-as-Possible SVBRDF Acquisition. 35, 6, Article 189 (dec 2016), 12 pages. https://doi.org/10.1145/2980179.2980247
