“Deep SVBRDF Estimation from Single Image under Learned Planar Lighting” by Zhang, Gao, Wang, Yu, Cheng, et al. …
Conference:
Type:
Title:
- Deep SVBRDF Estimation from Single Image under Learned Planar Lighting
Session/Category Title: Neural Capturing
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Estimating spatially varying BRDF from a single image without complicated acquisition devices is a challenging problem. In this paper, a deep learning based method was proposed to improve the capture efficiency of single image significantly by learning the lighting pattern of a planar light source, and reconstruct high-quality SVBRDF by learning the global correlation prior of the input image. In our framework, the lighting pattern optimization is embedded in the training process of the network by introducing an online rendering process. The rendering process not only renders images online as the input of network, but also efficiently back propagates gradients from the network to optimize the lighting pattern. Once trained, the network can estimate SVBRDFs from real photographs captured under the learned lighting pattern. Additionally, we describe an onsite capture setup that needs no careful calibration to capture the material sample efficiently. In particular, even a cell phone can be used for illumination. We demonstrate on synthetic and real data that our method could recover a wide range of materials from a single image casually captured under the learned lighting pattern.
References:
1. Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance Modeling by Neural Texture Synthesis. ACM Transactions on Graphics 35, 4 (2016), 65.1–65.13.
2. M. Aittala, T. Weyrich, and J. Lehtinen. 2013. Practical SVBRDF capture in the frequency domain. Acm Transactions on Graphics 32, 4CD (2013), 1–12.
3. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein GAN. arxiv:1701.07875 [stat.ML]
4. D. R. Baum, H. E. Rushmeire, and J. M. Winget. 1989. Improving radiosity solutions through the use of analytically determined form-factors. In Proceedings of the 16st Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1989.
5. Guojun Chen, Yue Dong, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Reflectance scanning: estimating shading frame and BRDF with generalized linear light sources. ACM Transactions on Graphics (TOG) – Proceedings of ACM SIGGRAPH 2014 (2014).
6. Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. 2022. Simple baselines for image restoration. arXiv preprint arXiv:2204.04676 (2022).
7. Robert L. Cook and Kenneth E. Torrance. 1981. A reflectance model for computer graphics. Acm Siggraph Computer Graphics 15, 3 (1981), 307–316.
8. Kristin J Dana, Bram Van Ginneken, Shree K Nayar, and Jan J Koenderink. 1999. Reflectance and texture of real-world surfaces. ACM Transactions On Graphics (TOG) 18, 1 (1999), 1–34.
9. Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-Image SVBRDF Capture with a Rendering-Aware Deep Network. ACM Transactions on Graphics 37, 4CD (2018), 128.1–128.15.
10. Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2019. Flexible SVBRDF Capture with a Multi‐Image Deep Network. Computer Graphics Forum 38, 4 (2019).
11. Valentin Deschaintre, George Drettakis, and Adrien Bousseau. 2020. Guided Fine-Tuning for Large-Scale Material Transfer. Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering) 39, 4 (2020). http://www-sop.inria.fr/reves/Basilic/2020/DDB20
12. Michal Drobot. 2018. Physically based area lights. In GPU Pro 360 Guide to Lighting. AK Peters/CRC Press, 177–210.
13. Duan Gao, Xiao Li, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. 2019. Deep Inverse Rendering for High-Resolution SVBRDF Estimation from an Arbitrary Number of Images. ACM Trans. Graph. 38, 4, Article 134 (July 2019), 15 pages. https://doi.org/10.1145/3306346.3323042
14. A. Gardner, C. Tchou, T. Hawkins, and Paul Debevec University of Southern California Institute for Creative Technologies Graphics Laboratory. 2003. Linear Light Source Reflectometry. ACM Transactions on Graphics (TOG) (2003).
15. A. Ghosh, S. Achutha, W. Heidrich, and M. O’Toole. 2007. BRDF Acquisition with Basis Illumination. In IEEE International Conference on Computer Vision.
16. A. Ghosh, T. Chen, P. Peers, C. A. Wilson, and P. Debevec. 2010. Estimating Specular Roughness and Anisotropy from Second Order Spherical Gradient Illumination. Computer Graphics Forum 28, 4 (2010), 1161–1170.
17. J. Guo, S. Lai, C. Tao, Y. Cai, and L. Q. Yan. 2021. Highlight-aware two-stream network for single-image SVBRDF acquisition. ACM Transactions on Graphics 40, 4 (2021), 1–14.
18. Yu Guo, Cameron Smith, Miloš Hašan, Kalyan Sunkavalli, and Shuang Zhao. 2020. MaterialGAN: reflectance capture using a generative SVBRDF model. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–13.
19. E. Heitz, J. Dupuy, S. Hill, and D. Neubelt. 2016. Real-time polygonal-light shading with linearly transformed cosines. ACM Transactions on Graphics (TOG) 35, 4 (2016), 41.1–41.8.
20. P. Henzler, V. Deschaintre, N. J. Mitra, and T. Ritschel. 2021. Generative Modelling of BRDF Textures from Flash Images. (2021).
21. Michael Holroyd, Jason Lawrence, and Todd Zickler. 2010. A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance. ACM Transactions on Graphics (TOG) 29, 4 (2010), 1–12.
22. J., Riviere, P., Peers, A., and Ghosh. 2015. Mobile Surface Reflectometry. Computer Graphics Forum (2015).
23. Wenzel Jakob. 2014. Mitsuba renderer 0.5. (2014). http://www.mitsuba-renderer.org/index_old.html
24. Kaizhang Kang, Zimin Chen, Jiaping Wang, Kun Zhou, and Hongzhi Wu. 2018. Efficient reflectance capture using an autoencoder. ACM Transactions on Graphics 37, 4CD (2018), 1–10.
25. Kaizhang Kang, Minyi Gu, Cihui Xie, Xuanda Yang, Hongzhi Wu, and Kun Zhou. 2021. Neural Reflectance Capture in the View-Illumination Domain. IEEE Transactions on Visualization and Computer Graphics (2021).
26. K. Kang, C. Xie, C. He, M. Yi, and H. Wu. 2019. Learning efficient illumination multiplexing for joint capture of reflectance and shape. ACM Transactions on Graphics 38, 6 (2019), 1–12.
27. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
28. Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular Primitives for High-Performance Differentiable Rendering. ACM Transactions on Graphics 39, 6 (2020).
29. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. Acm Transactions on Graphics 36, 4 (2017), 45.
30. Xiao Li, Peiran Ren, Yue Dong, Gang Hua, Xin Tong, and Baining Guo. 2019. Capturing Piecewise SVBRDFs with Content Aware Lighting. In Advances in Computer Graphics: 36th Computer Graphics International Conference, CGI 2019, Calgary, AB, Canada, June 17–20, 2019, Proceedings 36. Springer, 371–379.
31. Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker. 2018. Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image. (2018).
32. X. Ma, K. Kang, R. Zhu, H. Wu, and K. Zhou. 2021. Free-form scanning of non-planar appearance with neural trace photography. ACM Transactions on Graphics (TOG) (2021).
33. Rosalie Martin, Arthur Roullier, Romain Rouffet, Adrien Kaiser, and Tamy Boubekeur. 2022. MaterIA: Single Image High-Resolution Material Capture in the Wild. Computer Graphics Forum 41, 2 (2022), 163–17715 pages. https://doi.org/10.1111/cgf.14466
34. Giljoo Nam, Joo Ho Lee, Hongzhi Wu, Diego Gutierrez, and Min H. Kim. 2016. Simultaneous acquisition of microscale reflectance and normals. Acm Transactions on Graphics 35, 6 (2016), 185.
35. Emilie Nogue, Yiming Lin, and Abhijeet Ghosh. 2022. Polarization-imaging Surface Reflectometry using Near-field Display. In Eurographics Symposium on Rendering. The Eurographics Association, Vol. 2.
36. P. Ren, J. Wang, J. Snyder, T. Xin, and B. Guo. 2011. Pocket Reflectometry. ACM Transactions on Graphics 30, 4 (2011), 45.
37. Jérémy Riviere, Ilya Reshetouski, Luka Filipi, and Abhijeet Ghosh. 2017. Polarization imaging reflectometry in the wild. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1–14.
38. B. Tunwattanapong, G. Fyffe, P. Graham, J. Busch, X. Yu, A. Ghosh, and P. Debevec. 2013. Acquiring reflectance and shape from continuous spherical harmonic illumination. Acm Transactions on Graphics (2013).
39. Giuseppe Vecchio, Simone Palazzo, and Concetto Spampinato. 2021. SurfaceNet: Adversarial SVBRDF Estimation from a Single Image. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12840–12848.
40. C. P. Wang, N. Snavely, and S. Marschner. 2011. Estimating dual-scale properties of glossy surfaces from step-edge lighting. Acm Transactions on Graphics 30, 6CD (2011), 1–12.
41. Tao Wen, Beibei Wang, Lei Zhang, Jie Guo, and Nicolas Holzschuch. 2022. SVBRDF Recovery from a Single Image with Highlights Using a Pre-trained Generative Adversarial Network. In Computer Graphics Forum. Wiley Online Library.
42. Lance Williams. 1983. Pyramidal parametrics. In Proceedings of the 10th annual conference on Computer graphics and interactive techniques. 1–11.
43. Wenjie Ye, Yue Dong, Pieter Peers, and Baining Guo. 2021. Deep Reflectance Scanning: Recovering Spatially-varying Material Appearance from a Flash-lit Video Sequence. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 409–427.
44. Wenjie Ye, Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2018. Single Image Surface Appearance Modeling with Self-augmented CNNs and Inexact Supervision. Computer Graphics Forum 37, 7 (2018), 201–211.
45. Y. Zhao, B. Wang, Y. Xu, Z. Zeng, and N. Holzschuch. 2020. Joint SVBRDF Recovery and Synthesis From a Single Image using an Unsupervised Generative Adversarial Network. In EGSR 2020.
46. X. Zhou and N. K. Kalantari. 2021. Adversarial Single-Image SVBRDF Estimation with Hybrid Training. Computer Graphics Forum: Journal of the European Association for Computer Graphics2 (2021), 40.
47. Xilong Zhou and Nima Khademi Kalantari. 2022. Look-Ahead Training with Learned Reflectance Loss for Single-Image SVBRDF Estimation. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1–12.
