“Generative modelling of BRDF textures from flash images” by Henzler, Deschaintre, Mitra and Ritschel
Conference:
Type(s):
Title:
- Generative modelling of BRDF textures from flash images
Session/Category Title: Light Interactions and Differentiable Rendering
Presenter(s)/Author(s):
Abstract:
We learn a latent space for easy capture, consistent interpolation, and efficient reproduction of visual material appearance. When users provide a photo of a stationary natural material captured under flashlight illumination, first it is converted into a latent material code. Then, in the second step, conditioned on the material code, our method produces an infinite and diverse spatial field of BRDF model parameters (diffuse albedo, normals, roughness, specular albedo) that subsequently allows rendering in complex scenes and illuminations, matching the appearance of the input photograph. Technically, we jointly embed all flash images into a latent space using a convolutional encoder, and -conditioned on these latent codes- convert random spatial fields into fields of BRDF parameters using a convolutional neural network (CNN). We condition these BRDF parameters to match the visual characteristics (statistics and spectra of visual features) of the input under matching light. A user study compares our approach favorably to previous work, even those with access to BRDF supervision. Project webpage: https://henzler.github.io/publication/neuralmaterial/.
References:
1. Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance modeling by neural texture synthesis. ACM Trans Graph (Proc. SIGGRAPH) 35, 4 (2016).
2. Brett Allen, Brian Curless, and Zoran Popović. 2003. The space of human body shapes: reconstruction and parameterization from range scans. ACM Trans Graph (Proc. SIGGRAPH) 22, 3 (2003), 587–94.
3. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. Patch-Match: A randomized correspondence algorithm for structural image editing. ACM Trans Graph (Proc. SIGGRAPH) 28, 3 (2009).
4. Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. 2017. Learning texture manifolds with the periodic spatial GAN. In J MLR. 469–77.
5. Volker Blanz, Thomas Vetter, et al. 1999. A morphable model for the synthesis of 3D faces.. In Siggraph, Vol. 99. 187–194.
6. Mark Boss, Varun Jampani, Kihwan Kim, Hendrik P.A. Lensch, and Jan Kautz. 2020. Two-shot Spatially-varying BRDF and Shape Estimation. In CVPR.
7. Robert L Cook and Tony DeRose. 2005. Wavelet noise. ACM Trans Graph 24, 3 (2005), 803–11.
8. Robert L Cook and Kenneth E Torrance. 1982. A reflectance model for computer graphics. ACM Trans Graph 1, 1 (1982), 7–24.
9. Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-image svbrdf capture with a rendering-aware deep network. ACM Trans Graph (Proc. SIGGRAPH) 37, 4 (2018).
10. Valentin Deschaintre, Miika Aittala, Frédo Durand, George Drettakis, and Adrien Bousseau. 2019. Flexible SVBRDF Capture with a Multi-Image Deep Network. Comp Graph Forum 38, 4 (2019).
11. Valentin Deschaintre, George Drettakis, and Adrien Bousseau. 2020. Guided Fine-Tuning for Large-Scale Material Transfer. Comp Graph Forum (Proc. EGSR) 39, 4 (2020).
12. Valentin Deschaintre, Yiming Lin, and Abhijeet Ghosh. 2021. Deep polarization imaging for 3D shape and SVBRDF acquisition. In CVPR.
13. Alexei A Efros and Thomas K Leung. 1999. Texture synthesis by non-parametric sampling. In ICCV, Vol. 2.
14. Duan Gao, Xiao Li, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. 2019. Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images. ACM Trans Graph (Proc. SIGGRAPH Asia) 38, 4 (2019).
15. Leon Gatys, Alexander S Ecker, and Matthias Bethge. 2015. Texture synthesis using convolutional neural networks. In NIPS.
16. Stamatios Georgoulis, Konstantinos Rematas, Tobias Ritschel, Efstratios Gavves, Mario Fritz, Luc Van Gool, and Tinne Tuytelaars. 2017. Reflectance and natural illumination from single-material specular objects using deep learning. PAMI 40, 8 (2017), 1932–47.
17. Darya Guarnera, Giuseppe Claudio Guarnera, Abhijeet Ghosh, Cornelia Denk, and Mashhuda Glencross. 2016. BRDF representation and acquisition. Comp Graph Forum 35, 2 (2016), 625–50.
18. Jie Guo, Shuichang Lai, Chengzhi Tao, Yuelong Cai, Lei Wang, Yanwen Guo, and Ling-Qi Yan. 2021. Highlight-aware two-stream network for single-image SVBRDF acquisition. ACM Trans Graph (Proc. SIGGRAPH) 40, 4 (2021).
19. Yu Guo, Miloš Hašan, Lingqi Yan, and Shuang Zhao. 2020a. A Bayesian Inference Framework for Procedural Material Parameter Estimation. Comp Graph Forum 39, 7 (2020).
20. Yu Guo, Cameron Smith, Miloš Hašan, Kalyan Sunkavalli, and Shuang Zhao. 2020b. MaterialGAN: Reflectance Capture using a Generative SVBRDF Model. ACM Trans Graph 39, 6 (2020).
21. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770–8.
22. David J Heeger and James R Bergen. 1995. Pyramid-based texture analysis/synthesis. In Proc. SIGGRAPH. 229–38.
23. Eric Heitz. 2014. Understanding the Masking-Shadowing Function in Microfacet-Based BRDFs. Journal of Computer Graphics Techniques (JCGT) 3, 2 (30 June 2014), 48–107. http://jcgt.org/published/0003/02/03/
24. Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. 2020. Learning a Neural 3D Texture Space from 2D Exemplars. CVPR (2020).
25. Yiwei Hu, Julie Dorsey, and Holly Rushmeier. 2019. A Novel Framework for Inverse Procedural Texture Modeling. ACM Trans. Graph. 38, 6 (2019).
26. Xun Huang and Serge Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV. 1501–10.
27. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In ECCV.
28. Bela Julesz. 1965. Texture and visual perception. Scientific American 212, 2 (1965).
29. Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In CVPR. 4401–10.
30. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv:1312.6114 (2013).
31. Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra. 2005. Texture optimization for example-based synthesis. ACM Trans Graph 24, 3 (2005).
32. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM Trans Graph 36, 4 (2017).
33. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2019. Synthesizing 3D Shapes From Silhouette Image Collections Using Multi-Projection Generative Adversarial Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
34. Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker. 2018a. Materials for masses: SVBRDF acquisition with a single mobile phone image. In ECCV. 72–87.
35. Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018b. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans Graph (Proc. SIGGRAPH Asia) 37, 4 (2018).
36. Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. 2017. Material editing using a physically based rendering network. In ICCV. 2261–9.
37. Gang Liu, Yann Gousseau, and Gui-Song Xia. 2016. Texture synthesis through convolutional neural networks and spectrum constraints. In ICPR. 3234–9.
38. Benoit B Mandelbrot. 1983. The fractal geometry of nature. Vol. 173. WH Freeman New York.
39. W Matusik. 2003. A data-driven reflectance model. ACM Trans Graph 22, 3 (2003), 759–69.
40. Wojciech Matusik, Matthias Zwicker, and Frédo Durand. 2005. Texture design using a simplicial complex of morphable textures. ACM Trans Graph (Proc. SIGGRAPH) 24, 3 (2005).
41. Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H. Kim. 2018. Practical SVBRDF Acquisition of 3D Objects with Unstructured Flash Photography. ACM Trans. Graph. 37, 6 (2018).
42. Chuong H Nguyen, Tobias Ritschel, and Hans-Peter Seidel. 2015. Data-driven color manifolds. ACM Trans Graph 34, 2 (2015).
43. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A retargetable forward and inverse renderer. ACM Trans Graph (Proc. SIGGRAPH Asia) 38, 6 (2019).
44. Keunhong Park, Konstantinos Rematas, Ali Farhadi, and Steven M Seitz. 2019. Photo-shape: photorealistic materials for large-scale shape collections. ACM Trans Graph (Proc. SIGGRAPH Asia) 37, 6 (2019).
45. Ken Perlin. 1985. An Image Synthesizer. SIGGRAPH Comput Graph 19, 3 (1985).
46. Bui Tuong Phong. 1975. Illumination for computer generated pictures. Commun ACM 18, 6 (1975), 311–17.
47. Javier Portilla and Eero P Simoncelli. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. Int J Comp Vis 40, 1 (2000), 49–70.
48. K. Rematas, T. Ritschel, M. Fritz, E. Gavves, and T. Tuytelaars. 2016. Deep Reflectance Maps. In CVPR.
49. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In MICCAI. 234–41.
50. Christophe Schlick. 1994. An inexpensive BRDF model for physically-based rendering. Comp Graph Forum 13, 3 (1994), 233–46.
51. Christopher Schwartz, Ralf Sarlette, Michael Weinmann, and Reinhard Klein. 2013. DOME II: A Parallelized BTF Acquisition System. In MAM. 25–31.
52. Omry Sendik and Daniel Cohen-Or. 2017. Deep correlations for texture synthesis. ACM Trans Graph 36, 5 (2017).
53. Ana Serrano, Diego Gutierrez, Karol Myszkowski, Hans-Peter Seidel, and Belen Masia. 2016. An intuitive control space for material appearance. ACM Trans Graph (Proc. SIGGRAPH Asia) 35, 6 (2016).
54. Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. 2019. Singan: Learning a generative model from a single natural image. In ICCV.
55. Liang Shi, Beichen Li, Miloš Hašan, Kalyan Sunkavalli, Tamy Boubekeur, Radomir Mech, and Wojciech Matusik. 2020. Match: Differentiable Material Graphs for Procedural Material Capture. ACM Trans. Graph. 39, 6 (2020).
56. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014).
57. Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor S Lempitsky. 2016. Texture Networks: Feed-forward Synthesis of Textures and Stylized Images.. In ICML.
58. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2017. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In CVPR.
59. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet models for refraction through rough surfaces. In Proc. EGSR. 195–206.
60. Li-Yi Wei and Marc Levoy. 2000. Fast texture synthesis using tree-structured vector quantization. In Proc. SIGGRAPH.
61. Wenjie Ye, Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2018. Single Image Surface Appearance Modeling with Self-augmented CNNs and Inexact Supervision. Comp Graph Forum 37, 7 (2018), 201–11.
62. Yezi Zhao, Beibei Wang, Yanning Xu, Zheng Zeng, Lu Wang, and Nicolas Holzschuch. 2020. Joint SVBRDF Recovery and Synthesis From a Single Image using an Unsupervised Generative Adversarial Network. In EGSR.
63. Xilong Zhou and Nima Khademi Kalantari. 2021. Adversarial Single-Image SVBRDF Estimation with Hybrid Training. Computer Graphics Forum (Proc. Eurographics) (2021).
64. Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2018. Non-stationary texture synthesis by adversarial expansion. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–13.
65. Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian material synthesis. ACM Trans. Graph. 37, 4 (2018).
