“Neural Biplane Representation for BTF Rendering and Acquisition” by Fan, Wang, Hasan, Yang and Yan
Conference:
Type(s):
Title:
- Neural Biplane Representation for BTF Rendering and Acquisition
Session/Category Title: Material Acquisition
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Bidirectional Texture Functions (BTFs) are able to represent complex materials with greater generality than traditional analytical models. This holds true for both measured real materials and synthetic ones. Recent advancements in neural BTF representations have significantly reduced storage costs, making them more practical for use in rendering. These representations typically combine spatial feature (latent) textures with neural decoders that handle angular dimensions per spatial location. However, these models have yet to combine fast compression and inference, accuracy, and generality. In this paper, we propose a biplane representation for BTFs, which uses a feature texture in the half-vector domain as well as the spatial domain. This allows the learned representation to encode high-frequency details in both the spatial and angular domains. Our decoder is small yet general, meaning it is trained once and fixed. Additionally, we optionally combine this representation with a neural offset module for parallax and masking effects. Our model can represent a broad range of BTFs and has fast compression and inference due to its lightweight architecture. Furthermore, it enables a simple way to capture BTF data. By taking about 20 cell phone photos with a collocated camera and flash, our model can plausibly recover the entire BTF, despite never observing function values with differing view and light directions. We demonstrate the effectiveness of our model in the acquisition of many measured materials, including challenging materials such as fabrics.
References:
1. Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance Modeling by Neural Texture Synthesis. ACM Trans. Graph. 35, 4 (2016), 1–13.
2. Miika Aittala, Tim Weyrich, Jaakko Lehtinen, 2015. Two-shot SVBRDF capture for stationary materials.ACM Trans. Graph. 34, 4 (2015), 110–1.
3. Mark Boss, Varun Jampani, Kihwan Kim, Hendrik Lensch, and Jan Kautz. 2020. Two-shot spatially-varying brdf and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3982–3991.
4. Dennis den Brok, Michael Weinmann, and Reinhard Klein. 2017. Towards Sparse and Multiplexed Acquisition of Material BTFs. In Workshop on Material Appearance Modeling, Reinhard Klein and Holly Rushmeier (Eds.). The Eurographics Association. https://doi.org/10.2312/mam.20171326
5. Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, 2022. Efficient geometry-aware 3D generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16123–16133.
6. Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, and Kui Jia. 2022. TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decomposition. In Advances in Neural Information Processing Systems (NeurIPS).
7. Kristin J. Dana, Bram van Ginneken, Shree K. Nayar, and Jan J. Koenderink. 1999. Reflectance and Texture of Real-World Surfaces. ACM Trans. Graph. 18, 1 (jan 1999), 1–34. https://doi.org/10.1145/300776.300778
8. Dennis Den Brok, Michael Weinmann, and Reinhard Klein. 2015. Linear Models for Material BTFs and Possible Applications.. In Material Appearance Modeling.
9. Hong Deng, Yang Liu, Beibei Wang, Jian Yang, Lei Ma, Nicolas Holzschuch, and Ling-Qi Yan. 2022. Constant-Cost Spatio-Angular Prefiltering of Glinty Appearance Using Tensor Decomposition. ACM Transactions on Graphics 41, 2 (2022), 22:1–22:17.
10. Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-Image SVBRDF Capture with a Rendering-aware Deep Network. ACM Trans. Graph. 37, 4 (2018), 1–15.
11. Valentin Deschaintre, Miika Aittala, Frédo Durand, George Drettakis, and Adrien Bousseau. 2019. Flexible svbrdf capture with a multi-image deep network. In Computer graphics forum, Vol. 38. Wiley Online Library, 1–13.
12. Jiahui Fan, Beibei Wang, Milos Hasan, Jian Yang, and Ling-Qi Yan. 2022. Neural Layered BRDFs. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH ’22). Association for Computing Machinery, New York, NY, USA, Article 4, 8 pages. https://doi.org/10.1145/3528233.3530732
13. Duan Gao, Xiao Li, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. 2019. Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images. ACM Transactions on Graphics (TOG) 38, 4 (2019), 134.
14. D. Guarnera, G.C. Guarnera, A. Ghosh, C. Denk, and M. Glencross. 2016. BRDF Representation and Acquisition. Computer Graphics Forum 35, 2 (2016), 625–650. https://doi.org/10.1111/cgf.12867 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.12867
15. Jie Guo, Shuichang Lai, Chengzhi Tao, Yuelong Cai, Lei Wang, Yanwen Guo, and Ling-Qi Yan. 2021. Highlight-aware Two-stream Network for Single-image SVBRDF Acquisition. ACM Trans. Graph. 40, 4 (2021), 1–14.
16. Yu Guo, Miloš Hašan, Lingqi Yan, and Shuang Zhao. 2020a. A bayesian inference framework for procedural material parameter estimation. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 255–266.
17. Y. Guo, C. Smith, Miloš Hašan, Kalyan Sunkavalli, and Shuang Zhao. 2020b. MaterialGAN: Reflectance Capture using a Generative SVBRDF Model. ArXiv abs/2010.00114 (2020).
18. Michal Haindl, Jiří Filip, and Vavra Radomir. 2012. Digital Material Appearance: the Curse of Tera-Bytes. ERCIM News (01 2012), 49–50.
19. Jefferson Y. Han and Ken Perlin. 2003. Measuring Bidirectional Texture Reflectance with a Kaleidoscope. ACM Trans. Graph. 22, 3 (jul 2003), 741–748. https://doi.org/10.1145/882262.882341
20. Philipp Henzler, Valentin Deschaintre, Niloy J Mitra, and Tobias Ritschel. 2021. Generative Modelling of BRDF Textures from Flash Images. ACM Trans. Graph. 40, 6 (2021), 1–13.
21. Bingyang Hu, Jie Guo, Yanjun Chen, Mengtian Li, and Yanwen Guo. 2020. DeepBRDF: A Deep Representation for Manipulating Measured BRDF. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 157–166.
22. Ivo Ihrke, Ilya Reshetouski, Alkhazur Manakov, Art Tevs, Michael Wand, and Hans-Peter Seidel. 2012. A kaleidoscopic approach to surround geometry and reflectance acquisition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 29–36. https://doi.org/10.1109/CVPRW.2012.6239347
23. Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.
24. Alexandr Kuznetsov, Krishna Mullia, Zexiang Xu, Miloš Hašan, and Ravi Ramamoorthi. 2021. NeuMIP: Multi-Resolution Neural Materials. Transactions on Graphics (Proceedings of SIGGRAPH) 40, 4, Article 175 (July 2021), 13 pages.
25. Alexandr Kuznetsov, Xuezheng Wang, Krishna Mullia, Fujun Luan, Zexiang Xu, Milos Hasan, and Ravi Ramamoorthi. 2022. Rendering Neural Materials on Curved Surfaces. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH ’22). Association for Computing Machinery, New York, NY, USA, Article 9, 9 pages. https://doi.org/10.1145/3528233.3530721
26. Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1–11.
27. G. Müller, J. Meseth, M. Sattler, R. Sarlette, and R. Klein. 2005. Acquisition, Synthesis, and Rendering of Bidirectional Texture Functions. Computer Graphics Forum 24, 1 (2005), 83–109. https://doi.org/10.1111/j.1467-8659.2005.00830.x arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8659.2005.00830.x
28. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
29. Gilles Rainer, Wenzel Jakob, Abhijeet Ghosh, and Tim Weyrich. 2019. Neural BTF Compression and Interpolation. Computer Graphics Forum (Proceedings of Eurographics) 38, 2 (March 2019).
30. Gilles Rainer, Abhijeet Ghosh, Wenzel Jakob, and Tim Weyrich. 2020. Unified Neural Encoding of BTFs. Computer Graphics Forum (Proceedings of Eurographics) 39, 2 (June 2020). https://doi.org/10.1111/cgf.13921
31. Szymon M Rusinkie wicz. 1998. A new change of variables for efficient BRDF representation. Rendering techniques 98 (1998), 11–22.
32. Mirko Sattler, Ralf Sarlette, and Reinhard Klein. 2003. Efficient and Realistic Visualization of Cloth. In Proceedings of the 14th Eurographics Workshop on Rendering (Leuven, Belgium) (EGRW ’03). Eurographics Association, Goslar, DEU, 167–177.
33. Christopher Schwartz, Ralf Sarlette, Michael Weinmann, and Reinhard Klein. 2013. DOME II: A Parallelized BTF Acquisition System. In Proceedings of the Eurographics 2013 Workshop on Material Appearance Modeling: Issues and Acquisition (Zaragoza, Spain) (MAM ’13). Eurographics Association, Goslar, DEU, 25–31.
34. C. Schwartz, M. Weinmann, R. Ruiters, and R. Klein. 2011. Integrated High-Quality Acquisition of Geometry and Appearance for Cultural Heritage. In Proceedings of the 12th International Conference on Virtual Reality, Archaeology and Cultural Heritage (Prato, Italy) (VAST’11). Eurographics Association, Goslar, DEU, 25–32.
35. Alejandro Sztrajman, Gilles Rainer, Tobias Ritschel, and Tim Weyrich. 2021. Neural BRDF Representation and Importance Sampling. Computer Graphics Forum n/a, n/a (2021). https://doi.org/10.1111/cgf.14335
36. Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. 2021. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11358–11367.
37. Tanaboon Tongbuasirilai, Jonas Unger, Christine Guillemot, and Ehsan Miandji. 2022. A Sparse Non-parametric BRDF Model. ACM Transactions on Graphics 41, 5 (2022), 1–18.
38. Michael Weinmann, Juergen Gall, and Reinhard Klein. 2014. Material Classification Based on Training Data Synthesized Using a BTF Database. In Computer Vision – ECCV 2014 – 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part III. Springer Internatifonal Publishing, 156–171.
39. Yezi Zhao, Beibei Wang, Yanning Xu, Zheng Zeng, L. Wang, and N. Holzschuch. 2020. Joint SVBRDF Recovery and Synthesis From a Single Image using an Unsupervised Generative Adversarial Network. In EGSR. Wiley.
40. Chuankun Zheng, Ruzhang Zheng, Rui Wang, Shuang Zhao, and Hujun Bao. 2021. A Compact Representation of Measured BRDFs Using Neural Processes. ACM Transactions on Graphics (TOG) 41, 2 (2021), 1–15.
