“Real-Time Neural Appearance Models”
Conference:
Type(s):
Title:
- Real-Time Neural Appearance Models
Presenter(s)/Author(s):
Abstract:
We present a real-time rendering system for film-quality materials which bakes complex layered material graphs into compact neural representations. We use learned hierarchical textures and neural decoders which use graphics priors to produce neural BSDFs. Accelerated tensor operations facilitate seamless integration into real-time path tracers, ensuring high performance.
References:
[1]
Tomas Akenine-M?ller, Cyril Crassin, Jakub Boksansky, Laurent Belcour, Alexey Panteleev, and Oli Wright. 2021. Improved shader and texture level of detail using ray cones. Journal of Computer Graphics Techniques 10, 1(2021), 1?24. Retrieved from http://jcgt.org/published/0010/01/01/
[2]
Pontus Andersson, Jim Nilsson, Tomas Akenine-M?ller, Magnus Oskarsson, Kalle ?str?m, and Mark D. Fairchild. 2020. LIP: A difference evaluator for alternating images. Proceedings of the ACM on Computer Graphics and Interactive Techniques 3, 2, Article 15 (Aug 2020), 23 pages.
[3]
Hendrik Baatz, Jonathan Granskog, Marios Papas, Fabrice Rousselle, and Jan Nov?k. 2022. NeRF-Tex: Neural reflectance field textures. Computer Graphics Forum 41, 6 (2022), 287?301.
[4]
Yaoyi Bai, Songyin Wu, Zheng Zeng, Beibei Wang, and Ling-Qi Yan. 2022. BSDF Importance Baking: A Lightweight Neural Solution to Importance Sampling Parametric BSDFs. DOI:
[5]
Steve Bako, Pradeep Sen, and Anton Kaplanyan. 2023. Deep appearance prefiltering. ACM Transactions on Graphics 42, 2, Article 23 (Jan 2023), 23 pages.
[6]
Petrik Clarberg, Simon Kallweit, Craig Kolb, Pawel Kozlowski, Yong He, Lifan Wu, and Edward Liu. 2022a. Research Advances Toward Real-Time Path Tracing. Game Developers Conference (GDC).
[7]
Petrik Clarberg, Simon Kallweit, Craig Kolb, Pawel Kozlowski, Yong He, Lifan Wu, Edward Liu, Benedikt Bitterli, and Matt Pharr. 2022b. Real-Time Path Tracing and Beyond. HPG 2022 Keynote.
[8]
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. 2017. Density estimation using Real NVP. In Proceedings of the International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=HkpbnH9lx
[9]
Jonathan Dupuy. 2015. Photorealistic Surface Rendering with Microfacet Theory. Ph. D. Dissertation. Universit? Claude Bernard – Lyon I ; Universit? de Montr?al.
[10]
Jonathan Dupuy, Eric Heitz, Jean-Claude Iehl, Pierre Poulin, Fabrice Neyret, and Victor Ostromoukhov. 2013. Linear efficient antialiased displacement and reflectance mapping. ACM Transactions on Graphics 32, 6, Article 211 (2013), 11 pages.
[11]
Jonathan Dupuy and Wenzel Jakob. 2018. An adaptive parameterization for efficient material acquisition and rendering. ACM Transactions on Graphics 37, 6, Article 274 (2018), 14 pages.
[12]
Jiahui Fan, Beibei Wang, Milo? Ha?an, Jian Yang, and Ling-Qi Yan. 2022. Neural layered BRDFs. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada). Association for Computing Machinery, New York, NY, USA, 8 pages.
[13]
Alban Gauthier, Robin Faury, J?r?my Levallois, Th?o Thonat, Jean-Marc Thiery, and Tamy Boubekeur. 2022. MIPNet: Neural normal-to-anisotropic-roughness MIP mapping. ACM Transactions on Graphics 41, 6(2022), 12 pages.
[14]
Yong He, Kayvon Fatahalian, and Theresa Foley. 2018. Slang: Language mechanisms for extensible real-time shading systems. ACM Transactions on Graphics 37, 4, Article 141 (2018), 13 pages.
[15]
Wenzel Jakob, Andrea Weidlich, Andrew Beddini, Rob Piek?, Hanzhi Tang, Luca Fascione, and Johannes Hanika. 2019. Path tracing in production: Part 2: Making movies. In Proceedings of the ACM SIGGRAPH 2019 Courses (Los Angeles, California). Association for Computing Machinery, New York, NY, USA, 41 pages.
[16]
Simon Kallweit, Petrik Clarberg, Craig Kolb, Tom?? Davidovi?, Kai-Hwa Yao, Theresa Foley, Yong He, Lifan Wu, Lucy Chen, Tomas Akenine-M?ller, Chris Wyman, Cyril Crassin, and Nir Benty. 2022. The Falcor Rendering Framework (version 5.2). Retrieved 01-May-2024 from https://github.com/NVIDIAGameWorks/Falcor
[17]
Alexandr Kuznetsov, Milo? Ha?an, Zexiang Xu, Ling-Qi Yan, Bruce Walter, Nima Khademi Kalantari, Steve Marschner, and Ravi Ramamoorthi. 2019. Learning generative models for rendering specular microgeometry. ACM Transactions on Graphics 38, 6, Article 225 (2019), 14 pages.
[18]
Alexandr Kuznetsov, Krishna Mullia, Zexiang Xu, Milo? Ha?an, and Ravi Ramamoorthi. 2021. NeuMIP: Multi-resolution neural materials. ACM Transactions on Graphics 40, 4, Article 175 (2021), 13 pages.
[19]
Alexandr Kuznetsov, Xuezheng Wang, Krishna Mullia, Fujun Luan, Zexiang Xu, Milos Hasan, and Ravi Ramamoorthi. 2022. Rendering neural materials on curved surfaces. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada). Association for Computing Machinery, New York, NY, USA, 9 pages.
[20]
Samuli Laine, Tero Karras, and Timo Aila. 2013. Megakernels considered harmful: Wavefront path tracing on GPUs. In Proceedings of the 5th High-Performance Graphics Conference (Anaheim, California). Association for Computing Machinery, New York, NY, USA, 137?143.
[21]
Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A data-driven reflectance model. ACM Transactions on Graphics 22, 3 (2003), 759?769.
[22]
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing scenes as neural radiance fields for view synthesis. In Computer Vision ? ECCV 2020. Springer International Publishing, Cham, 405?421.
[23]
Thomas M?ller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics 41, 4(2022), 15 pages. DOI:
[24]
Thomas M?ller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Nov?k. 2019. Neural importance sampling. ACM Transactions on Graphics 38, 5, Article 145 (2019), 19 pages.
[25]
Thomas M?ller, Fabrice Rousselle, Alexander Keller, and Jan Nov?k. 2020. Neural control variates. ACM Transactions on Graphics 39, 6(2020), 19 pages. DOI:
[26]
Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, and Tobias Ritschel. 2017. Deep shading: Convolutional neural networks for screen space shading. Computer Graphics Forum 36, 4 (2017), 65?78.
[27]
Marc Olano and Dan Baker. 2010. LEAN mapping. In Proceedings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (Washington, D.C.). Association for Computing Machinery, New York, NY, USA, 181?188.
[28]
Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically Based Rendering, Third Edition: From Theory to Implementation. Morgan Kaufmann.
[29]
Gilles Rainer, Abhijeet Ghosh, Wenzel Jakob, and Tim Weyrich. 2020. Unified neural encoding of BTFs. Computer Graphics Forum 39, 2 (2020), 167?178.
[30]
Gilles Rainer, Wenzel Jakob, Abhijeet Ghosh, and Tim Weyrich. 2019. Neural BTF compression and interpolation. Computer Graphics Forum 38, 2 (2019), 235?244.
[31]
Daniel Rebain, Mark J. Matthews, Kwang Moo Yi, Gopal Sharma, Dmitry Lagun, and Andrea Tagliasacchi. 2023. Attention beats concatenation for conditioning neural fields. Transactions on Machine Learning Research (2023). https://openreview.net/forum?id=GzqdMrFQsE
[32]
Szymon M. Rusinkiewicz. 1998. A new change of variables for efficient BRDF representation. In Proceedings of the Eurographics Workshop on Rendering Techniques. Springer, Springer Vienna, Vienna, 11?22.
[33]
Doug Smythe and Jonathan Stone. 2021. MaterialX: An Open Standard for Network-Based CG Object Looks, Version 1.38. Retrieved 01-May-2024 from https://materialx.org/assets/MaterialX.v1.38.Spec.pdf
[34]
Alejandro Sztrajman, Gilles Rainer, Tobias Ritschel, and Tim Weyrich. 2021. Neural BRDF representation and importance sampling. Computer Graphics Forum 40, 6 (2021), 332?346.
[35]
Justus Thies, Michael Zollh?fer, and Matthias Nie?ner. 2019. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics 38, 4(2019), 12 pages.
[36]
T. S. Trowbridge and K. P. Reitz. 1975. Average irregularity representation of a rough surface for ray reflection. Journal of the Optical Society of America 65, 5 (1975), 531?536.
[37]
Karthik Vaidyanathan, Marco Salvi, Bartlomiej Wronski, Tomas Akenine-M?ller, Pontus Ebelin, and Aaron Lefohn. 2023. Random-access neural compression of material textures. ACM Transactions on Graphics 42, 4, Article 88 (2023), 25 pages.
[38]
Dietger van Antwerpen. 2011. Improving SIMD efficiency for parallel Monte Carlo light transport on the GPU. In Proceedings of the ACM SIGGRAPH Symposium on High Performance Graphics (Vancouver, British Columbia, Canada). Association for Computing Machinery, New York, NY, USA, 41?50.
[39]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ?ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc.
[40]
Eric Veach and Leonidas J. Guibas. 1995. Optimally combining sampling techniques for Monte Carlo rendering. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH?95). Association for Computing Machinery, New York, NY, USA, 419?428.
[41]
Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard R?thlin, Alex Harvill, David Adler, Mark Meyer, and Jan Nov?k. 2018. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2018) 37, 4 (2018), 124:1?124:15 pages. DOI:
[42]
Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet models for refraction through rough surfaces. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (Grenoble, France) (EGSR?07). Eurographics Association, Goslar, DEU, 195?206.
[43]
Bing Xu, Liwen Wu, Milo? Ha?an, Fujun Luan, Iliyan Georgiev, Zexiang Xu, and Ravi Ramamoorthi. 2023. NeuSample: Importance sampling for neural materials. In ACM SIGGRAPH 2023 Conference Proceedings (Los Angeles, CA, USA). Association for Computing Machinery, New York, NY, USA, Article 41 (2023), 10 pages.
[44]
Chuankun Zheng, Ruzhang Zheng, Rui Wang, Shuang Zhao, and Hujun Bao. 2021. A compact representation of measured BRDFs using neural processes. ACM Transactions on Graphics 41, 2, Article 14 (Nov 2021), 15 pages.
