“NeuMIP: multi-resolution neural materials” by Kuznetsov, Mullia, Xu, Hasan and Ramamoorthi

We propose NeuMIP, a neural method for representing and rendering a variety of material appearances at different scales. Classical prefiltering (mipmapping) methods work well on simple material properties such as diffuse color, but fail to generalize to normals, self-shadowing, fibers or more complex microstructures and reflectances. In this work, we generalize traditional mipmap pyramids to pyramids of neural textures, combined with a fully connected network. We also introduce neural offsets, a novel method which enables rendering materials with intricate parallax effects without any tessellation. This generalizes classical parallax mapping, but is trained without supervision by any explicit heightfield. Neural materials within our system support a 7-dimensional query, including position, incoming and outgoing direction, and the desired filter kernel size. The materials have small storage (on the order of standard mipmapping except with more texture channels), and can be integrated within common Monte-Carlo path tracing systems. We demonstrate our method on a variety of materials, resulting in complex appearance across levels of detail, with accurate parallax, self-shadowing, and other effects.

1. Adobe. 2021. Substance Source library. https://source.substance3d.com.Google Scholar
2. Barry G Becker and Nelson L Max. 1993. Smooth transitions between bump rendering algorithms. In SIGGRAPH 93. 183–190.Google Scholar
3. Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824 (2020).Google Scholar
4. Eric Bruneton and Fabrice Neyret. 2011. A survey of nonlinear prefiltering methods for efficient and accurate surface shading. IEEE Transactions on Visualization and Computer Graphics 18, 2 (2011), 242–260.Google ScholarDigital Library
5. Matt Jen-Yuan Chiang, Benedikt Bitterli, Chuck Tappan, and Brent Burley. 2016. A Practical and Controllable Hair and Fur Model for Production Path Tracing. Computer Graphics Forum 35, 2 (2016), 275–283.Google ScholarCross Ref
6. Kristin J Dana, Bram Van Ginneken, Shree K Nayar, and Jan J Koenderink. 1999. Reflectance and texture of real-world surfaces. ACM Transactions on Graphics (TOG) 18, 1 (1999), 1–34.Google ScholarDigital Library
7. Jonathan Dupuy, Eric Heitz, Jean-Claude Iehl, Pierre Poulin, Fabrice Neyret, and Victor Ostromoukhov. 2013. Linear efficient antialiased displacement and reflectance mapping. ACM Transactions on Graphics (TOG) 32, 6 (2013), 1–11.Google ScholarDigital Library
8. Jiři Filip and Michal Haindl. 2008. Bidirectional texture function modeling: A state of the art survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 11 (2008), 1921–1940.Google ScholarDigital Library
9. Charles Han, Bo Sun, Ravi Ramamoorthi, and Eitan Grinspun. 2007. Frequency Domain Normal Map Filtering. ACM Transactions on Graphics (TOG) 26, 3, 28.Google ScholarDigital Library
10. Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.Google Scholar
11. Wenzel Jakob, Miloš Hašan, Ling-Qi Yan, Jason Lawrence, Ravi Ramamoorthi, and Steve Marschner. 2014. Discrete stochastic microfacet models. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–10.Google ScholarDigital Library
12. Tomomichi Kaneko, Toshiyuki Takahei, Masahiko Inami, Naoki Kawakami, Yasuyuki Yanagida, Taro Maeda, and Susumu Tachi. 2001. Detailed shape representation with parallax mapping. In Proceedings of the ICAT 2001 (01 2001).Google Scholar
13. Anton S Kaplanyan, Stephan Hill, Anjul Patney, and Aaron E Lefohn. 2016. Filtering distributions of normals for shading antialiasing.. In High Performance Graphics. 151–162.Google Scholar
14. Melissa L Koudelka, Sebastian Magda, Peter N Belhumeur, and David J Kriegman. 2003. Acquisition, compression, and synthesis of bidirectional texture functions. In 3rd International Workshop on Texture Analysis and Synthesis (Texture 2003). 59–64.Google Scholar
15. Alexandr Kuznetsov, Milos Hasan, Zexiang Xu, Ling-Qi Yan, Bruce Walter, Nima Khademi Kalantari, Steve Marschner, and Ravi Ramamoorthi. 2019. Learning generative models for rendering specular microgeometry. ACM Transactions on Graphics (TOG) 38, 6 (2019), 225.Google ScholarDigital Library
16. Maxim Maximov, Laura Leal-Taixé, Mario Fritz, and Tobias Ritschel. 2019. Deep appearance maps. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8729–8738.Google ScholarCross Ref
17. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV 2020.Google ScholarDigital Library
18. Gero Müller, Jan Meseth, and Reinhard Klein. 2003. Compression and Real-Time Rendering of Measured BTFs Using Local PCA.. In VMV. 271–279.Google Scholar
19. Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural Importance Sampling. ACM Transactions on Graphics (TOG) 38, 5, Article 145 (2019), 19 pages.Google ScholarDigital Library
20. Thomas Müller, Fabrice Rousselle, Alexander Keller, and Jan Novák. 2020. Neural Control Variates. ACM Transactions on Graphics (TOG) 39, 6, Article 243 (2020), 19 pages.Google ScholarDigital Library
21. Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, and Tobias Ritschel. 2017. Deep Shading: Convolutional Neural Networks for Screen-Space Shading. Computer Graphics Forum (Proc. EGSR 2017) 36, 4 (2017), 65–78.Google Scholar
22. Marc Olano and Dan Baker. 2010. Lean mapping. In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. 181–188.Google ScholarDigital Library
23. Manuel M Oliveira and Fabio Policarpo. 2005. An efficient representation for surface details. UFRGS Technical Report (2005).Google Scholar
24. Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2018. PBRT v3. https://github.com/mmp/pbrt-v3/.Google Scholar
25. Gilles Rainer, Abhijeet Ghosh, Wenzel Jakob, and Tim Weyrich. 2020. Unified Neural Encoding of BTFs. Computer Graphics Forum (Proceedings of Eurographics) 39, 2 (2020), 167–178.Google ScholarCross Ref
26. Gilles Rainer, Wenzel Jakob, Abhijeet Ghosh, and Tim Weyrich. 2019. Neural BTF Compression and Interpolation. Computer Graphics Forum (Proceedings of Eurographics) 38, 2 (2019), 235–244.Google ScholarCross Ref
27. Konstantinos Rematas, Tobias Ritschel, Mario Fritz, Efstratios Gavves, and Tinne Tuytelaars. 2016. Deep reflectance maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4508–4516.Google ScholarCross Ref
28. Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–12.Google ScholarDigital Library
29. Jiaping Wang, Xin Tong, John Snyder, Yanyun Chen, Baining Guo, and Heung-Yeung Shum. 2005. Capturing and rendering geometry details for BTF-mapped surfaces. The Visual Computer 21, 8 (2005), 559–568.Google ScholarCross Ref
30. Michael Weinmann, Juergen Gall, and Reinhard Klein. 2014. Material Classification Based on Training Data Synthesized Using a BTF Database. In ECCV 2014. 156–171.Google ScholarCross Ref
31. Lance Williams. 1983. Pyramidal parametrics. In SIGGRAPH 83. 1–11.Google Scholar
32. Lifan Wu, Shuang Zhao, Ling-Qi Yan, and Ravi Ramamoorthi. 2019. Accurate appearance preserving prefiltering for rendering displacement-mapped surfaces. ACM Transactions on Graphics (TOG) 38, 4 (2019), 137.Google ScholarDigital Library
33. Ling-Qi Yan, Weilun Sun, Henrik Wann Jensen, and Ravi Ramamoorthi. 2017. A BSSRDF Model for Efficient Rendering of Fur with Global Illumination. ACM Transactions on Graphics (TOG) 36, 6 (2017), 208.Google ScholarDigital Library
34. Shuang Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling scattering parameters for rendering anisotropic media. ACM Transactions on Graphics (TOG) 35, 6 (2016), 166.Google ScholarDigital Library