“Random-Access Neural Compression of Material Textures” by Vaidyanathan, Salvi, Wronski, Akenine-Moller, Ebelin, et al. …
Conference:
Type:
Title:
- Random-Access Neural Compression of Material Textures
Session/Category Title: Cloud Rendering: Your GPU Is Somewhere Else
Presenter(s)/Author(s):
- Karthik Vaidyanathan
- Marco Salvi
- Bartlomiej Wronski
- Tomas Akenine-Moller
- Pontus Ebelin
- Aaron E. Lefohn
Moderator(s):
Abstract:
The continuous advancement of photorealism in rendering is accompanied by a growth in texture data and, consequently, increasing storage and memory demands. To address this issue, we propose a novel neural compression technique specifically designed for material textures. We unlock two more levels of detail, i.e., 16× more texels, using low bitrate compression, with image quality that is better than advanced image compression techniques, such as AVIF and JPEG XL.At the same time, our method allows on-demand, real-time decompression with random access similar to block texture compression on GPUs, enabling compression on disk and memory. The key idea behind our approach is compressing multiple material textures and their mipmap chains together, and using a small neural network, that is optimized for each material, to decompress them. Finally, we use a custom training implementation to achieve practical compression speeds, whose performance surpasses that of general frameworks, like PyTorch, by an order of magnitude.
References:
1. Jyrki Alakuijala, Robert Obryk, Ostap Stoliarchuk, Zoltan Szabadka, Lode Vandevenne, and Jan Wassenberg. 2017. Guetzli: Perceptually Guided JPEG Encoder. arXiv:1703.04421 (2017).
2. Jyrki Alakuijala, Ruud Van Asseldonk, Sami Boukortt, Martin Bruse, Iulia-Maria Coms, a, Moritz Firsching, Thomas Fischbacher, Evgenii Kliuchnikov, Sebastian Gomez, Robert Obryk, et al. 2019. JPEG XL Next-Generation Image Compression Architecture and Coding Tools. In Applications of Digital Image Processing XLII, Vol. 11137. SPIE, 112–124.
3. Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D Fairchild. 2020. LIP: A Difference Evaluator for Alternating Images. Proceedings of the ACM on Computer Graphics and Interactive Techniques 3, 2 (2020), 1–23.
4. Thomas Arcila. 2022. FidelityFX Super Resolution 2.0. In Game Developers Conference.
5. Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. 2017. End-to-end Optimized Image Compression. In International Conference on Learning Representations.
6. Johannes Ballé, Philip Chou, David Minnen, Saurabh Singh, Nick Johnston, Eirikur Agustsson, Sung Hwang, and George Toderici. 2021. Nonlinear Transform Coding. IEEE Journal of Selected Topics in Signal Processing 15, 2 (2021), 339–353.
7. Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. 2018. Variational Image Compression with a Scale Hyperprior. In International Conference on Learning Representations.
8. Yochai Blau and Tomer Michaeli. 2018. The Perception-Distortion Tradeoff. In IEEE Conference on Computer Vision and Pattern Recognition. 6228–6237.
9. Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J. Sullivan, and Jens-Rainer Ohm. 2021. Overview of the Versatile Video Coding (VVC) Standard and its Applications. IEEE Transactions on Circuits and Systems for Video Technology 31, 10 (2021), 3736–3764.
10. Brent Burley and Walt Disney Animation Studios. 2012. Physically-Based Shading at Disney. In ACM SIGGRAPH course.
11. Graham Campbell, Thomas A. DeFanti, Jeff Frederiksen, Stephen A. Joyce, Lawrence A. Leske, John A. Lindberg, and Daniel J. Sandin. 1986. Two Bit/Pixel Full Color Encoding. Computer Graphics (SIGGRAPH) 20, 4 (1986), 215–223.
12. Thomas Chambon, Eric Heitz, and Laurent Belcour. 2021. Passing Multi-Channel Material Textures to a 3-Channel Loss. In ACM SIGGRAPH Talks. Article 12, 2 pages.
13. Yue Chen, Debargha Murherjee, Jingning Han, Adrian Grange, Yaowu Xu, Zoe Liu, Sarah Parker, Cheng Chen, Hui Su, Urvang Joshi, Ching-Han Chiang, Yunqing Wang, Paul Wilkins, Jim Bankoski, Luc Trudeau, Nathan Egge, Jean-Marc Valin, Thomas Davies, Steinar Midtskogen, Andrey Norkin, and Peter de Rivaz. 2018. An Overview of Core Coding Tools in the AV1 Video Codec. In Picture Coding Symposium. IEEE, 41–45.
14. Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. 2020. Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
15. Shin-Fang Chng, Sameera Ramasinghe, Jamie Sherrah, and Simon Lucey. 2022. Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII. Springer, 264–280.
16. Adrian Courrèges. 2020. Graphics Studies Compilation. http://www.adriancourreges.com/blog/2020/12/29/graphics-studies-compilation/
17. I.A. Cunningham, M.S. Westmore, and A. Fenster. 1995. A Stochastic Convolution that Describes both Image Blur and Image Noise using Linear Systems Theory. In International Conference of the Engineering in Medicine and Biology Society, Vol. 1. 555–556.
18. E. Delp and O. Mitchell. 1979. Image Compression Using Block Truncation Coding. IEEE Transactions on Communications 27, 9 (1979), 1335–1342.
19. Manfred Ernst, Marc Stamminger, and Gunther Greiner. 2006. Filter Importance Sampling. In 2006 IEEE Symposium on Interactive Ray Tracing. 125–132.
20. Simon Fenney. 2003. Texture Compression Using Low-Frequency Signal Modulation. In Graphics Hardware. 84–91.
21. The Khronos Group. 2019a. KHR_shader_subgroup. https://github.com/KhronosGroup/GLSL/blob/master/extensions/nv/GLSL_NV_cooperative_matrix.txt
22. The Khronos Group. 2019b. NV_cooperative_matrix. https://github.com/KhronosGroup/GLSL/blob/master/extensions/nv/GLSL_NV_cooperative_matrix.txt
23. The Khronos Group. 2023. Vulkan 1.3.246 – A Specification. https://registry.khronos.org/vulkan/specs/1.3/html/vkspec.html
24. J. Hasselgren, J. Munkberg, M. Salvi, A. Patney, and A. Lefohn. 2020. Neural Temporal Adaptive Sampling and Denoising. Computer Graphics Forum 39, 2 (2020), 147–155.
25. Yong He, Kayvon Fatahalian, and T. Foley. 2018. Slang: Language Mechanisms for Extensible Real-Time Shading Systems. ACM Transactions on Graphics 37, 4, Article 141 (2018).
26. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.08415 (2016).
27. Nikolai Hofmann, Jon Hasselgren, Petrik Clarberg, and Jacob Munkberg. 2021. Interactive Path Tracing and Reconstruction of Sparse Volumes. Proceedings of the ACM on Computer Graphics and Interactive Techniques 4, 1, Article 5 (apr 2021).
28. Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer Deedforward Networks are Universal Approximators. Neural Networks 2, 5 (1989), 359–366.
29. A. Howard, M. Sandler, B. Chen, W. Wang, L. Chen, M. Tan, G. Chu, V. Vasudevan, Y. Zhu, R. Pang, H. Adam, and Q. Le. 2019. Searching for MobileNetV3. In International Conference on Computer Vision. 1314–1324.
30. Stephanie Hurlburt and Rich Geldreich. 2022. Basis Universal Supercompressed GPU Texture Codec. https://github.com/BinomialLLC/basis_universal
31. Intel. 2022. Intel® Arc™ Graphics Developer Guide for Real-Time Ray Tracing in Games. https://www.intel.com/content/www/us/en/developer/articles/guide/real-time-ray-tracing-in-games.html
32. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. In IEEE Conference on Computer Vision and Pattern Recognition. 2704–2713.
33. Robert Kawiak, Hisham Chowdhury, Rense de Boer, Gabriel N. Ferreira, and Lucas Xavier. 2022. Intel Xe Super Sampling (XeSS)—an AI-based Upscaling for Real-Time Rendering. In Game Developers Conference.
34. Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980 (2014).
35. G. Knittel, A. Schilling, A. Kugler, and W. Straßer. 1996. Hardware for Superior Texture Performance. Computers & Graphics 20, 4 (1996), 475–481.
36. Alexandr Kuznetsov, Krishna Mullia, Zexiang Xu, Miloš Hašan, and Ravi Ramamoorthi. 2021. NeuMIP: Multi-Resolution Neural Materials. ACM Transactions on Graphics 40, 4 (2021).
37. Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-Realistic Single Image Super-Resolution using a Generative Adversarial Network. In IEEE Conference on Computer Vision and Pattern Recognition. 4681–4690.
38. David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. 2022. Bacon: Band-limited coordinate networks for multiscale scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16252–16262.
39. Edward Liu. 2022. DLSS 2.0 – Image Reconstruction for Real-Time Rendering with Deep learning. In Game Developers Conference.
40. Haojie Liu, Tong Chen, Peiyao Guo, Qiu Shen, Xun Cao, Yao Wang, and Zhan Ma. 2019. Non-local Attention Optimized Deep Image Compression. ArXiv abs/1904.09757 (2019).
41. Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural Volumes: Learning Dynamic Renderable Volumes from Images. arXiv:1906.07751 (2019).
42. Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations.
43. Fabian Mentzer, George D Toderici, Michael Tschannen, and Eirikur Agustsson. 2020. High-Fidelity Generative Image Compression. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. 11913–11924.
44. Microsoft. 2015. Direct3D 11.3 Functional Specification. https://microsoft.github.io/DirectX-Specs/d3d/archive/D3D11_3_FunctionalSpec.htm
45. Microsoft. 2020. Texture Block Compression in Direct3D 11. https://learn.microsoft.com/en-us/windows/win32/direct3d11/texture-block-compression-in-direct3d-11
46. Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing Scenes as Neural Radiance Fields for View Synthesis. Commun. ACM 65, 1 (2021), 99–106.
47. Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics 41, 4 (2022), 1–15.
48. Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller. 2021. Real-time Neural Radiance Caching for Path Tracing. ACM Transactions on Graphics 40, 4 (2021), 1–16.
49. Jacob Munkberg, Petrik Clarberg, Jon Hasselgren, and Tomas Akenine-Möller. 2006. High Dynamic Range Texture Compression for Graphics Hardware. ACM Transactions on Graphics 25, 3 (2006), 698–706.
50. Krzysztof Narkowicz. 2016. ACES Filmic Tone Mapping Curve. https://knarkowicz.wordpress.com/2016/01/06/aces-filmic-tone-mapping-curve/, January 6.
51. David Neubelt and Matt Pettineo. 2013. Crafting a Next-Gen Material Pipeline for the Order: 1886. Physically Based Shading in Theory and Practice, SIGGRAPH Courses (2013).
52. NVIDIA. 2022a. NVVM IR Specification. https://docs.nvidia.com/cuda/nvvm-ir-spec/index.html
53. NVIDIA. 2022b. Shader Execution Reordering. https://developer.nvidia.com/blog/improve-shader-performance-and-in-game-frame-rates-with-shader-execution-reordering/
54. NVIDIA. 2023. Matrix Fragments for mma.m16n8k16. https://docs.nvidia.com/cuda/parallel-thread-execution/#warp-level-matrix-fragment-mma-16816-float
55. J. Nystad, A. Lassen, A. Pomianowski, S. Ellis, and T. Olson. 2012. Adaptive Scalable Texture Compression. In High-Performance Graphics. 105–114.
56. Yaobin Ouyang, Shiqiu Liu, Markus Kettunen, Matt Pharr, and Jacopo Pantaleoni. 2021. ReSTIR GI: Path Resampling for Real-Time Path Tracing. Computer Graphics Forum 40, 8 (2021), 17–29.
57. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Love-grove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In Conference on Computer Vision and Pattern Recognition.
58. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32.
59. Gregg William Perkins and Santiago Echeverry. 2022. Virtual Production in Action: A Creative Implementation of Expanded Cinematography and Narratives. In ACM SIGGRAPH 2022 Posters. Article 21.
60. Aras Pranckevičius. 2020. Texture Compression in 2020. https://aras-p.info/blog/2020/12/08/Texture-Compression-in-2020/
61. Oren Rippel and Lubomir Bourdev. 2017. Real-Time Adaptive Image Compression. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 2922–2930.
62. Kimmo Roimela, Tomi Aarnio, and Joonas Itäranta. 2006. High Dynamic Range Texture Compression. ACM Transactions on Graphics 25, 3 (2006), 707–712.
63. Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. 2023. WIRE: Wavelet Implicit Neural Representations. arXiv preprint arXiv:2301.05187 (2023).
64. Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. 2020. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems 33 (2020), 7462–7473.
65. A. Skodras, C. Christopoulos, and T. Ebrahimi. 2001. The JPEG 2000 still image compression standard. IEEE Signal Processing Magazine 18, 5 (2001), 36–58.
66. Jacob Ström and Tomas Akenine-Möller. 2005. iPACKMAN: High-Quality, Low-Complexity Texture Compression for Mobile Phones. In Graphics Hardware. 63–70.
67. Jacob Ström and Martin Pettersson. 2007. ETC2: Texture Compression Using Invalid Combinations. In Graphics Hardware. 49–54.
68. Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. 2022. Variable Bitrate Neural Fields. In ACM SIGGRAPH 2022 Conference Proceedings. 1–9.
69. Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. 2020. Fourier Features let Networks Learn High Frequency Functions in Low Dimensional Domains. Advances in Neural Information Processing Systems 33 (2020), 7537–7547.
70. Natalya Tatarchuk, Jonathan Dupuy, Thomas Deliot, Daniel Wright, Krzysztof Narkowicz, Patrick Kelly, Aleksander Netzel, and Tiago Costa. 2022. Advances in Real-Time Rendering in Games: Part I. In ACM SIGGRAPH Courses. Article 18.
71. Natalya Tatarchuk, Timothy Lottes, Kleber Garcia, Thomas Deliot, Jonathan Dupuy, Kees Rijnen, Xiaoling Yao, Brian Karis, Graham Wihlidal, Rune Stubbe, and Ari Silvennoinen. 2021. Advances in Real-Time Rendering in Games: Part I. In ACM SIGGRAPH Courses.
72. Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. 2022. Advances in Neural Rendering. In Computer Graphics Forum, Vol. 41. 703–735.
73. Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. 2017. Lossy Image Compression with Compressive Autoencoders. In International Conference on Learning Representations.
74. Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred Neural Rendering: Image Synthesis using Neural Textures. ACM Transactions on Graphics 38, 4 (2019), 1–12.
75. Manu Mathew Thomas, Gabor Liktor, Christoph Peters, Sungye Kim, Karthik Vaidyanathan, and Angus G. Forbes. 2022. Temporally Stable Real-Time Joint Neural Denoising and Supersampling. Proceedings of the ACM on Computer Graphics and Interactive Techniques 5, 3, Article 21 (jul 2022).
76. T. S. Trowbridge and K. P. Reitz. 1975. Average Irregularity Representation of a Rough Surface for Ray Reflection. Journal of the Optical Society of America 65, 5 (May 1975), 531–536.
77. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018. Deep Image Prior. In IEEE Conference on Computer Vision and Pattern Recognition. 9446–9454.
78. Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural Discrete Representation Learning. In International Conference on Neural Information Processing Systems.
79. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).
80. Gregory K Wallace. 1991. The JPEG Still Picture Compression Standard. Commun. ACM 34, 4 (1991), 30–44.
81. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600–612.
82. Wikipedia. 2022. S3 Texture Compression. https://en.wikipedia.org/wiki/S3_Texture_Compression (accessed on 2022-11-14).
83. Alan Wolfe, Nathan Morrical, Tomas Akenine-Möller, and Ravi Ramamoorthi. 2022. Spatiotemporal Blue Noise Masks. In Eurographics Symposium on Rendering.
84. Bartlomiej Wronski. 2020. Dimensionality Reduction for Image and Texture Set Compression. https://bartwronski.com/2020/05/21/dimensionality-reduction-for-image-and-texture-set-compression/
85. Bartlomiej Wronski. 2021. Neural Material (De)compression – Data-Driven Nonlinear Dimensionality Reduction. https://bartwronski.com/2021/05/30/neural-material-decompression-data-driven-nonlinear-dimensionality-reduction/
86. Da Yan, Wei Wang, and Xiaowen Chu. 2020. Demystifying Tensor Cores to Optimize Half-Precision Matrix Multiply. In IEEE International Parallel and Distributed Processing Symposium. 634–643.
87. Lei Yang, Shiqiu Liu, and Marco Salvi. 2020. A Survey of Temporal Antialiasing Techniques. Computer Graphics Forum 39, 2 (2020), 607–621.
88. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In IEEE Conference on Computer Vision and Pattern Recognition. 586–595.
89. Maria Zontak and Michal Irani. 2011. Internal Statistics of a Single Natural Image. In Conference on Computer Vision and Pattern Recognition. 977–984.
