“Real-time neural radiance caching for path tracing” by Müller, Rousselle, Novák and Keller

We present a real-time neural radiance caching method for path-traced global illumination. Our system is designed to handle fully dynamic scenes, and makes no assumptions about the lighting, geometry, and materials. The data-driven nature of our approach sidesteps many difficulties of caching algorithms, such as locating, interpolating, and updating cache points. Since pretraining neural networks to handle novel, dynamic scenes is a formidable generalization challenge, we do away with pretraining and instead achieve generalization via adaptation, i.e. we opt for training the radiance cache while rendering. We employ self-training to provide low-noise training targets and simulate infinite-bounce transport by merely iterating few-bounce training updates. The updates and cache queries incur a mild overhead—about 2.6ms on full HD resolution—thanks to a streaming implementation of the neural network that fully exploits modern hardware. We demonstrate significant noise reduction at the cost of little induced bias, and report state-of-the-art, real-time performance on a number of challenging scenarios.

1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. http://tensorflow.org/Google Scholar
2. Michael Abrash. 1997. Quake’s lighting model: Surface caching. In Graphics Programming Black Book. Coriolis Group, Chapter 68, 1245–1256.Google Scholar
3. Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D. Fairchild. 2020. FLIP: A Difference Evaluator for Alternating Images. Proc. ACM Comput. Graph. Interact. Tech. 3, 2, Article 15 (Aug. 2020), 23 pages. Google ScholarDigital Library
4. Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. 2020. Second Order Optimization Made Practical. arXiv:2002.09018 (Feb. 2020).Google Scholar
5. James Arvo. 1986. Backward Ray Tracing. In In ACM SIGGRAPH ’86 Course Notes – Developments in Ray Tracing. 259–263.Google Scholar
6. Colin Barré-Brisebois. 2017. A Certain Slant of Light: Past, Present and Future Challenges of Global Illumination in Games. In Open problems in real-time rendering. ACM SIGGRAPH 2017 Courses.Google Scholar
7. Philippe Bekaert, Philipp Slusallek, Ronald Cools, Vlastimil Havran, and Hans-Peter Seidel. 2003. A custom designed Density Estimation Method for Light Transport. Technical Report. Max-Planck-Institut für Informatik, Saarbrücken, Germany.Google Scholar
8. Laurent Belcour, Cyril Soler, Kartic Subr, Nicolas Holzschuch, and Fredo Durand. 2013. 5D Covariance Tracing for Efficient Defocus and Motion Blur. ACM Trans. Graph. 32, 3, Article Article 31 (July 2013), 18 pages. Google ScholarDigital Library
9. Nir Benty, Kai-Hwa Yao, Petrik Clarberg, Lucy Chen, Simon Kallweit, Tim Foley, Matthew Oakes, Conor Lavelle, and Chris Wyman. 2020. The Falcor Rendering Framework. https://github.com/NVIDIAGameWorks/Falcor https://github.com/NVIDIAGameWorks/Falcor.Google Scholar
10. Nikolaus Binder, Sascha Fricke, and Alexander Keller. 2018. Fast Path Space Filtering by Jittered Spatial Hashing. In ACM SIGGRAPH 2018 Talks (SIGGRAPH ’18). Association for Computing Machinery, New York, NY, USA, Article 71, 2 pages. Google ScholarDigital Library
11. Benedikt Bitterli, Chris Wyman, Matt Pharr, Peter Shirley, Aaron Lefohn, and Wojciech Jarosz. 2020. Spatiotemporal reservoir resampling for real-time ray tracing with dynamic direct lighting. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020). https://doi.org/10/gg8xc7Google Scholar
12. Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2018. The Design and Evolution of Disney’s Hyperion Renderer. ACM Trans. Graph. 37, 3, Article 33 (July 2018), 22 pages. Google ScholarDigital Library
13. Cyril Crassin, Fabrice Neyret, Miguel Sainz, Simon Green, and Elmar Eisemann. 2011. Interactive Indirect Illumination Using Voxel Cone Tracing. Computer Graphics Forum 30, 7 (2011), 1921–1930. Google ScholarCross Ref
14. Carsten Dachsbacher and Marc Stamminger. 2005. Reflective Shadow Maps. In Proceedings of the 2005 Symposium on Interactive 3D Graphics and Games (I3D ’05). Association for Computing Machinery, New York, NY, USA, 203–231. Google ScholarDigital Library
15. Ken Dahm and Alexander Keller. 2018. Learning Light Transport the Reinforced Way. In Monte Carlo and Quasi-Monte Carlo Methods, Art B. Owen and Peter W. Glynn (Eds.). Springer International Publishing, 181–195.Google Scholar
16. William J. Dally, Yatish Turakhia, and Song Han. 2020. Domain-Specific Hardware Accelerators. Commun. ACM 63, 7 (June 2020), 48–57. Google ScholarDigital Library
17. Johannes Deligiannis and Jan Schmid. 2019. “It Just Works” Ray-Traced Reflections in ‘Battlefield V’. In Game Developers Conference.Google Scholar
18. Addis Dittebrandt, Johannes Hanika, and Carsten Dachsbacher. 2020. Temporal Sample Reuse for Next Event Estimation and Path Guiding for Real-Time Path Tracing. In Eurographics Symposium on Rendering – DL-only Track, Carsten Dachsbacher and Matt Pharr (Eds.). The Eurographics Association. Google ScholarCross Ref
19. Renaud Adrien Dubouchet, Laurent Belcour, and Derek Nowrouzezahrai. 2017. Frequency Based Radiance Cache for Rendering Animations. In Proceedings of the Eurographics Symposium on Rendering: Experimental Ideas & Implementations (EGSR ’17). Eurographics Association, Goslar, DEU, 41–53. Google ScholarDigital Library
20. Boris Ginsburg, Patrice Castonguay, Oleksii Hrinchuk, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Huyen Nguyen, and Jonathan M. Cohen. 2019. Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks. arXiv:1905.11286 (June 2019).Google Scholar
21. Gene Greger, Peter Shirley, Philip M. Hubbard, and Donald P. Greenberg. 1998. The irradiance volume. IEEE Computer Graphics and Applications 18, 2 (1998), 32–43.Google ScholarDigital Library
22. Vineet Gupta, Tomer Koren, and Yoram Singer. 2018. Shampoo: Preconditioned Stochastic Tensor Optimization. arXiv:1802.09568 (Aug. 2018).Google Scholar
23. Jon Hasselgren, Jacob Munkberg, Marco Salvi, Anjul Patney, and Aaron Lefohn. 2020. Neural Temporal Adaptive Sampling and Denoising. Computer Graphics Forum 39, 2 (2020), 147–155. Google ScholarCross Ref
24. Paul S. Heckbert. 1990. Adaptive Radiosity Textures for Bidirectional Ray Tracing. SIGGRAPH Comput. Graph. 24, 4 (Sept. 1990), 145–154. Google ScholarDigital Library
25. Sebastian Herholz, Oskar Elek, Jens Schindel, Jaroslav Křivánek, and Hendrik P. A. Lensch. 2018. A Unified Manifold Framework for Efficient BRDF Sampling based on Parametric Mixture Models. In Eurographics Symposium on Rendering – Experimental Ideas & Implementations, Wenzel Jakob and Toshiya Hachisuka (Eds.). The Eurographics Association.Google Scholar
26. Sebastian Herholz, Oskar Elek, Jiří Vorba, Hendrik Lensch, and Jaroslav Křivánek. 2016. Product Importance Sampling for Light Transport Path Guiding. Computer Graphics Forum (2016). Google ScholarCross Ref
27. Pedro Hermosilla, Sebastian Maisch, Tobias Ritschel, and Timo Ropinski. 2019. Deep-learning the Latent Space of Light Transport. Computer Graphics Forum 38, 4 (2019).Google Scholar
28. John T. Hooker. 2016. Volumetric Global Illumination at Treyarch. In Advances in real-time rendering, part I. ACM SIGGRAPH 2016 Courses.Google Scholar
29. Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. 2020. Meta-Learning in Neural Networks: A Survey. arXiv:2004.05439 (April 2020).Google Scholar
30. Michał Iwanicki and Peter-Pike Sloan. 2017. Precomputed Lighting in Call of Duty: Infinite Warfare. In Advances in Real-Time Rendering, Part I (ACM SIGGRAPH 2017 Courses). Association for Computing Machinery, New York, NY, USA, Article 7a, 1 pages. Google ScholarDigital Library
31. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging Weights Leads to Wider Optima and Better Generalization. arXiv:1803.05407 (March 2018).Google Scholar
32. Wojciech Jarosz, Craig Donner, Matthias Zwicker, and Henrik Wann Jensen. 2008. Radiance Caching for Participating Media. ACM Trans. Graph. 27, 1, Article 7 (March 2008), 11 pages. Google ScholarDigital Library
33. Henrik Wann Jensen. 1995. Importance Driven Path Tracing using the Photon Map. In Rendering Techniques. Springer Vienna, Vienna, 326–335. Google ScholarCross Ref
34. Henrik Wann Jensen. 1996. Global Illumination using Photon Maps. In Rendering Techniques ’96, Xavier Pueyo and Peter Schröder (Eds.). Springer Vienna, Vienna, 21–30.Google Scholar
35. Giulio Jiang and Bernhard Kainz. 2021. Deep radiance caching: Convolutional autoencoders deeper in ray tracing. Computers & Graphics 94 (2021), 22 — 31. Google ScholarCross Ref
36. James T. Kajiya. 1986. The Rendering Equation. Computer Graphics 20 (1986), 143–150.Google ScholarDigital Library
37. Simon Kallweit, Thomas Müller, Brian McWilliams, Markus Gross, and Jan Novák. 2017. Deep Scattering: Rendering Atmospheric Clouds with Radiance-Predicting Neural Networks. ACM Trans. Graph. 36, 6, Article 231 (Nov. 2017), 11 pages. Google ScholarDigital Library
38. Anton Kaplanyan and Carsten Dachsbacher. 2010. Cascaded Light Propagation Volumes for Real-Time Indirect Illumination. In Proceedings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D ’10). Association for Computing Machinery, New York, NY, USA, 99–107. Google ScholarDigital Library
39. Alexander Keller. 1997. Instant Radiosity. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’97). ACM Press/Addison-Wesley Publishing Co., USA, 49–56. Google ScholarDigital Library
40. Alexander Keller and Ken Dahm. 2019. Integral Equations and Machine Learning. Mathematics and Computers in Simulation 161 (2019), 2–12.Google ScholarDigital Library
41. Alexander Keller, Timo Viitanen, Colin Barré-Brisebois, Christoph Schied, and Morgan McGuire. 2019. Are We Done with Ray Tracing?. In ACM SIGGRAPH 2019 Courses (SIGGRAPH’19). Association for Computing Machinery, New York, NY, USA, Article 3, 381 pages. Google ScholarDigital Library
42. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 (June 2014).Google Scholar
43. Jaroslav Křivánek and Pascal Gautron. 2009. Practical Global Illumination with Irradiance Caching. Morgan & Claypool.Google Scholar
44. Jaroslav Křivánek, Pascal Gautron, Sumanta Pattanaik, and Kadi Bouatouch. 2005. Radiance caching for efficient global illumination computation. IEEE Transactions on Visualization and Computer Graphics 11, 5 (2005), 550–561.Google ScholarDigital Library
45. Eric P. Lafortune and Yves D. Willems. 1995. A 5D Tree to Reduce the Variance of Monte Carlo Ray Tracing. In Proc. EGWR. 11–20.Google Scholar
46. Sébastien Lagarde and Charles de Rousiers. 2014. Moving Frostbite to Physically Based Rendering 3.0. In ACM SIGGRAPH Courses: Physically Based Shading in Theory and Practice, Chapter 10 (SIGGRAPH ’14). ACM, New York, NY, USA. Google ScholarDigital Library
47. Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2Noise: Learning Image Restoration without Clean Data. arXiv:1803.04189 (March 2018).Google Scholar
48. Zander Majercik, Jean-Philippe Guertin, Derek Nowrouzezahrai, and Morgan McGuire. 2019. Dynamic Diffuse Global Illumination with Ray-Traced Irradiance Fields. Journal of Computer Graphics Techniques (JCGT) 8, 2 (5 June 2019), 1–30. http://jcgt.org/published/0008/02/01/Google Scholar
49. Julio Marco, Adrian Jarabo, Wojciech Jarosz, and Diego Gutierrez. 2018. Second-Order Occlusion-Aware Volumetric Radiance Caching. ACM Trans. Graph. 37, 2, Article 20 (July 2018), 14 pages. Google ScholarDigital Library
50. Sam Martin and Per Einarsson. 2010. A Real-Time Radiosity Architecture for Video Games. SIGGRAPH 2010 Course: Advances in Real-Time Rendering in 3D Graphics and Games.Google Scholar
51. Morgan McGuire, Mike Mara, Derek Nowrouzezahrai, and David Luebke. 2017. Real-Time Global Illumination Using Precomputed Light Field Probes. In Proceedings of the 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D ’17). Association for Computing Machinery, New York, NY, USA, Article 2, 11 pages. Google ScholarDigital Library
52. Johannes Meng, Marios Papas, Ralf Habel, Carsten Dachsbacher, Steve Marschner, Markus Gross, and Wojciech Jarosz. 2015. Multi-Scale Modeling and Rendering of Granular Materials. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 34, 4 (July 2015). https://doi.org/10/gfzndrGoogle Scholar
53. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.Google Scholar
54. Martin Mittring. 2007. Finding next Gen: CryEngine 2. In ACM SIGGRAPH 2007 Courses (SIGGRAPH ’07). Association for Computing Machinery, New York, NY, USA, 97–121. Google ScholarDigital Library
55. Pierre Moreau, Matt Pharr, and Petrik Clarberg. 2019. Dynamic Many-Light Sampling for Real-Time Ray Tracing. In High-Performance Graphics 2019 – Short Papers, Strasbourg, France, July 8-10, 2019, Markus Steinberger and Tim Foley (Eds.). Eurographics Association, 21–26. Google ScholarDigital Library
56. Thomas Müller. 2021. Tiny CUDA Neural Network Framework. https://github.com/nvlabs/tiny-cuda-nn.Google Scholar
57. Thomas Müller, Markus Gross, and Jan Novák. 2017. Practical Path Guiding for Efficient Light-Transport Simulation. Computer Graphics Forum 36, 4 (June 2017), 91–100. Google ScholarDigital Library
58. Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural Importance Sampling. ACM Trans. Graph. 38, 5, Article 145 (Oct. 2019), 19 pages. Google ScholarDigital Library
59. Thomas Müller, Fabrice Rousselle, Alexander Keller, and Jan Novák. 2020. Neural Control Variates. ACM Trans. Graph. 39, 6, Article 243 (Nov. 2020), 19 pages. Google ScholarDigital Library
60. Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, and Tobias Ritschel. 2017. Deep Shading: Convolutional Neural Networks for Screen-Space Shading. 36, 4 (2017).Google Scholar
61. Christopher Oat. 2005. Irradiance Volumes for Games. In Game Developers Conference.Google Scholar
62. Jacopo Pantaleoni. 2020. Online path sampling control with progressive spatio-temporal filtering. arXiv:2005.07547 (May 2020).Google Scholar
63. Hauke Rehfeld, Tobias Zirr, and Carsten Dachsbacher. 2014. Clustered Pre-Convolved Radiance Caching. In Proceedings of the 14th Eurographics Symposium on Parallel Graphics and Visualization (PGV ’14). Eurographics Association, Goslar, DEU, 25–32.Google ScholarDigital Library
64. Peiran Ren, Jiaping Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013. Global Illumination with Radiance Regression Functions. ACM Trans. Graph. 32, 4, Article 130 (July 2013), 12 pages. Google ScholarDigital Library
65. Tobias Ritschel, Carsten Dachsbacher, Thorsten Grosch, and Jan Kautz. 2012. The State of the Art in Interactive Global Illumination. Computer Graphics Forum 31, 1 (2012), 160–188. Google ScholarDigital Library
66. Tobias Ritschel, Thomas Engelhardt, Thorsten Grosch, Hans-Peter Seidel, Jan Kautz, and Carsten Dachsbacher. 2009a. Micro-Rendering for Scalable, Parallel Final Gathering. ACM Trans. Graph. 28, 5 (Dec. 2009), 1–8. Google ScholarDigital Library
67. Tobias Ritschel, Thorsten Grosch, Min H. Kim, Hans-Peter Seidel, Carsten Dachsbacher, and Jan Kautz. 2008. Imperfect Shadow Maps for Efficient Computation of Indirect Illumination. ACM Trans. Graph. 27, 5, Article 129 (Dec. 2008), 8 pages. Google ScholarDigital Library
68. Tobias Ritschel, Thorsten Grosch, and Hans-Peter Seidel. 2009b. Approximating Dynamic Global Illumination in Image Space. In Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games (I3D ’09). Association for Computing Machinery, New York, NY, USA, 75–82. Google ScholarDigital Library
69. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2011. Adaptive Sampling and Reconstruction Using Greedy Error Minimization. ACM Trans. Graph. 30, 6, Article 159 (Dec. 2011), 12 pages. Google ScholarDigital Library
70. Daniel Scherzer, Chuong H. Nguyen, Tobias Ritschel, and Hans-Peter Seidel. 2012. Pre-Convolved Radiance Caching. Comput. Graph. Forum 31, 4 (June 2012), 1391–1397. Google ScholarDigital Library
71. Dario Seyb, Peter-Pike Sloan, Ari Silvennoinen, Michał Iwanicki, and Wojciech Jarosz. 2020. The design and evolution of the UberBake light baking system. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020).Google ScholarDigital Library
72. Ari Silvennoinen and Jaakko Lehtinen. 2017. Real-Time Global Illumination by Pre-computed Local Reconstruction from Sparse Radiance Probes. ACM Trans. Graph. 36, 6, Article 230 (Nov. 2017), 13 pages. Google ScholarDigital Library
73. Peter-Pike Sloan, Jan Kautz, and John Snyder. 2002. Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Lighting Environments. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’02). Association for Computing Machinery, New York, NY, USA, 527–536. Google ScholarDigital Library
74. Tiago Sousa, Nickolay Kasyan, and Nicolas Schulz. 2011. Secrets of CryENGINE 3 Graphics Technology. In Advances in Real-Time Rendering in Games: Part I. ACM SIGGRAPH 2011 Courses.Google Scholar
75. Eric Tabellion and Arnauld Lamorlette. 2004. An Approximate Global Illumination System for Computer Generated Films. In ACM SIGGRAPH 2004 Papers (SIGGRAPH ’04). Association for Computing Machinery, New York, NY, USA, 469–476. Google ScholarDigital Library
76. Sergios Theodoridis. 2008. Pattern Recognition. Elsevier.Google Scholar
77. Kostas Vardis, Georgios Papaioannou, and Anastasios Gkaravelis. 2014. Real-time Radiance Caching using Chrominance Compression. Journal of Computer Graphics Techniques (JCGT) 3, 4 (16 December 2014), 111–131. http://jcgt.org/published/0003/04/06/Google Scholar
78. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.03762 (June 2017).Google Scholar
79. Eric Veach and Leonidas J. Guibas. 1995. Optimally Combining Sampling Techniques for Monte Carlo Rendering. In Proc. SIGGRAPH. 419–428. Google ScholarDigital Library
80. Jiří Vorba, Johannes Hanika, Sebastian Herholz, Thomas Müller, Jaroslav Křivánek, and Alexander Keller. 2019. Path Guiding in Production. In ACM SIGGRAPH Courses. ACM, New York, NY, USA, 18:1–18:77. Google ScholarDigital Library
81. Jiří Vorba, Ondřej Karlík, Martin Šik, Tobias Ritschel, and Jaroslav Křivánek. 2014. On-line Learning of Parametric Mixture Models for Light Transport Simulation. ACM Trans. Graph. 33, 4 (Aug. 2014).Google ScholarDigital Library
82. Gregory J. Ward, Francis M. Rubinstein, and Robert D. Clear. 1988. A Ray Tracing Solution for Diffuse Interreflection. In Proceedings of the 15th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’88). Association for Computing Machinery, New York, NY, USA, 85–92. Google ScholarDigital Library
83. Yangyang Zhao, Laurent Belcour, and Derek Nowrouzezahrai. 2019. View-Dependent Radiance Caching. In Proceedings of the 45th Graphics Interface Conference on Proceedings of Graphics Interface 2019 (GI’19). Canadian Human-Computer Communications Society, Waterloo, CAN, Article 22, 9 pages. Google ScholarDigital Library