“Monte Carlo estimators for differential light transport” by Zeltner, Speierer, Georgiev and Jakob

Physically based differentiable rendering algorithms propagate derivatives through realistic light transport simulations and have applications in diverse areas including inverse reconstruction and machine learning. Recent progress has led to unbiased methods that can simultaneously compute derivatives with respect to millions of parameters. At the same time, elementary properties of these methods remain poorly understood.Current algorithms for differentiable rendering are constructed by mechanically differentiating a given primal algorithm. While convenient, such an approach is simplistic because it leaves no room for improvement. Differentiation produces major changes in the integrals that occur throughout the rendering process, which indicates that the primal and differential algorithms should be decoupled so that the latter can suitably adapt.This leads to a large space of possibilities: consider that even the most basic Monte Carlo path tracer already involves several design choices concerning the techniques for sampling materials and emitters, and their combination, e.g. via multiple importance sampling (MIS). Differentiation causes a veritable explosion of this decision tree: should we differentiate only the estimator, or also the sampling technique? Should MIS be applied before or after differentiation? Are specialized derivative sampling strategies of any use? How should visibility-related discontinuities be handled when millions of parameters are differentiated simultaneously? In this paper, we provide a taxonomy and analysis of different estimators for differential light transport to provide intuition about these and related questions.

1. Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In Proceedings of Computer Vision and Pattern Recognition (CVPR), IEEE.Google ScholarCross Ref
2. Sai Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased Warped-Area Sampling for Differentiable Rendering. ACM Transactions on Graphics 39, 6 (2020), 245:1–245:18.Google ScholarDigital Library
3. Petr Beckmann and Andre Spizzichino. 1987. The scattering of electromagnetic waves from rough surfaces. Norwood (1987).Google Scholar
4. Laurent Belcour, Cyril Soler, Kartic Subr, Nicolas Holzschuch, and Fredo Durand. 2013. 5D Covariance Tracing for Efficient Defocus and Motion Blur. ACM Transactions on Graphics 32, 3 (July 2013). Google ScholarDigital Library
5. Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.Google Scholar
6. Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2018. Inverse Transport Networks. arXiv preprint arXiv:1809.10820 (2018).Google Scholar
7. Min Chen and James Arvo. 2000. Theory and Application of Specular Path Perturbation. ACM Transactions on Graphics 19, 4 (Oct. 2000), 246–278.Google ScholarDigital Library
8. Robert L Cook and Kenneth E. Torrance. 1982. A reflectance model for computer graphics. ACM Transactions on Graphics (ToG) 1, 1 (1982), 7–24.Google ScholarDigital Library
9. Luc Devroye. 1986. Non-Uniform Random Variate Generation. Springer-Verlag.Google Scholar
10. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An evaluation of computational imaging techniques for heterogeneous inverse scattering. In European Conference on Computer Vision. Springer, 685–701.Google ScholarCross Ref
11. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse Volume Rendering with Material Dictionaries. ACM Transactions on Graphics 32, 6, Article 162 (Nov. 2013).Google ScholarDigital Library
12. Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. Vol. 105. SIAM.Google Scholar
13. Carole K. Hayakawa, Jerome Spanier, Frédéric Bevilacqua, Andrew K. Dunn, Joon S. You, Bruce J. Tromberg, and Vasan Venugopalan. 2001. Perturbation Monte Carlo methods to solve inverse photon migration problems in heterogeneous tissues. Opt. Lett. 26, 17 (Sep 2001), 1335–1337.Google ScholarCross Ref
14. Paul S Heckbert. 1989. Fundamentals of texture mapping and image warping. Master’s thesis.Google Scholar
15. Eric Heitz and Eugene D’Eon. 2014. Importance Sampling Microfacet-Based BSDFs using the Distribution of Visible Normals. Computer Graphics Forum 33, 4 (July 2014), 103–112. Google ScholarDigital Library
16. Binh-Son Hua, Adrien Gruson, Victor Petitjean, Matthias Zwicker, Derek Nowrouzezahrai, Elmar Eisemann, and Toshiya Hachisuka. 2019. A Survey on Gradient-Domain Rendering. Computer Graphics Forum 38, 2 (2019), 455–472.Google ScholarCross Ref
17. Homan Igehy. 1999. Tracing Ray Differentials. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 99). 179–186. Google ScholarDigital Library
18. Wenzel Jakob and Steve Marschner. 2012. Manifold Exploration: A Markov Chain Monte Carlo Technique for Rendering Scenes with Difficult Specular Transport. ACM Transactions on Graphics 31, 4 (July 2012).Google ScholarDigital Library
19. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
20. Jaroslav Krivánek, Pascal Gautron, Sumanta Pattanaik, and Kadi Bouatouch. 2005. Radiance caching for efficient global illumination computation. IEEE Transactions on Visualization and Computer Graphics 11, 5 (2005), 550–561.Google ScholarDigital Library
21. Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular Primitives for High-Performance Differentiable Rendering. ACM Transactions on Graphics 39, 6 (2020).Google ScholarDigital Library
22. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1–222:11.Google Scholar
23. Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. 2019. Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction. CoRR abs/1901.05567 (2019). arXiv:1901.05567 http://arxiv.org/abs/1901.05567Google Scholar
24. Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In European Conference on Computer Vision. Springer.Google ScholarCross Ref
25. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38, 6 (Dec. 2019).Google ScholarDigital Library
26. Iván Lux and Lázló Koblinger. 1990. Monte Carlo Particle Transport Methods: Neutron and Photon Calculations. CRC Press, Boston.Google Scholar
27. Don Mitchell and Pat Hanrahan. 1992. Illumination from curved reflectors. In Proceedings of the 19th annual conference on Computer graphics and interactive techniques. 283–291.Google ScholarDigital Library
28. Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020).Google Scholar
29. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38, 6 (Dec. 2019).Google ScholarDigital Library
30. Art B. Owen. 2013. Monte Carlo theory, methods and examples. https://statweb.stanford.edu/~owen/mc/Google Scholar
31. Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich. 2010. OptiX: A General Purpose Ray Tracing Engine. ACM Transactions on Graphics 29, 4, Article 66 (July 2010), 13 pages. Google ScholarDigital Library
32. Felix Petersen, Amit H. Bermano, Oliver Deussen, and Daniel Cohen-Or. 2019. Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer. CoRR abs/1903.11149 (2019). arXiv:1903.11149 http://arxiv.org/abs/1903.11149Google Scholar
33. Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically Based Rendering: From Theory to Implementation (3rd ed.) (3rd ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. 1266 pages.Google ScholarDigital Library
34. Ravi Ramamoorthi, Dhruv Mahajan, and Peter Belhumeur. 2007. A first-order analysis of lighting, shading, and shadows. ACM Transactions on Graphics (TOG) 26, 1 (2007).Google ScholarDigital Library
35. Helge Rhodin, Nadia Robertini, Christian Richardt, Hans-Peter Seidel, and Christian Theobalt. 2015. A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation. In Proceedings of ICCV 2015.Google ScholarDigital Library
36. Kenneth E Torrance and Ephraim M Sparrow. 1967. Theory for off-specular reflection from roughened surfaces. Josa 57, 9 (1967), 1105–1114.Google ScholarCross Ref
37. T. S. Trowbridge and K. P. Reitz. 1975. J. Opt. Soc. Am. 65, 5 (May 1975), 531–536.Google ScholarCross Ref
38. Eric Veach and Leonidas J. Guibas. 1995. Optimally Combining Sampling Techniques for Monte Carlo Rendering. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’95). Association for Computing Machinery, New York, NY, USA, 419–428. Google ScholarDigital Library
39. Darko Veberic. 2010. Having Fun with Lambert W(x) Function. CoRR abs/1003.1628 (2010). arXiv:1003.1628 http://arxiv.org/abs/1003.1628Google Scholar
40. Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021. Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time. Transactions on Graphics (Proceedings of SIGGRAPH) 40, 4 (Aug. 2021). Google ScholarDigital Library
41. Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (Grenoble, France) (EGSR’07). Eurographics Association, Goslar, DEU, 195–206.Google Scholar
42. G J Ward and P S Heckbert. 1992. Irradiance gradients. Technical Report. LawrenceGoogle Scholar
43. Berkeley Lab., CA (United States); Ecole Polytechnique Federale, Lausanne (Switzerland); Technische Hogeschool Delft (Netherlands). Dept. of Technical Mathematics and Informatics.Google Scholar
44. Tizian Zeltner, Iliyan Georgiev, and Wenzel Jakob. 2020. Specular Manifold Sampling for Rendering High-Frequency Caustics and Glints. Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020). Google ScholarDigital Library
45. Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM Transactions on Graphics 39, 4 (2020), 143:1–143:19.Google ScholarDigital Library
46. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A Differential Theory of Radiative Transfer. ACM Transactions on Graphics 38, 6 (2019), 227:1–227:16.Google ScholarDigital Library
47. Shaung Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling Scattering Parameters for Rendering Anisotropic Media. ACM Transactions on Graphics 35, 6 (2016).Google ScholarDigital Library