“Langevin monte carlo rendering with gradient-based adaptation” by Luan, Zhao, Bala and Gkioulekas

  • ©Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas



Session Title:

    Differentiable Rendering and Applications


    Langevin monte carlo rendering with gradient-based adaptation



    We introduce a suite of Langevin Monte Carlo algorithms for efficient photorealistic rendering of scenes with complex light transport effects, such as caustics, interreflections, and occlusions. Our algorithms operate in primary sample space, and use the Metropolis-adjusted Langevin algorithm (MALA) to generate new samples. Drawing inspiration from state-of-the-art stochastic gradient descent procedures, we combine MALA with adaptive preconditioning and momentum schemes that re-use previously-computed first-order gradients, either in an online or in a cache-driven fashion. This combination allows MALA to adapt to the local geometry of the primary sample space, without the computational overhead associated with previous Hessian-based adaptation algorithms. We use the theory of controlled Markov chain Monte Carlo to ensure that these combinations remain ergodic, and are therefore suitable for unbiased Monte Carlo rendering. Through extensive experiments, we show that our algorithms, MALA with online and cache-driven adaptation, can successfully handle complex light transport in a large variety of scenes, leading to improved performance (on average more than 3× variance reduction at equal time, and 7× for motion blur) compared to state-of-the-art Markov chain Monte Carlo rendering algorithms.


    1. Luke Anderson, Tzu-Mao Li, Jaakko Lehtinen, and Frédo Durand. 2017. Aether: An Embedded Domain Specific Sampling Language for Monte Carlo Rendering. ACM TOG (July 2017).Google ScholarDigital Library
    2. Christophe Andrieu and Johannes Thoms. 2008. A tutorial on adaptive MCMC. Statistics and computing (2008).Google Scholar
    3. James Arvo. 1986. Backward ray tracing. ACM SIGGRAPH Course Notes (1986).Google Scholar
    4. James Arvo. 1994. The irradiance Jacobian for partially occluded polyhedral sources. SIGGRAPH (1994).Google Scholar
    5. Yves F Atchade. 2006. An adaptive version for the Metropolis adjusted Langevin algorithm with a truncated drift. MCAP (2006).Google Scholar
    6. Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. CVPR (2019).Google Scholar
    7. Bradley M Bell. 2003-2015. Cppad: A package for differentiation of C++ algorithms.Google Scholar
    8. Michael Betancourt. 2013. A general metric for Riemannian manifold Hamiltonian Monte Carlo. International Conference on Geometric Science of Information (2013).Google ScholarCross Ref
    9. Michael Betancourt. 2017. A conceptual introduction to Hamiltonian Monte Carlo. arXiv preprint arXiv:1701.02434 (2017).Google Scholar
    10. Benedikt Bitterli, Wenzel Jakob, Jan Novák, and Wojciech Jarosz. 2018. Reversible jump metropolis light transport using inverse mappings. ACM TOG (2018).Google Scholar
    11. Benedikt Bitterli and Wojciech Jarosz. 2019. Selectively metropolised Monte Carlo light transport simulation. ACM TOG (2019).Google Scholar
    12. Jose Luis Blanco and Pranjal Kumar Rai. 2014. nanoflann: a C++ header-only fork of FLANN, a library for Nearest Neighbor (NN) with KD-trees.Google Scholar
    13. Ken W Brodlie, AR Gourlay, and John Greenstadt. 1973. Rank-one and rank-two corrections to positive definite matrices expressed in product form. IMA Journal of Applied Mathematics (1973).Google Scholar
    14. Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. 2011. Handbook of markov chain monte carlo. CRC press.Google Scholar
    15. Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2018. Inverse Transport Networks. arXiv preprint arXiv:1809.10820 (2018).Google Scholar
    16. Changyou Chen, David Carlson, Zhe Gan, Chunyuan Li, and Lawrence Carin. 2016. Bridging the gap between stochastic gradient MCMC and stochastic optimization. Artificial Intelligence and Statistics (2016).Google Scholar
    17. Min Chen and James Arvo. 2000. Theory and application of specular path perturbation. ACM TOG (2000).Google Scholar
    18. Tianqi Chen, Emily Fox, and Carlos Guestrin. 2014. Stochastic gradient hamiltonian monte carlo. ICML (2014).Google Scholar
    19. Simon Duane, Anthony D Kennedy, Brian J Pendleton, and Duncan Roweth. 1987. Hybrid monte carlo. Physics letters B (1987).Google Scholar
    20. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. JMLR (2011).Google ScholarDigital Library
    21. Alain Durmus, Umut Simsekli, Eric Moulines, Roland Badeau, and Gaël Richard. 2016. Stochastic gradient Richardson-Romberg Markov chain Monte Carlo. NeurIPS (2016).Google Scholar
    22. Philip Dutre, Philippe Bekaert, and Kavita Bala. 2006. Advanced global illumination. AK Peters/CRC Press.Google Scholar
    23. Philip Dutré, Eric P. Lafortune, and Yves D. Willems. 1993. Monte Carlo Light Tracing with Direct Computation of Pixel Intensities. CGVT (1993).Google Scholar
    24. Mark Girolami and Ben Calderhead. 2011. Riemann manifold langevin and hamiltonian monte carlo methods. JRSS (2011).Google Scholar
    25. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An evaluation of computational imaging techniques for heterogeneous inverse scattering. ECCV (2016).Google Scholar
    26. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse volume rendering with material dictionaries. ACM TOG (2013).Google Scholar
    27. Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. Siam.Google Scholar
    28. Toshiya Hachisuka and Henrik Wann Jensen. 2009. Stochastic progressive photon mapping. ACM TOG (2009).Google Scholar
    29. Toshiya Hachisuka and Henrik Wann Jensen. 2011. Robust adaptive photon tracing using photon path visibility. ACM TOG (2011).Google Scholar
    30. Toshiya Hachisuka, Anton S Kaplanyan, and Carsten Dachsbacher. 2014. Multiplexed metropolis light transport. ACM TOG (2014).Google Scholar
    31. Johannes Hanika, Anton Kaplanyan, and Carsten Dachsbacher. 2015. Improved half vector space light transport. In CGF.Google Scholar
    32. W Keith Hastings. 1970. Monte Carlo sampling methods using Markov chains and their applications. (1970).Google Scholar
    33. Nicolas Holzschuch and François X Sillion. 1995. Accurate computation of the radiosity gradient with constant and linear emitters. EGSR (1995).Google Scholar
    34. Homan Igehy. 1999. Tracing ray differentials. SIGGRAPH (1999).Google Scholar
    35. Wenzel Jakob. 2019. Enoki: structured vectorization and differentiation on modern processor architectures. https://github.com/mitsuba-renderer/enoki.Google Scholar
    36. Wenzel Jakob and Steve Marschner. 2012. Manifold exploration: a Markov Chain Monte Carlo technique for rendering scenes with difficult specular transport. ACM TOG (2012).Google ScholarDigital Library
    37. Wojciech Jarosz, Craig Donner, Matthias Zwicker, and Henrik Wann Jensen. 2008a. Radiance caching for participating media. ACM TOG (2008).Google Scholar
    38. Wojciech Jarosz, Derek Nowrouzezahrai, Robert Thomas, Peter-Pike Sloan, and Matthias Zwicker. 2011. Progressive Photon Beams. ACM TOG (2011).Google Scholar
    39. Wojciech Jarosz, Volker Schönefeld, Leif Kobbelt, and Henrik Wann Jensen. 2012. Theory, Analysis and Applications of 2D Global Illumination. ACM TOG (2012).Google Scholar
    40. Wojciech Jarosz, Matthias Zwicker, and Henrik Wann Jensen. 2008b. Irradiance Gradients in the Presence of Participating Media and Occlusions. EGSR (2008).Google Scholar
    41. Henrik Wann Jensen. 1995. Importance driven path tracing using the photon map. Rendering Techniques (1995).Google Scholar
    42. Henrik Wann Jensen. 2001. Realistic image synthesis using photon mapping. AK Peters/CRC Press.Google Scholar
    43. James T Kajiya. 1986. The rendering equation. SIGGRAPH (1986).Google Scholar
    44. Anton S Kaplanyan, Johannes Hanika, and Carsten Dachsbacher. 2014. The natural-constraint representation of the path space for efficient light transport simulation. ACM TOG (2014).Google Scholar
    45. Csaba Kelemen, László Szirmay-Kalos, György Antal, and Ferenc Csonka. 2002. A simple and robust mutation strategy for the metropolis light transport algorithm. CGF (2002).Google Scholar
    46. Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching Real Fabrics with Micro-Appearance Models. ACM TOG (2015).Google Scholar
    47. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
    48. Jaroslav Krivánek, Kadi Bouatouch, Sumanta N Pattanaik, and Jiri Zara. 2006. Making Radiance and Irradiance Caching Practical: Adaptive Caching and Neighbor Clamping. Rendering Techniques (2006).Google Scholar
    49. Jaroslav Krivanek, Pascal Gautron, Sumanta Pattanaik, and Kadi Bouatouch. 2005. Radiance caching for efficient global illumination computation. IEEE TVCG (2005).Google ScholarDigital Library
    50. Jaroslav Křivánek, Iliyan Georgiev, Toshiya Hachisuka, Petr Vévoda, Martin Šik, Derek Nowrouzezahrai, and Wojciech Jarosz. 2014. Unifying Points, Beams, and Paths in Volumetric Light Transport Simulation. ACM TOG (2014).Google Scholar
    51. Eric P. Lafortune and Yves D. Willems. 1993. Bi-directional path tracing. Compugraphics (1993).Google Scholar
    52. Jaakko Lehtinen, Tero Karras, Samuli Laine, Miika Aittala, Frédo Durand, and Timo Aila. 2013. Gradient-domain metropolis light transport. ACM TOG (2013).Google Scholar
    53. Don Lemons and Anthony Gythiel. 1997. Paul Langevin’s 1908 paper “On the Theory of Brownian Motion”. American Journal of Physics (1997).Google Scholar
    54. Tzu-Mao Li. 2015. dpt. https://github.com/BachiLi/dpt/.Google Scholar
    55. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable monte carlo ray tracing through edge sampling. ACM TOG (2018).Google Scholar
    56. Tzu-Mao Li, Jaakko Lehtinen, Ravi Ramamoorthi, Wenzel Jakob, and Frédo Durand. 2015. Anisotropic Gaussian Mutations for Metropolis Light Transport through Hessian-Hamiltonian Dynamics. ACM TOG (2015).Google Scholar
    57. Samuel Livingstone and Giacomo Zanella. 2019. On the robustness of gradient-based MCMC algorithms. arXiv preprint arXiv:1908.11812 (2019).Google Scholar
    58. Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2020. Project Website. https://research.cs.cornell.edu/langevin-mcmc.Google Scholar
    59. Yi-An Ma, Tianqi Chen, and Emily Fox. 2015. A complete recipe for stochastic gradient MCMC. NeurIPS (2015).Google Scholar
    60. Julio Marco, Adrian Jarabo, Wojciech Jarosz, and Diego Gutierrez. 2018. Second-Order Occlusion-Aware Volumetric Radiance Caching. ACM TOG (2018).Google Scholar
    61. Gisiro Maruyama. 1955. Continuous Markov processes and stochastic equations. Rendiconti del Circolo Matematico di Palermo (1955).Google Scholar
    62. Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Teller. 1953. Equation of state calculations by fast computing machines. The journal of chemical physics (1953).Google ScholarCross Ref
    63. Thomas Müller, Markus Gross, and Jan Novák. 2017. Practical path guiding for efficient light-transport simulation. CGF (2017).Google Scholar
    64. Thomas Müller, Brian Mcwilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural Importance Sampling. ACM TOG (2019).Google Scholar
    65. Radford M Neal et al. 2011. MCMC using Hamiltonian dynamics. Handbook of markov chain monte carlo (2011).Google Scholar
    66. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: a retargetable forward and inverse renderer. ACM TOG (2019).Google ScholarDigital Library
    67. Jorge Nocedal and Stephen Wright. 2006. Numerical optimization. Springer.Google Scholar
    68. Bernt Øksendal. 2003. Stochastic differential equations. In Stochastic differential equations. Springer.Google Scholar
    69. Hisanari Otsu, Johannes Hanika, Toshiya Hachisuka, and Carsten Dachsbacher. 2018. Geometry-Aware Metropolis Light Transport. ACM TOG (2018).Google Scholar
    70. Hisanari Otsu, Anton S. Kaplanyan, Johannes Hanika, Carsten Dachsbacher, and Toshiya Hachisuka. 2017. Fusing State Spaces for Markov Chain Monte Carlo Rendering. ACM TOG (2017).Google Scholar
    71. Jacopo Pantaleoni. 2017. Charted Metropolis Light Transport. ACM TOG (2017).Google Scholar
    72. Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically based rendering: From theory to implementation. Morgan Kaufmann.Google Scholar
    73. Boris T Polyak. 1964. Some methods of speeding up the convergence of iteration methods. U. S. S. R. Comput. Math. and Math. Phys. (1964).Google Scholar
    74. Ravi Ramamoorthi, Dhruv Mahajan, and Peter Belhumeur. 2007. A First-Order Analysis of Lighting, Shading, and Shadows. ACM TOG (2007).Google Scholar
    75. Florian Reibold, Johannes Hanika, Alisa Jung, and Carsten Dachsbacher. 2019. Selective Guided Sampling with Complete Light Transport Paths. ACM TOG (2019).Google Scholar
    76. Gareth O Roberts and Jeffrey S Rosenthal. 2007. Coupling and ergodicity of adaptive Markov chain Monte Carlo algorithms. Journal of applied probability (2007).Google Scholar
    77. Gareth O Roberts and Jeffrey S Rosenthal. 2009. Examples of adaptive MCMC. Journal of Computational and Graphical Statistics (2009).Google Scholar
    78. Gareth O Roberts and Osnat Stramer. 2002. Langevin diffusions and Metropolis-Hastings algorithms. Methodology and computing in applied probability (2002).Google Scholar
    79. Gareth O Roberts and Richard L Tweedie. 1996. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli (1996).Google Scholar
    80. Jorge Schwarzhaupt, Henrik Wann Jensen, and Wojciech Jarosz. 2012. Practical Hessian-Based Error Control for Irradiance Caching. ACM TOG (2012).Google Scholar
    81. Martin Šik, Hisanari Otsu, Toshiya Hachisuka, and Jaroslav Křivánek. 2016. Robust light transport simulation via metropolised bidirectional estimators. ACM TOG (2016).Google Scholar
    82. Umut Simsekli, Roland Badeau, Taylan Cemgil, and Gaël Richard. 2016. Stochastic quasi-newton langevin monte carlo. ICML (2016).Google Scholar
    83. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the importance of initialization and momentum in deep learning. ICML (2013).Google Scholar
    84. Frank Suykens and Yves D Willems. 2001. Path differentials and applications. Rendering Techniques (2001).Google Scholar
    85. Chia-Yin Tsai, Aswin C Sankaranarayanan, and Ioannis Gkioulekas. 2019. Beyond Volumetric Albedo-A Surface Optimization Framework for Non-Line-Of-Sight Imaging. CVPR (2019).Google Scholar
    86. Eric Veach. 1997. Robust Monte Carlo methods for light transport simulation. Stanford University PhD thesis.Google ScholarDigital Library
    87. Eric Veach and Leonidas Guibas. 1995. Bidirectional estimators for light transport. Photorealistic Rendering Techniques (1995).Google Scholar
    88. Eric Veach and Leonidas J Guibas. 1997. Metropolis light transport. SIGGRAPH (1997).Google Scholar
    89. Jiří Vorba, Ondřej Karlík, Martin Šik, Tobias Ritschel, and Jaroslav Křivánek. 2014. On-Line Learning of Parametric Mixture Models for Light Transport Simulation. ACM TOG (2014).Google Scholar
    90. Gregory J Ward and Paul S Heckbert. 1992. Irradiance gradients. Technical Report.Google Scholar
    91. Gregory J Ward, Francis M Rubinstein, and Robert D Clear. 1988. A ray tracing solution for diffuse interreflection. SIGGRAPH (1988).Google Scholar
    92. Max Welling and Yee W Teh. 2011. Bayesian learning via stochastic gradient Langevin dynamics. ICML (2011).Google Scholar
    93. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A differential theory of radiative transfer. ACM TOG (2019).Google Scholar
    94. Yichuan Zhang and Charles A. Sutton. 2011. Quasi-Newton Methods for Markov Chain Monte Carlo. NeurIPS (2011).Google Scholar
    95. Shuang Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling scattering parameters for rendering anisotropic media. ACM TOG (2016).Google Scholar
    96. Quan Zheng and Matthias Zwicker. 2019. Learning to importance sample in primary sample space. CGF (2019).Google Scholar

ACM Digital Library Publication: