“Path-space differentiable rendering” by Zhang, Miller, Yan, Gkioulekas and Zhao

  • ©Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao



Session Title:

    Differentiable Rendering and Applications


    Path-space differentiable rendering



    Physics-based differentiable rendering, the estimation of derivatives of radiometric measures with respect to arbitrary scene parameters, has a diverse array of applications from solving analysis-by-synthesis problems to training machine learning pipelines incorporating forward rendering processes. Unfortunately, general-purpose differentiable rendering remains challenging due to the lack of efficient estimators as well as the need to identify and handle complex discontinuities such as visibility boundaries.In this paper, we show how path integrals can be differentiated with respect to arbitrary differentiable changes of a scene. We provide a detailed theoretical analysis of this process and establish new differentiable rendering formulations based on the resulting differential path integrals. Our path-space differentiable rendering formulation allows the design of new Monte Carlo estimators that offer significantly better efficiency than state-of-the-art methods in handling complex geometric discontinuities and light transport phenomena such as caustics.We validate our method by comparing our derivative estimates to those generated using the finite-difference method. To demonstrate the effectiveness of our technique, we compare inverse-rendering performance with a few state-of-the-art differentiable rendering methods.


    1. Luke Anderson, Tzu-Mao Li, Jaakko Lehtinen, and Frédo Durand. 2017. Aether: an embedded domain specific sampling language for Monte Carlo rendering. ACMGoogle Scholar
    2. Trans. Graph. 36, 4 (2017), 99:1–99:16.Google Scholar
    3. James Arvo. 1994. The Irradiance Jacobian for partially occluded polyhedral sources. In SIGGRAPH ’94. 343–350.Google ScholarDigital Library
    4. Dejan Azinovic, Tzu-Mao Li, Anton Kaplanyan, and Matthias Niessner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
    5. Paolo Cermelli, Eliot Fried, and Morton E Gurtin. 2005. Transport relations for surface integrals arising in the formulation of balance laws for evolving fluid interfaces. Journal of Fluid Mechanics 544 (2005), 339–351.Google ScholarCross Ref
    6. Subrahmanyan Chandrasekhar. 1960. Radiative Transfer. Courier Corporation.Google Scholar
    7. Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2018. Inverse transport networks. arXiv preprint arXiv:1809.10820 (2018).Google Scholar
    8. Min Chen and James Arvo. 2000. Theory and application of specular path perturbation. ACM Trans. Graph. 19, 4 (2000), 246–278.Google ScholarDigital Library
    9. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An Evaluation of Computational Imaging Techniques for Heterogeneous Inverse Scattering. In Computer Vision – ECCV 2016. Springer International Publishing, 685–701.Google ScholarCross Ref
    10. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse volume rendering with material dictionaries. ACM Trans. Graph. 32, 6 (2013), 162:1–162:13.Google ScholarDigital Library
    11. Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. Vol. 105. Siam.Google Scholar
    12. Pavel Grinfeld. 2013. Introduction to tensor analysis and the calculus of moving surfaces. Springer.Google Scholar
    13. Morton E Gurtin. 1981. An introduction to continuum mechanics. Academic press.Google Scholar
    14. Milovš Hašan and Ravi Ramamoorthi. 2013. Interactive albedo editing in path-traced volumetric materials. ACM Trans. Graph. 32, 2 (2013), 11:1–11:11.Google ScholarDigital Library
    15. Homan Igehy. 1999. Tracing ray differentials. In SIGGRAPH ’99. 179–186.Google ScholarDigital Library
    16. Wenzel Jakob and Steve Marschner. 2012. Manifold Exploration: A Markov Chain Monte Carlo Technique for Rendering Scenes with Difficult Specular Transport. ACM Trans. Graph. 31, 4 (2012), 58:1–58:13.Google ScholarDigital Library
    17. Henrik Wann Jensen. 1995. Importance driven path tracing using the photon map. In Rendering Techniques’ 95. Springer, 326–335.Google Scholar
    18. James T. Kajiya. 1986. The Rendering Equation. SIGGRAPH Comput. Graph. 20, 4 (1986), 143–150.Google ScholarDigital Library
    19. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D mesh renderer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3907–3916.Google ScholarCross Ref
    20. Csaba Kelemen, László Szirmay-Kalos, György Antal, and Ferenc Csonka. 2002. A simple and robust mutation strategy for the Metropolis light transport algorithm. In Computer Graphics Forum, Vol. 21. Wiley Online Library, 531–540.Google Scholar
    21. Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching real fabrics with micro-appearance models. ACM Trans. Graph. 35, 1 (2015), 1:1–1:26.Google ScholarDigital Library
    22. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
    23. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018a. Differentiable Monte Carlo ray tracing through edge sampling. ACM Trans. Graph. 37, 6 (2018), 222:1–222:11.Google ScholarDigital Library
    24. Tzu-Mao Li, Jaakko Lehtinen, Ravi Ramamoorthi, Wenzel Jakob, and Frédo Durand. 2015. Anisotropic Gaussian mutations for Metropolis light transport through Hessian-Hamiltonian dynamics. ACM Trans. Graph. 34, 6 (2015), 209:1–209:13.Google ScholarDigital Library
    25. Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018b. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans. Graph. 37, 6 (2018), 269:1–269:11.Google ScholarDigital Library
    26. Matthew M Loper and Michael J Black. 2014. OpenDR: an approximate differentiable renderer. In European Conference on Computer Vision. Springer, 154–169.Google ScholarCross Ref
    27. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. 38, 6 (2019).Google ScholarDigital Library
    28. James L McClelland, David E Rumelhart, PDP Research Group, et al. 1986. Parallel distributed processing. Explorations in the microstructure of cognition 2 (1986), 216–271.Google Scholar
    29. Abhimitra Meka, Maxim Maximov, Michael Zollhoefer, Avishek Chatterjee, Hans-Peter Seidel, Christian Richardt, and Christian Theobalt. 2018. Lime: Live intrinsic material estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6315–6324.Google ScholarCross Ref
    30. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: a retargetable forward and inverse renderer. ACM Transactions on Graphics (TOG) 38, 6 (2019), 203.Google ScholarDigital Library
    31. Mark Pauly, Thomas Kollig, and Alexander Keller. 2000. Metropolis light transport for participating media. In Rendering Techniques 2000. Springer, 11–22.Google ScholarCross Ref
    32. Ravi Ramamoorthi, Dhruv Mahajan, and Peter Belhumeur. 2007. A first-order analysis of lighting, shading, and shadows. ACM Trans. Graph. 26, 1 (2007), 2:1–2:21.Google ScholarDigital Library
    33. Osborne Reynolds. 1903. Papers on mechanical and physical subjects: the sub-mechanics of the universe. Vol. 3. The University Press.Google Scholar
    34. Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs. 2018. SfSNet: Learning shape, reflectance and illuminance of faces in the wild’. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6296–6305.Google ScholarCross Ref
    35. Denis Sumin, Tobias Rittig, Vahid Babaei, Thomas Nindel, Alexander Wilkie, Piotr Didyk, Bernd Bickel, Jaroslav Křivánek, Karol Myszkowski, and Tim Weyrich. 2019. Geometry-aware scattering compensation for 3D printing. ACM Trans. Graph. 38, 4 (2019), 111:1–111:14.Google ScholarDigital Library
    36. Chia-Yin Tsai, Aswin C. Sankaranarayanan, and Ioannis Gkioulekas. 2019. Beyond volumetric albedo—a surface optimization framework for non-line-of-sight imaging. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
    37. Eric Veach. 1997. Robust Monte Carlo methods for light transport simulation. Vol. 1610. Stanford University PhD thesis.Google ScholarDigital Library
    38. Eric Veach and Leonidas Guibas. 1995. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques. Springer, 145–167.Google Scholar
    39. Eric Veach and Leonidas J. Guibas. 1997. Metropolis Light Transport. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’97). ACM Press/Addison-Wesley Publishing Co., 65–76.Google Scholar
    40. Zdravko Velinov, Marios Papas, Derek Bradley, Paulo Gotardo, Parsa Mirdehghan, Steve Marschner, Jan Novák, and Thabo Beeler. 2018. Appearance Capture and Modeling of Human Teeth. ACM Trans. Graph. 37, 6 (2018), 207:1–207:13.Google ScholarDigital Library
    41. Robert Edwin Wengert. 1964. A simple automatic derivative evaluation program. Commun. ACM 7, 8 (1964), 463–464.Google ScholarDigital Library
    42. Andrew P. Witkin and Paul S. Heckbert. 1994. Using particles to sample and control implicit surfaces. In Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’94). ACM, 269–277.Google Scholar
    43. Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. 2017. Neural scene de-rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 699–707.Google ScholarCross Ref
    44. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shaung Zhao. 2019. A differential theory of radiative transfer. ACM Trans. Graph. 38, 6 (2019), 227:1–227:16.Google ScholarDigital Library
    45. Shuang Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling scattering parameters for rendering anisotropic media. ACM Trans. Graph. 35, 6 (2016), 166:1–166:11.Google ScholarDigital Library
    46. Changxi Zheng and Doug L. James. 2012. Energy-based self-collision culling for arbitrary mesh deformations. ACM Trans. Graph. 31, 4 (2012), 98:1–98:12.Google ScholarDigital Library

ACM Digital Library Publication: