“Unbiased warped-area sampling for differentiable rendering” by Bangaru, Li and Durand
Conference:
Type(s):
Title:
- Unbiased warped-area sampling for differentiable rendering
Session/Category Title: Light transport: Sampling
Presenter(s)/Author(s):
Abstract:
Differentiable rendering computes derivatives of the light transport equation with respect to arbitrary 3D scene parameters, and enables various applications in inverse rendering and machine learning. We present an unbiased and efficient differentiable rendering algorithm that does not require explicit boundary sampling. We apply the divergence theorem to the derivative of the rendering integral to convert the boundary integral into an area integral. We rewrite the converted area integral to a form that is suitable for Monte Carlo rendering. We then develop an efficient Monte Carlo sampling algorithm for solving the area integral. Our method can be easily plugged into a traditional path tracer and does not require dedicated data structures for sampling boundaries.We analyze the convergence properties through bias-variance metrics, and demonstrate our estimator’s advantages over existing methods for some synthetic inverse rendering examples.
References:
1. Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance modeling by neural texture synthesis. ACM Trans. Graph. (Proc. SIGGRAPH) 35, 4 (2016), 65:1–65:13.Google ScholarDigital Library
2. James Arvo. 1994. The Irradiance Jacobian for Partially Occluded Polyhedral Sources. In SIGGRAPH. ACM Press/Addison-Wesley Publishing Co., 343–350.Google Scholar
3. James Arvo. 1995. Applications of Irradiance Tensors to the Simulation of non-Lambertian Phenomena. In SIGGRAPH. 335–342.Google Scholar
4. Alex Beatson and Ryan P Adams. 2019. Efficient optimization of loops and limits with randomized telescoping sums. In International Conference on Machine Learning.Google Scholar
5. Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In SIGGRAPH. ACM Press/Addison-Wesley Publishing Co., 187–194.Google Scholar
6. Thomas E Booth. 2007. Unbiased Monte Carlo estimation of the reciprocal of an integral. Nuclear science and engineering 156, 3 (2007), 403–407.Google Scholar
7. Min Chen and James Arvo. 2000. Theory and application of specular path perturbation. ACM Trans. Graph. 19, 4 (2000), 246–278.Google ScholarDigital Library
8. Martin de La Gorce, David J Fleet, and Nikos Paragios. 2011. Model-based 3D hand pose estimation from monocular video. IEEE Trans. Pattern Anal. Mach. Intell. 33, 9 (2011), 1793–1805.Google ScholarDigital Library
9. Harley Flanders. 1973. Differentiation under the integral sign. The American Mathematical Monthly 80, 6 (1973), 615–627.Google ScholarCross Ref
10. Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T. Freeman. 2018. Unsupervised Training for 3D Morphable Model Regression. In Computer Vision and Pattern Recognition.Google Scholar
11. Johannes Hanika, Anton Kaplanyan, and Carsten Dachsbacher. 2015. Improved Half Vector Space Light Transport. Comput. Graph. Forum (Proc. EGSR) 34, 4 (2015), 65–74.Google ScholarDigital Library
12. Aaron Hertzmann and Denis Zorin. 2000. Illustrating smooth surfaces. In SIGGRAPH. ACM Press/Addison-Wesley Publishing Co., 517–526.Google ScholarDigital Library
13. Wenzel Jakob and Steve Marschner. 2012. Manifold exploration: a Markov Chain Monte Carlo technique for rendering scenes with difficult specular transport. ACM Trans. Graph. 31, 4 (2012), 58:1–58:13.Google ScholarDigital Library
14. André Jalobeanu, Frank O Kuehnel, and John C Stutz. 2004. Modeling images of natural 3d surfaces: Overview and potential applications. In Conference on Computer Vision and Pattern Recognition Workshop. IEEE, 188–188.Google ScholarCross Ref
15. James T. Kajiya. 1986. The Rendering Equation. Comput. Graph. (Proc. SIGGRAPH) 20, 4 (1986), 143–150.Google ScholarDigital Library
16. Anton S Kaplanyan, Johannes Hanika, and Carsten Dachsbacher. 2014. The natural-constraint representation of the path space for efficient light transport simulation. ACM Trans. Graph. (Proc. SIGGRAPH) 33, 4 (2014), 102:1–102:13.Google ScholarDigital Library
17. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In Computer Vision and Pattern Recognition. IEEE, 3907–3916.Google Scholar
18. Tzu-Mao Li. 2019. Differentiable Visual Computing. Ph.D. Dissertation. Massachusetts Institute of Technology. Advisor(s) Durand, Frédo.Google Scholar
19. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1–222:11.Google Scholar
20. Tzu-Mao Li, Jaakko Lehtinen, Ravi Ramamoorthi, Wenzel Jakob, and Frédo Durand. 2015. Anisotropic Gaussian Mutations for Metropolis Light Transport through Hessian-Hamiltonian Dynamics. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 34, 6 (2015), 209:1–209:13.Google Scholar
21. Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2019. Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image. arXiv preprint arXiv:1905.02722 (2019).Google Scholar
22. Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. 2017. Material Editing Using a Physically Based Rendering Network. In International Conference on Computer Vision. IEEE, 2280–2288.Google ScholarCross Ref
23. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. International Conference on Computer Vision (2019).Google ScholarCross Ref
24. Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In European Conference on Computer Vision, Vol. 8695. ACM, 154–169.Google Scholar
25. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (2019), 228.Google Scholar
26. Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2020. Langevin Monte Carlo Rendering with Gradient-based Adaptation. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4 (2020).Google ScholarDigital Library
27. Anne-Marie Lyne, Mark Girolami, Yves Atchadé, Heiko Strathmann, Daniel Simpson, et al. 2015. On Russian roulette estimates for Bayesian inference with doubly-intractable likelihoods. Statistical science 30, 4 (2015), 443–467.Google Scholar
28. J.R. Magnus and H. Neudecker. 1999. Matrix Differential Calculus with Applications in Statistics and Econometrics. Wiley. https://books.google.com/books?id=0CXXdKKiIpQCGoogle Scholar
29. Don McLeish. 2010. A general method for debiasing a Monte Carlo estimator. Monte Carlo Methods and Applications 17 (2010), 301 — 315.Google Scholar
30. Don Mitchell and Pat Hanrahan. 1992. Illumination from curved reflectors. (1992), 283–291.Google Scholar
31. Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4 (July 2020). Google ScholarDigital Library
32. Matt Olson and Hao Zhang. 2006. Silhouette extraction in Hough space. Comput. Graph. Forum (Proc. Eurographics) 25, 3 (2006), 273–282.Google ScholarCross Ref
33. Art B. Owen. 2013. Monte Carlo theory, methods and examples.Google Scholar
34. Gustavo Patow and Xavier Pueyo. 2003. A survey of inverse rendering problems. Comput. Graph. Forum 22, 4 (2003), 663–687.Google ScholarCross Ref
35. Hao Qin, Xin Sun, Qiming Hou, Baining Guo, and Kun Zhou. 2015. Unbiased Photon Gathering for Light Transport Simulation. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 34, 6 (2015).Google Scholar
36. Ravi Ramamoorthi, Dhruv Mahajan, and Peter Belhumeur. 2007. A First-order Analysis of Lighting, Shading, and Shadows. ACM Trans. Graph. 26, 1 (2007), 2.Google ScholarDigital Library
37. Osborne Reynolds, Arthur William Brightmore, and William Henry Moorby. 1903. The sub-mechanics of the universe. Vol. 3. University Press.Google Scholar
38. Helge Rhodin, Nadia Robertini, Christian Richardt, Hans-Peter Seidel, and Christian Theobalt. 2015. A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation. In International Conference on Computer Vision. IEEE, 765–773.Google ScholarDigital Library
39. Damien Rioux-Lavoie, Joey Litalien, Adrien Gruson, Toshiya Hachisuka, and Derek Nowrouzezahrai. 2020. Delayed Rejection Metropolis Light Transport. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 3 (April 2020). Google ScholarDigital Library
40. Maxime Roger, Stéphane Blanco, Mouna El Hafi, and Richard Fournier. 2005. Monte Carlo estimates of domain-deformation sensitivities. Physical review letters 95, 18 (2005), 180601.Google Scholar
41. Pedro V. Sander, Xianfeng Gu, Steven J. Gortler, Hugues Hoppe, and John Snyder. 2000. Silhouette Clipping. In SIGGRAPH. ACM Press/Addison-Wesley Publishing Co., 327–334.Google ScholarDigital Library
42. Mikio Shinya, T. Takahashi, and Seiichiro Naito. 1987. Principles and Applications of Pencil Tracing. Comput. Graph. (Proc. SIGGRAPH) 21, 4 (1987), 45–54.Google ScholarDigital Library
43. Vadim N Smelyansky, Robin D Morris, Frank O Kuehnel, David A Maluf, and Peter Cheeseman. 2002. Dramatic improvements to feature based stereo. In European Conference on Computer Vision. Springer, 247–261.Google ScholarCross Ref
44. Eric Veach. 1998. Robust Monte Carlo Methods for Light Transport Simulation. Ph.D. Dissertation. Stanford University. Advisor(s) Guibas, Leonidas J.Google ScholarDigital Library
45. Ingo Wald, Sven Woop, Carsten Benthin, Gregory S Johnson, and Manfred Ernst. 2014. Embree: a kernel framework for efficient CPU ray tracing. ACM Trans. Graph. (Proc. SIGGRAPH) 33, 4 (2014), 143:1–143:8.Google ScholarDigital Library
46. Greg Ward and Paul Heckbert. 1992. Irradiance Gradients. In Eurographics Workshop on Rendering. Eurographics Association, 85–98.Google Scholar
47. Lifan Wu, Guangyan Cai, Shuang Zhao, and Ravi Ramamoorthi. 2020. Analytic spherical harmonic gradients for real-time rendering with many polygonal area lights. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4 (2020).Google ScholarDigital Library
48. Tizian Zeltner, Iliyan Georgiev, and Wenzel Jakob. 2020. Specular manifold sampling for rendering high-frequency caustics and glints. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4 (2020), 149.Google ScholarDigital Library
49. Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 6 (2020), 143:1–143:19.Google ScholarDigital Library
50. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A differential theory of radiative transfer. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (2019), 227.Google Scholar
