“Reparameterizing discontinuous integrands for differentiable rendering” by Loubet, Holzschuch and Jakob
Conference:
Type(s):
Title:
- Reparameterizing discontinuous integrands for differentiable rendering
Session/Category Title: Differentiable Rendering
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Differentiable rendering has recently opened the door to a number of challenging inverse problems involving photorealistic images, such as computational material design and scattering-aware reconstruction of geometry and materials from photographs. Differentiable rendering algorithms strive to estimate partial derivatives of pixels in a rendered image with respect to scene parameters, which is difficult because visibility changes are inherently non-differentiable.We propose a new technique for differentiating path-traced images with respect to scene parameters that affect visibility, including the position of cameras, light sources, and vertices in triangle meshes. Our algorithm computes the gradients of illumination integrals by applying changes of variables that remove or strongly reduce the dependence of the position of discontinuities on differentiable scene parameters. The underlying parameterization is created on the fly for each integral and enables accurate gradient estimates using standard Monte Carlo sampling in conjunction with automatic differentiation. Importantly, our approach does not rely on sampling silhouette edges, which has been a bottleneck in previous work and tends to produce high-variance gradients when important edges are found with insufficient probability in scenes with complex visibility and high-resolution geometry. We show that our method only requires a few samples to produce gradients with low bias and variance for challenging cases such as glossy reflections and shadows. Finally, we use our differentiable path tracer to reconstruct the 3D geometry and materials of several real-world objects from a set of reference photographs.
References:
1. Marilyne Andersen, Siân Kleindienst, Lu Yi, Jaime Lee, Magali Bodart, and Barbara Cutler. 2008. An intuitive daylighting performance analysis and optimization approach. Building Research & Information 36, 6 (2008), 593–607.Google ScholarCross Ref
2. Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.Google ScholarCross Ref
3. Serge Belongie. 2019. Rodrigues’ Rotation Formula. From MathWorld — A Wolfram Web Resource, created by Eric W. Weisstein. http://mathworld.wolfram.com/RodriguesRotationFormula.htmlGoogle Scholar
4. Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2018. Inverse Transport Networks. (2018). http://arxiv.org/abs/1809.10820Google Scholar
5. M. de La Gorce, N. Paragios, and D. J. Fleet. 2008. Model-based hand tracking with texture, shading and self-occlusions. In 2008 IEEE Conference on Computer Vision and Pattern Recognition. 1–8.Google Scholar
6. Amaël Delaunoy and Emmanuel Prados. 2011. Gradient Flows for Optimizing Triangular Mesh-based Surfaces: Applications to 3D Reconstruction Problems Dealing with Visibility. International Journal of Computer Vision 95, 2 (01 Nov. 2011), 100–123.Google ScholarDigital Library
7. P. Gargallo, E. Prados, and P. Sturm. 2007. Minimizing the Reprojection Error in Surface Reconstruction from Images. In ICCV 2007. 1–8.Google Scholar
8. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An Evaluation of Computational Imaging Techniques for Heterogeneous Inverse Scattering. In Computer Vision – ECCV 2016. Springer International Publishing, Cham, 685–701.Google ScholarCross Ref
9. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013.Google Scholar
10. Inverse Volume Rendering with Material Dictionaries. ACM Trans. Graph. 32, 6, Article 162 (Nov. 2013).Google Scholar
11. Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. Vol. 105. SIAM.Google Scholar
12. Yu Guo, Miloš Hašan, and Shaung Zhao. 2018. Position-Free Monte Carlo Simulation for Arbitrary Layered BSDFs. ACM Trans. Graph. 37, 6 (2018).Google ScholarDigital Library
13. Milovš Hašan and Ravi Ramamoorthi. 2013. Interactive Albedo Editing in Path-traced Volumetric Materials. ACM Trans. Graph. 32, 2 (April 2013).Google ScholarDigital Library
14. Eric Heitz, Jonathan Dupuy, Stephen Hill, and David Neubelt. 2016a. Real-time Polygonal-light Shading with Linearly Transformed Cosines. ACM Trans. Graph. 35, 4 (July 2016).Google ScholarDigital Library
15. Eric Heitz, Johannes Hanika, Eugene d’Eon, and Carsten Dachsbacher. 2016b. Multiple-scattering microfacet BSDFs with the Smith model. ACM Transactions on Graphics 35 (July 2016).Google Scholar
16. André Jalobeanu, Frank O. Kuehnel, and John C. Stutz. 2004. Modeling Images of Natural 3D Surfaces: Overview and Potential Applications. 2004 Conference on Computer Vision and Pattern Recognition Workshop (2004).Google ScholarCross Ref
17. James T. Kajiya. 1986. The Rendering Equation. SIGGRAPH Comput. Graph. 20, 4 (Aug. 1986), 143–150.Google ScholarDigital Library
18. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
19. Csaba Kelemen, László Szirmay-Kalos, György Antal, and Ferenc Csonka. 2002. A simple and robust mutation strategy for the metropolis light transport algorithm. In Computer Graphics Forum, Vol. 21. Wiley Online Library, 531–540.Google Scholar
20. Markus Kettunen, Marco Manzi, Miika Aittala, Jaakko Lehtinen, Frédo Durand, and Matthias Zwicker. 2015. Gradient-Domain Path Tracing. ACM Trans. Graph. 34, 4 (2015).Google ScholarDigital Library
21. Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching Real Fabrics with Micro-Appearance Models. ACM Trans. Graph. 35, 1, Article 1 (Dec. 2015), 26 pages.Google ScholarDigital Library
22. Jaakko Lehtinen, Tero Karras, Samuli Laine, Miika Aittala, Frédo Durand, and Timo Aila. 2013. Gradient-Domain Metropolis Light Transport. ACM Trans. Graph. 32, 4 (2013).Google ScholarDigital Library
23. Tzu-Mao Li. 2019. Redner. https://github.com/BachiLi/redner.Google Scholar
24. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1–222:11.Google Scholar
25. Hsueh-Ti Derek Liu, Michael Tao, and Alec Jacobson. 2018. Paparazzi: Surface Editing by way of Multi-View Image Processing. ACM Transactions on Graphics (2018).Google Scholar
26. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. (2019). arXiv:1904.01786 http://arxiv.org/abs/1904.01786Google Scholar
27. Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In Computer Vision – ECCV 2014. Springer International Publishing, Cham.Google Scholar
28. Thu Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yong-Liang Yang. 2018. RenderNet: A deep convolutional network for differentiable rendering from 3D shapes. (2018). arXiv:1806.06575 http://arxiv.org/abs/1806.06575Google Scholar
29. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38, 6 (Nov. 2019).Google ScholarDigital Library
30. Matt Olson and Hao Zhang. 2006. Silhouette Extraction in Hough Space. Comput. Graph. Forum 25 (Sept. 2006).Google Scholar
31. Marios Papas, Christian Regg, Wojciech Jarosz, Bernd Bickel, Philip Jackson, Wojciech Matusik, Steve Marschner, and Markus Gross. 2013. Fabricating Translucent Materials Using Continuous Pigment Mixtures. ACM Trans. Graph. 32, 4, Article 146 (July 2013).Google ScholarDigital Library
32. Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich. 2010. OptiX: A General Purpose Ray Tracing Engine (SIGGRAPH ’10).Google Scholar
33. Gustavo Patow and Xavier Pueyo. 2003. A Survey of Inverse Rendering Problems. Computer Graphics Forum 22, 4 (2003), 663–687.Google ScholarCross Ref
34. Felix Petersen, Amit Bermano, Oliver Deussen, and Daniel Cohen-Or. 2019. Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer. (2019). arXiv:1903.11149 http://arxiv.org/abs/1903.11149Google Scholar
35. Ravi Ramamoorthi, Dhruv Mahajan, and Peter Belhumeur. 2007. A First-order Analysis of Lighting, Shading, and Shadows. ACM Trans. Graph. 26, 1 (2007).Google ScholarDigital Library
36. Helge Rhodin, Nadia Robertini, Christian Richardt, Hans-Peter Seidel, and Christian Theobalt. 2015. A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation (ICCV ’15). IEEE Computer Society.Google Scholar
37. Fabrice Rousselle, Wojciech Jarosz, and Jan Novák. 2016. Image-space Control Variates for Rendering. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 35, 6 (Dec. 2016).Google ScholarDigital Library
38. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1. MIT Press, Cambridge, MA, USA, Chapter Learning Internal Representations by Error Propagation, 318–362.Google ScholarDigital Library
39. Yuliy Schwartzburg, Romain Testuz, Andrea Tagliasacchi, and Mark Pauly. 2014. High-contrast Computational Caustic Design. ACM Trans. Graph. 33, 4 (July 2014).Google ScholarDigital Library
40. K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations.Google Scholar
41. V. N. Smelyansky, R. D. Morris, F. O. Kuehnel, D. A. Maluf, and P. Cheeseman. 2002. Dramatic Improvements to Feature Based Stereo. In Computer Vision — ECCV 2002. Springer.Google Scholar
42. Chia-Yin Tsai, Aswin C. Sankaranarayanan, and Ioannis Gkioulekas. 2019. Beyond Volumetric Albedo – A Surface Optimization Framework for Non-Line-Of-Sight Imaging. In CVPR 2019.Google Scholar
43. Eric Veach. 1998. Robust Monte Carlo Methods for Light Transport Simulation. Ph.D. Dissertation. Stanford, CA, USA. Advisor(s) Guibas, Leonidas J.Google ScholarDigital Library
44. Zdravko Velinov, Marios Papas, Derek Bradley, Paulo Gotardo, Parsa Mirdehghan, Steve Marschner, Jan Novák, and Thabo Beeler. 2018. Appearance Capture and Modeling of Human Teeth. ACM Trans. Graph. 37, 6 (Dec. 2018).Google ScholarDigital Library
45. Ingo Wald, Sven Woop, Carsten Benthin, Gregory S. Johnson, and Manfred Ernst. 2014. Embree: A Kernel Framework for Efficient CPU Ray Tracing. ACM Trans. Graph. 33, 4 (July 2014).Google ScholarDigital Library
46. Z Wang, Eero Simoncelli, and Alan Bovik. 2003. Multiscale structural similarity for image quality assessment. Conference Record of the Asilomar Conference on Signals, Systems and Computers 2.Google ScholarCross Ref
47. R. E. Wengert. 1964. A Simple Automatic Derivative Evaluation Program. Commun. ACM 7, 8 (Aug. 1964), 463–464.Google ScholarDigital Library
48. E Woodcock, T Murphy, P Hemmings, and S Longworth. 1965. Techniques used in the GEM code for Monte Carlo neutronics calculations in reactors and other systems of complex geometry. In Proc. Conf. Applications of Computing Methods to Reactor Problems, Vol. 557.Google Scholar
49. Shaung Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling Scattering Parameters for Rendering Anisotropic Media. ACM Trans. Graph. 35, 6 (2016).Google ScholarDigital Library
