“Differentiable Monte Carlo ray tracing through edge sampling”
Conference:
Type(s):
Title:
- Differentiable Monte Carlo ray tracing through edge sampling
Session/Category Title: Beyond light transport
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Gradient-based methods are becoming increasingly important for computer graphics, machine learning, and computer vision. The ability to compute gradients is crucial to optimization, inverse problems, and deep learning. In rendering, the gradient is required with respect to variables such as camera parameters, light sources, scene geometry, or material appearance. However, computing the gradient of rendering is challenging because the rendering integral includes visibility terms that are not differentiable. Previous work on differentiable rendering has focused on approximate solutions. They often do not handle secondary effects such as shadows or global illumination, or they do not provide the gradient with respect to variables other than pixel coordinates.We introduce a general-purpose differentiable ray tracer, which, to our knowledge, is the first comprehensive solution that is able to compute derivatives of scalar functions over a rendered image with respect to arbitrary scene parameters such as camera pose, scene geometry, materials, and lighting parameters. The key to our method is a novel edge sampling algorithm that directly samples the Dirac delta functions introduced by the derivatives of the discontinuous integrand. We also develop efficient importance sampling methods based on spatial hierarchies. Our method can generate gradients in times running from seconds to minutes depending on scene complexity and desired precision.We interface our differentiable ray tracer with the deep learning library PyTorch and show prototype applications in inverse rendering and the generation of adversarial examples for neural networks.
References:
1. Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance modeling by neural texture synthesis. ACM Trans. Graph. (Proc. SIGGRAPH) 35, 4 (2016), 65:1–65:13. Google ScholarDigital Library
2. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2013. Practical SVBRDF Capture In The Frequency Domain. ACM Trans. Graph. (Proc. SIGGRAPH) 32, 4 (2013), 110:1–110:12. Google ScholarDigital Library
3. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF Capture for Stationary Materials. ACM Trans. Graph. (Proc. SIGGRAPH) 34, 4 (2015), 110:1–110:13. Google ScholarDigital Library
4. James Arvo. 1994. The Irradiance Jacobian for Partially Occluded Polyhedral Sources. In SIGGRAPH. 343–350. Google ScholarDigital Library
5. Jonathan T Barron and Jitendra Malik. 2015. Shape, Illumination, and Reflectance from Shading. Transactions on Pattern Analysis and Machine Intelligence 37, 8 (2015), 1670–1687.Google ScholarDigital Library
6. Bruce Guenther Baumgart. 1974. Geometric modeling for computer vision. Technical Report. Stanford University Department of Computer Science.Google Scholar
7. Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.Google Scholar
8. Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In SIGGRAPH. 187–194. Google ScholarDigital Library
9. Adrien Bousseau, Emmanuelle Chapoulie, Ravi Ramamoorthi, and Maneesh Agrawala. 2011. Optimizing environment maps for material depiction. Computer Graphics Forum (Proc. EGSR) 30, 4 (2011), 1171–1180. Google ScholarDigital Library
10. Brent Burley. 2012. Physically-based shading at Disney. In SIGGRAPH Course Notes. Practical physically-based shading in film and game production., Vol. 2012. 1–7. Google ScholarDigital Library
11. Min Chen and James Arvo. 2000. Theory and application of specular path perturbation. ACM Trans. Graph. 19, 4 (2000), 246–278. Google ScholarDigital Library
12. Alejandro Conty Estevez and Christopher Kulla. 2018. Importance Sampling of Many Lights with Adaptive Tree Splitting. ACM Comput. Graph. Interact. Tech. (Proc. HPG) 1, 2 (2018), 25:1–25:17. Google ScholarDigital Library
13. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An evaluation of computational imaging techniques for heterogeneous inverse scattering. In European Conference on Computer Vision. Springer, 685–701.Google ScholarCross Ref
14. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse Volume Rendering with Material Dictionaries. ACM Trans. Graph. 32, 6 (2013), 162:1–162:13. Google ScholarDigital Library
15. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations.Google Scholar
16. Andreas Griewank and Andrea Walther. 2008. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (second ed.). Society for Industrial and Applied Mathematics, Philadelphia, PA, USA. Google ScholarDigital Library
17. Eric Heitz, Jonathan Dupuy, Stephen Hill, and David Neubelt. 2016. Real-time polygonal-light shading with linearly transformed cosines. ACM Trans. Graph. (Proc. SIGGRAPH) 35, 4 (2016), 41:1–41:8. Google ScholarDigital Library
18. Eric Heitz and Stephen Hill. 2017. Linear-Light Shading with Linearly Transformed Cosines. In GPU Zen.Google Scholar
19. Aaron Hertzmann. 1999. Introduction to 3D Non-Photorealistic Rendering: Silhouettes and Outlines. In SIGGRAPH Course Notes. Course on Non-Photorelistic Rendering, Stuart Green (Ed.). ACM Press/ACM SIGGRAPH, New York.Google Scholar
20. Aaron Hertzmann and Denis Zorin. 2000. Illustrating smooth surfaces. In SIGGRAPH. 517–526. Google ScholarDigital Library
21. Lars Hörmander. 1983. The analysis of linear partial differential operators I: Distribution theory and Fourier analysis.Google Scholar
22. Homan Igehy. 1999. Tracing Ray Differentials. SIGGRAPH, 179–186. Google ScholarDigital Library
23. Wenzel Jakob, Miloš Hašan, Ling-Qi Yan, Jason Lawrence, Ravi Ramamoorthi, and Steve Marschner. 2014. Discrete stochastic microfacet models. ACM Transs Graph. (Proc. SIGGRAPH) 33, 4 (2014), 115:1–115:10. Google ScholarDigital Library
24. Wenzel Jakob and Steve Marschner. 2012. Manifold exploration: a Markov Chain Monte Carlo technique for rendering scenes with difficult specular transport. ACM Trans. Graph. (Proc. SIGGRAPH) 31, 4 (2012), 58:1–58:13. Google ScholarDigital Library
25. Wojciech Jarosz, Volker Schönefeld, Leif Kobbelt, and Henrik Wann Jensen. 2012. Theory, analysis and applications of 2D global illumination. ACM Trans. Graph. 31, 5 (2012), 125:1–125:21. Google ScholarDigital Library
26. Michael J. Jones and Tomaso Poggio. 1996. Model-Based Matching by Linear Combinations of Prototypes. Technical Report. Google Scholar
27. Nathaniel Louis Jones. 2017. Validated interactive daylighting analysis for architectural design. Ph.D. Dissertation. Massachusetts Institute of Technology.Google Scholar
28. James T. Kajiya. 1986. The Rendering Equation. Computer Graphics (Proc. SIGGRAPH) 20, 4 (1986), 143–150. Google ScholarDigital Library
29. Anton S Kaplanyan, Johannes Hanika, and Carsten Dachsbacher. 2014. The natural-constraint representation of the path space for efficient light transport simulation. ACM Trans. Graph. (Proc. SIGGRAPH) 33, 4 (2014), 102:1–102:13. Google ScholarDigital Library
30. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In Conference on Computer Vision and Pattern Recognition. 3907–3916.Google ScholarCross Ref
31. Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching Real Fabrics with Micro-Appearance Models. ACM Trans. Graph. 35, 1 (2015), 1:1–1:26. Google ScholarDigital Library
32. Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.Google Scholar
33. Jaroslav Krivanek, Pascal Gautron, Sumanta Pattanaik, and Kadi Bouatouch. 2005. Radiance Caching for Efficient Global Illumination. (2005), 550–561. Google ScholarDigital Library
34. Tzu-Mao Li, Jaakko Lehtinen, Ravi Ramamoorthi, Wenzel Jakob, and Frédo Durand. 2015. Anisotropic Gaussian Mutations for Metropolis Light Transport through Hessian-Hamiltonian Dynamics. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 34, 6 (2015), 209:1–209:13. Google ScholarDigital Library
35. Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. 2017. Material Editing Using a Physically Based Rendering Network. In International Conference on Computer Vision. 2280–2288.Google ScholarCross Ref
36. Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In European Conference on Computer Vision, Vol. 8695. 154–169.Google Scholar
37. Morgan McGuire. 2017. Computer Graphics Archive. https://casual-effects.com/dataGoogle Scholar
38. Eric Paquette, Pierre Poulin, and George Drettakis. 1998. A Light Hierarchy for Fast Rendering of Scenes with Many Lights. Computer Graphics Forum (Proc. Eurographics) (1998), 63–74.Google Scholar
39. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).Google Scholar
40. Gustavo Patow and Xavier Pueyo. 2003. A survey of inverse rendering problems. Computer Graphics Forum 22, 4 (2003), 663–687.Google ScholarCross Ref
41. Ravi Ramamoorthi, Dhruv Mahajan, and Peter Belhumeur. 2007. A First-order Analysis of Lighting, Shading, and Shadows. ACM Trans. Graph. 26, 1 (2007), 2:1–2:21. Google ScholarDigital Library
42. Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel. 2017. Learning detailed face reconstruction from a single image. In Conference on Computer Vision and Pattern Recognition. 5553–5562.Google ScholarCross Ref
43. Pedro V. Sander, Xianfeng Gu, Steven J. Gortler, Hugues Hoppe, and John Snyder. 2000. Silhouette Clipping. In SIGGRAPH. 327–334. Google ScholarDigital Library
44. Ram Shacked and Dani Lischinski. 2001. Automatic lighting design using a perceptual quality metric. Computer Graphics Forum 20, 3 (2001), 215–227.Google ScholarCross Ref
45. Mikio Shinya, T. Takahashi, and Seiichiro Naito. 1987. Principles and Applications of Pencil Tracing. Comput. Graph. (Proc. SIGGRAPH) 21, 4 (1987), 45–54. Google ScholarDigital Library
46. K. Simonyan and A. Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
47. Frank Suykens and Yves D. Willems. 2001. Path Differentials and Applications. In Eurographics Workshop on Rendering Techniques. 257–268. Google ScholarDigital Library
48. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations.Google Scholar
49. Eric Veach and Leonidas J. Guibas. 1995. Optimally Combining Sampling Techniques for Monte Carlo Rendering. In SIGGRAPH. 419–428. Google ScholarDigital Library
50. Eric Veach and Leonidas J. Guibas. 1997. Metropolis Light Transport. In SIGGRAPH. 65–76. Google ScholarDigital Library
51. Ingo Wald, Sven Woop, Carsten Benthin, Gregory S Johnson, and Manfred Ernst. 2014. Embree: a kernel framework for efficient CPU ray tracing. ACM Trans. on Graph. (Proc. SIGGRAPH) 33, 4 (2014), 143. Google ScholarDigital Library
52. Bruce Walter. 2005. Notes on the Ward BRDF. Program of Computer Graphics, Cornell University, Technical report PCG-05 6 (2005).Google Scholar
53. Bruce Walter, Adam Arbree, Kavita Bala, and Donald P Greenberg. 2006. Multidimensional lightcuts. ACM Trans. Graph. (Proc. SIGGRAPH) 25, 3 (2006), 1081–1088. Google ScholarDigital Library
54. Bruce Walter, Sebastian Fernandez, Adam Arbree, Kavita Bala, Michael Donikian, and Donald P Greenberg. 2005. Lightcuts: a scalable approach to illumination. ACM Trans. Graph. (Proc. SIGGRAPH) 24, 3 (2005), 1098–1107. Google ScholarDigital Library
55. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet models for refraction through rough surfaces. Rendering Techniques (Proc. EGSR) (2007), 195–206. Google ScholarDigital Library
56. Greg Ward and Paul Heckbert. 1992. Irradiance Gradients. In Eurographics Rendering Workshop. 85–98. Google ScholarDigital Library
57. Y. Yang and C. Barnes. 2018. Approximate Program Smoothing Using Mean-Variance Statistics, with Application to Procedural Shader Bandlimiting. Computer Graphics Forum (Proc. Eurographics) 37, 2 (2018), 443–454.Google ScholarCross Ref
58. Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins. 1999. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In SIGGRAPH. 215–224. Google ScholarDigital Library
59. Xiaohui Zeng, Chenxi Liu, Weichao Qiu, Lingxi Xie, Yu-Wing Tai, Chi Keung Tang, and Alan L Yuille. 2017. Adversarial Attacks Beyond the Image Space. arXiv preprint arXiv:1711.07183 (2017).Google Scholar
