“Mitsuba 2: a retargetable forward and inverse renderer” by Nimier-David, Vicini, Zeltner and Jakob
Conference:
Type(s):
Title:
- Mitsuba 2: a retargetable forward and inverse renderer
Session/Category Title: Watch Your Language
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Modern rendering systems are confronted with a dauntingly large and growing set of requirements: in their pursuit of realism, physically based techniques must increasingly account for intricate properties of light, such as its spectral composition or polarization. To reduce prohibitive rendering times, vectorized renderers exploit coherence via instruction-level parallelism on CPUs and GPUs. Differentiable rendering algorithms propagate derivatives through a simulation to optimize an objective function, e.g., to reconstruct a scene from reference images. Catering to such diverse use cases is challenging and has led to numerous purpose-built systems—partly, because retrofitting features of this complexity onto an existing renderer involves an error-prone and infeasibly intrusive transformation of elementary data structures, interfaces between components, and their implementations (in other words, everything).We propose Mitsuba 2, a versatile renderer that is intrinsically retargetable to various applications including the ones listed above. Mitsuba 2 is implemented in modern C++ and leverages template metaprogramming to replace types and instrument the control flow of components such as BSDFs, volumes, emitters, and rendering algorithms. At compile time, it automatically transforms arithmetic, data structures, and function dispatch, turning generic algorithms into a variety of efficient implementations without the tedium of manual redesign. Possible transformations include changing the representation of color, generating a “wide” renderer that operates on bundles of light paths, just-in-time compilation to create computational kernels that run on the GPU, and forward/reverse-mode automatic differentiation. Transformations can be chained, which further enriches the space of algorithms derived from a single generic implementation.We demonstrate the effectiveness and simplicity of our approach on several applications that would be very challenging to create without assistance: a rendering algorithm based on coherent MCMC exploration, a caustic design method for gradient-index optics, and a technique for reconstructing heterogeneous media in the presence of multiple scattering.
References:
1. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.Google Scholar
2. Attila T. Áfra, Carsten Benthin, Ingo Wald, and Jacob Munkberg. 2016. Local Shading Coherence Extraction for SIMD-efficient Path Tracing on CPUs. In Proceedings of High Performance Graphics (HPG ’16). Eurographics Association.Google ScholarDigital Library
3. Luke Anderson, Tzu-Mao Li, Jaakko Lehtinen, and Frédo Durand. 2017. Aether: An embedded domain specific sampling language for Monte Carlo rendering. ACM Transactions on Graphics 36, 4 (2017).Google ScholarDigital Library
4. Christophe Andrieu and Gareth O Roberts. 2009. The pseudo-marginal approach for efficient Monte Carlo computations. The Annals of Statistics (2009).Google Scholar
5. Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In Proceedings of Computer Vision and Pattern Recognition (CVPR), IEEE.Google ScholarCross Ref
6. Seung-Hwan Baek, Daniel S. Jeon, Xin Tong, and Min H. Kim. 2018. Simultaneous Acquisition of Polarimetric SVBRDF and Normals. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2018) 36, 6 (2018).Google Scholar
7. Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.Google Scholar
8. Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2018. The Design and Evolution of Disney’s Hyperion Renderer. ACM Transactions on Graphics 37, 3, Article 33 (July 2018).Google ScholarDigital Library
9. Bob Carpenter, Matthew D. Hoffman, Marcus Brubaker, Daniel Lee, Peter Li, and Michael Betancourt. 2015. The Stan Math Library: Reverse-Mode Automatic Differentiation in C++. CoRR abs/1509.07164 (2015). arXiv:1509.07164 http://arxiv.org/abs/1509.07164Google Scholar
10. Subrahmanyan Chandrasekhar. 1960. Radiative transfer. Dover publications, New York.Google Scholar
11. Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2018. Inverse Transport Networks. arXiv preprint arXiv:1809.10820 (2018).Google Scholar
12. Edward Collett. 1993. Polarized light : fundamentals and application. Marcel Dekker New York.Google Scholar
13. Oskar Elek, Denis Sumin, Ran Zhang, Tim Weyrich, Karol Myszkowski, Bernd Bickel, Alexander Wilkie, and Jaroslav Křivánek. 2017. Scattering-aware Texture Reproduction for 3D Printing. ACM Transactions on Graphics 36, 6 (Nov. 2017).Google ScholarDigital Library
14. Luca Fascione, Johannes Hanika, Marcos Fajardo, Per Christensen, Brent Burley, and Brian Green. 2017. Path Tracing in Production – Part 1: Production Renderers. In ACM SIGGRAPH 2017 Courses (SIGGRAPH ’17).Google ScholarDigital Library
15. Luca Fascione, Johannes Hanika, Mark Leone, Marc Droske, Jorge Schwarzhaupt, Tomáš Davidovič, Andrea Weidlich, and Johannes Meng. 2018. Manuka: A Batch-Shading Architecture for Spectral Path Tracing in Movie Production. ACM Transactions on Graphics 37, 3, Article 31 (Aug. 2018).Google ScholarDigital Library
16. Mathieu Galtier, Stéphane Blanco, Cyril Caliot, Christophe Coustet, Jérémi Dauchet, Mouna El Hafi, Vincent Eymet, Richard Fournier, Jacques Gautrais, Anaïs Khuong, et al. 2013. Integral formulation of null-collision Monte Carlo algorithms. Journal of Quantitative Spectroscopy and Radiative Transfer 125 (2013).Google Scholar
17. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse Volume Rendering with Material Dictionaries. ACM Transactions on Graphics 32, 6, Article 162 (Nov. 2013).Google ScholarDigital Library
18. A Griewank and S Reese. 1991. On the calculation of Jacobian matrices by the Markowitz rule. Technical Report. Argonne National Lab., IL (United States).Google Scholar
19. Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. Vol. 105. SIAM.Google Scholar
20. Gaël Guennebaud, Benoît Jacob, et al. 2010. Eigen v3. http://eigen.tuxfamily.org.Google Scholar
21. Johannes Hanika. 2019. Personal communication.Google Scholar
22. Eric Heitz and Eugene d’Eon. 2014. Importance Sampling Microfacet-Based BSDFs using the Distribution of Visible Normals. In Computer Graphics Forum, Vol. 33.Google ScholarDigital Library
23. Robin J. Hogan. 2014. Fast Reverse-Mode Automatic Differentiation Using ExpressionGoogle Scholar
24. Templates in C++. ACM Trans. Math. Software 40, 4, Article 26 (July 2014).Google Scholar
25. Wenzel Jakob. 2010. Mitsuba renderer. https://www.mitsuba-renderer.org. (Date accessed: 2019-08-25).Google Scholar
26. Wenzel Jakob. 2019. Enoki: structured vectorization and differentiation on modern processor architectures. https://github.com/mitsuba-renderer/enoki. (Date accessed: 2019-08-25).Google Scholar
27. Wenzel Jakob and Johannes Hanika. 2019. A Low-Dimensional Function Space for Efficient Spectral Upsampling. Computer Graphics Forum (Proceedings of Eurographics) 38, 2 (March 2019).Google ScholarCross Ref
28. Wenzel Jakob, Jason Rhinelander, and Dean Moldovan. 2017. pybind11—Seamless operability between C++11 and Python. https://github.com/pybind/pybind11.Google Scholar
29. Adrian Jarabo and Victor Arellano. 2018. Bidirectional Rendering of Vector Light Transport. Computer Graphics Forum 37, 6 (2018).Google Scholar
30. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2017. Neural 3D Mesh Renderer. CoRR abs/1711.07566 (2017). arXiv:1711.07566 http://arxiv.org/abs/1711.07566Google Scholar
31. Csaba Kelemen, László Szirmay-Kalos, György Antal, and Ferenc Csonka. 2002. A simple and robust mutation strategy for the metropolis light transport algorithm. In Computer Graphics Forum, Vol. 21. Wiley Online Library.Google Scholar
32. Alexander Keller, Carsten Wächter, Matthias Raab, Daniel Seibert, Dietger van Antwerpen, Johann Korndörfer, and Lutz Kettner. 2017. The Iray Light Transport Simulation and Rendering System. In ACM SIGGRAPH 2017 Talks (SIGGRAPH ’17). ACM, New York, NY, USA.Google Scholar
33. Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching Real Fabrics with Micro-Appearance Models. ACM Transactions on Graphics 35, 1 (2015).Google ScholarDigital Library
34. Samuli Laine, Tero Karras, and Timo Aila. 2013. Megakernels Considered Harmful: Wavefront Path Tracing on GPUs. In Proceedings of the 5th High-Performance Graphics Conference (HPG ’13). ACM.Google ScholarDigital Library
35. Mark Lee, Brian Green, Feng Xie, and Eric Tabellion. 2017. Vectorized production path tracing. In Proceedings of High Performance Graphics. ACM.Google ScholarDigital Library
36. Roland Leißa, Klaas Boesche, Sebastian Hack, Arsène Pérard-Gayot, Richard Membarth, Philipp Slusallek, André Müller, and Bertil Schmidt. 2018. AnyDSL: A Partial Evaluation Framework for Programming High-Performance Libraries. Proceedings of the ACM on Programming Languages (PACMPL) 2, OOPSLA (Nov. 2018).Google Scholar
37. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018a. Differentiable Monte Carlo Ray Tracing Through Edge Sampling. ACM Transactions on Graphics 37, 6, Article 222 (Dec. 2018).Google ScholarDigital Library
38. Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, and Jonathan Ragan-Kelley. 2018b. Differentiable programming for image processing and deep learning in Halide. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 37, 4 (2018).Google Scholar
39. Tzu-Mao Li, Jaakko Lehtinen, Ravi Ramamoorthi, Wenzel Jakob, and Frédo Durand. 2015. Anisotropic gaussian mutations for metropolis light transport through hessian-hamiltonian dynamics. ACM Transactions on Graphics 34, 6 (2015).Google ScholarDigital Library
40. Jun S Liu, Faming Liang, and Wing Hung Wong. 2000. The multiple-try method and local optimization in Metropolis sampling. J. Amer. Statist. Assoc. 95, 449 (2000).Google ScholarCross Ref
41. Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. 2019. Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction. CoRR abs/1901.05567 (2019). arXiv:1901.05567 http://arxiv.org/abs/1901.05567Google Scholar
42. Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In European Conference on Computer Vision. Springer.Google ScholarCross Ref
43. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing Discontinuous Integrands for Differentiable Rendering. ACM Transactions on Graphics (Dec. 2019).Google Scholar
44. Duane Merrill. 2015. CUB library. http://nvlabs.github.io/cub. (Date accessed: 2019-08-25).Google Scholar
45. Scott Meyers. 2005. Effective C++: 55 specific ways to improve your programs and designs. Pearson Education.Google Scholar
46. Michal Mojzík, Tomáš Skřivan, Alexander Wilkie, and Jaroslav Křivánek. 2016. Bi-Directional Polarised Light Transport. In Eurographics Symposium on Rendering – Experimental Ideas & Implementations, Elmar Eisemann and Eugene Fiume (Eds.). The Eurographics Association.Google Scholar
47. Iain Murray. 2007. Advances in Markov chain Monte Carlo methods. University of London.Google Scholar
48. Uwe Naumann. 2007. Optimal Jacobian accumulation is NP-complete. Mathematical Programming 112 (2007).Google Scholar
49. Du T. Nguyen, Cameron Meyers, Timothy D. Yee, Nikola A. Dudukovic, Joel F. Destino, Cheng Zhu, Eric B. Duoss, Theodore F. Baumann, Tayyab Suratwala, James E. Smay, and Rebecca Dylla-Spears. 2017. 3D-Printed Transparent Glass. Advanced Materials 29, 26 (2017).Google Scholar
50. Melissa E. O’Neill. 2014. PCG: A Family of Simple Fast Space-Efficient Statistically Good Algorithms for Random Number Generation. Technical Report HMC-CS-2014-0905. Harvey Mudd College, Claremont, CA.Google Scholar
51. Marios Papas, Wojciech Jarosz, Wenzel Jakob, Szymon Rusinkiewicz, Wojciech Matusik, and Tim Weyrich. 2011. Goal-Based Caustics. Computer Graphics Forum (Proceedings of Eurographics) 30, 2 (June 2011).Google ScholarCross Ref
52. Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich. 2010. OptiX: A General Purpose Ray Tracing Engine. ACM Transactions on Graphics 29, 4, Article 66 (July 2010).Google ScholarDigital Library
53. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).Google Scholar
54. Felix Petersen, Amit H. Bermano, Oliver Deussen, and Daniel Cohen-Or. 2019. Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer. CoRR abs/1903.11149 (2019). arXiv:1903.11149 http://arxiv.org/abs/1903.11149Google Scholar
55. Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically based rendering: From theory to implementation (third ed.). Morgan Kaufmann.Google Scholar
56. Matt Pharr, Craig Kolb, Reid Gershbein, and Pat Hanrahan. 1997. Rendering Complex Scenes with Memory-coherent Ray Tracing. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’97). ACM Press/Addison-Wesley Publishing Co.Google ScholarDigital Library
57. Matt Pharr and William R Mark. 2012. ispc: A SPMD compiler for high-performance CPU programming. In 2012 Innovative Parallel Computing (InPar). IEEE.Google Scholar
58. Arsène Pérard-Gayot, Richard Membarth, Roland Leißa, Sebastian Hack, and Philipp Slusallek. 2019. Rodent: Generating Renderers without Writing a Generator. ACM Transactions on Graphics 38, 4 (7 2019).Google ScholarDigital Library
59. Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. 2013. Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines. SIGPLAN Notices 48, 6 (June 2013).Google ScholarDigital Library
60. Helge Rhodin, Nadia Robertini, Christian Richardt, Hans-Peter Seidel, and Christian Theobalt. 2015. A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation. In Proceedings of ICCV 2015.Google ScholarDigital Library
61. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1. MIT Press, Cambridge, MA, USA, Chapter Learning Internal Representations by Error Propagation.Google ScholarDigital Library
62. Yuliy Schwartzburg, Romain Testuz, Andrea Tagliasacchi, and Mark Pauly. 2014. High-contrast Computational Caustic Design. ACM Transactions on Graphics 33, 4, Article 74 (July 2014). Proc. SIGGRAPH 2014.Google ScholarDigital Library
63. Benjamin Segovia, Jean-Claude Iehl, and Bernard Péroche. 2007a. Coherent metropolis light transport with multiple-try mutations.Google Scholar
64. Benjamin Segovia, Jean Claude Iehl, and Bernard Péroche. 2007b. Metropolis instant radiosity. In Computer Graphics Forum, Vol. 26. Wiley Online Library.Google Scholar
65. Anurag Sharma, D Vizia Kumar, and Ajoy K Ghatak. 1982. Tracing rays through graded-index media: a new method. Applied Optics 21, 6 (1982).Google Scholar
66. Denis Sumin, Tobias Rittig, Vahid Babaei, Thomas Nindel, Alexander Wilkie, Piotr Didyk, Bernd Bickel, Jaroslav Křivánek, Karol Myszkowski, and Tim Weyrich. 2019. Geometry-Aware Scattering Compensation for 3D Printing. ACM Transactions on Graphics (2019).Google Scholar
67. JM Tregan, S Blanco, J Dauchet, M Hafi, R Fournier, L Ibarrart, P Lapeyre, and N Villefranque. 2019. Convergence issues in derivatives of Monte Carlo null-collision integral formulations: a solution. arXiv preprint arXiv:1903.06508 (2019).Google Scholar
68. Eric Veach. 1997. Robust monte carlo methods for light transport simulation. Number 1610. Stanford University PhD thesis.Google ScholarDigital Library
69. Todd Veldhuizen. 1995. Expression Templates. C++ Report 7 (1995).Google Scholar
70. Yu M Volin and GM Ostrovskii. 1985. Automatic computation of derivatives with the use of the multilevel differentiating technique—1. Algorithmic basis. Computers & mathematics with applications 11, 11 (1985).Google Scholar
71. Ingo Wald, Sven Woop, Carsten Benthin, Gregory S. Johnson, and Manfred Ernst. 2014. Embree: A Kernel Framework for Efficient CPU Ray Tracing. ACM Transactions on Graphics 33, 4, Article 143 (July 2014).Google ScholarDigital Library
72. Alexander Wilkie and Andrea Weidlich. 2012. Polarised Light in Computer Graphics. In SIGGRAPH Asia 2012 Courses (SA ’12). ACM, New York, NY, USA, Article 8.Google Scholar
73. E Woodcock, T Murphy, P Hemmings, and S Longworth. 1965. Techniques used in the GEM code for Monte Carlo neutronics calculations in reactors and other systems of complex geometry. In Proceedings of the Conference on Applications of Computing Methods to Reactor Problems, Vol. 557.Google Scholar
74. Pavan Yalamanchili, Umar Arshad, Zakiuddin Mohammed, Pradeep Garigipati, Peter Entschev, Brian Kloppenborg, James Malcolm, and John Melonakos. 2015. ArrayFire – A high performance software library for parallel computing with an easy-to-use API. https://github.com/arrayfire/arrayfireGoogle Scholar
75. Toshinobu Yoshida. 1987. Derivation of a computational process for partial derivatives of functions using transformations of a graph. Transactions of Information Processing Society of Japan 11, 19 (1987).Google Scholar
76. Yonghao Yue, Kei Iwasaki, Bing-Yu Chen, Yoshinori Dobashi, and Tomoyuki Nishita. 2014. Poisson-Based Continuous Surface Generation for Goal-Based Caustics. ACM Transactions on Graphics 33, 3, Article 31 (June 2014).Google ScholarDigital Library
77. Shaung Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling Scattering Parameters for Rendering Anisotropic Media. ACM Transactions on Graphics 35, 6 (2016).Google ScholarDigital Library
