“Adjoint nonlinear ray tracing” by Teh, O’Toole and Gkioulekas

Reconstructing and designing media with continuously-varying refractive index fields remains a challenging problem in computer graphics. A core difficulty in trying to tackle this inverse problem is that light travels inside such media along curves, rather than straight lines. Existing techniques for this problem make strong assumptions on the shape of the ray inside the medium, and thus limit themselves to media where the ray deflection is relatively small. More recently, differentiable rendering techniques have relaxed this limitation, by making it possible to differentiably simulate curved light paths. However, the automatic differentiation algorithms underlying these techniques use large amounts of memory, restricting existing differentiable rendering techniques to relatively small media and low spatial resolutions.We present a method for optimizing refractive index fields that both accounts for curved light paths and has a small, constant memory footprint. We use the adjoint state method to derive a set of equations for computing derivatives with respect to the refractive index field of optimization objectives that are subject to nonlinear ray tracing constraints. We additionally introduce discretization schemes to numerically evaluate these equations, without the need to store nonlinear ray trajectories in memory, significantly reducing the memory requirements of our algorithm. We use our technique to optimize high-resolution refractive index fields for a variety of applications, including creating different types of displays (multiview, lightfield, caustic), designing gradient-index optics, and reconstructing gas flows.

1. Marco Ament, Christoph Bergmann, and Daniel Weiskopf. 2014. Refractive radiative transfer equation. ACM TOG (2014).Google Scholar
2. Bradley Atcheson, Ivo Ihrke, Wolfgang Heidrich, Art Tevs, Derek Bradley, Marcus Magnor, and Hans-Peter Seidel. 2008. Time-resolved 3d capture of non-stationary gas flows. ACM TOG (2008).Google Scholar
3. Manushanker Balasubramanian, Sawyer D. Campbell, and Douglas H. Werner. 2020. Highly-efficient GRIN Lens Optimization Through Differential Ray Tracing. IEEE ISAP (2020).Google Scholar
4. Max Born and Emil Wolf. 2013. Principles of optics: electromagnetic theory of propagation, interference and diffraction of light.Google Scholar
5. Maysamreza Chamanzar, Matteo Giuseppe Scopelliti, Julien Bloch, Ninh Do, Minyoung Huh, Dongjin Seo, Jillian Iafrati, Vikaas S Sohal, Mohammad-Reza Alam, and Michel M Maharbiz. 2019. Ultrasonic sculpting of virtual optical waveguides in tissue. Nature Communications (2019).Google Scholar
6. Guy Chavent. 1974. Identification of Functional Parameters in Partial Differential Equations. Joint Automatic Control Conference.Google Scholar
7. Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018. Neural Ordinary Differential Equations. NeurIPS (2018).Google Scholar
8. Fouad El-Diasty. 2003. Evaluation of some GRIN fiber parameters and the associated fraction mode loss due to mechanically induced optical anisotropy. Applied optics (2003).Google Scholar
9. Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. 2020. ADD: analytically differentiable dynamics for multi-body systems with frictional contact. ACM TOG (2020).Google ScholarDigital Library
10. Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. 2017. The reversible residual network: Backpropagation without storing activations. NeurIPS (2017).Google Scholar
11. Andreas Griewank and Andrea Walther. 2008. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. SIAM review.Google ScholarCross Ref
12. Eduard Gröller. 1995. Nonlinear ray tracing: Visualizing strange worlds. The Visual Computer (1995).Google Scholar
13. Diego Gutierrez, A Muñoz, F Seron, E Jimenez, María de Luna, and Edificio Ada Byron. 2003. Global illumination in inhomogeneous media based on curved photon mapping. Visualization, Imaging, and Image Processing (2003).Google Scholar
14. Ernst Hairer, Christian Lubich, and Gerhard Wanner. 2006. Geometric numerical integration. Springer-Verlag, Berlin,.Google Scholar
15. Michael Hinze, René Pinnau, Michael Ulbrich, and Stefan Ulbrich. 2008. Optimization with PDE constraints. Springer.Google Scholar
16. Makiko Hisatomi, Michael C. Parker, and Stuart D. Walker. 2005. Comparison of zoned microstructure fiber geometries for low-dispersion waveguiding. Journal of Lightwave Technology (2005).Google Scholar
17. Ivo Ihrke. 2007. Reconstruction and rendering of time-varying natural phenomena. (2007).Google Scholar
18. Ivo Ihrke, Gernot Ziegler, Art Tevs, Christian Theobalt, Marcus Magnor, and Hans-Peter Seidel. 2007. Eikonal rendering: Efficient light transport in refractive objects. ACM TOG (2007).Google Scholar
19. Wenzel Jakob. 2019. Enoki: structured vectorization and differentiation on modern processor architectures. https://github.com/mitsuba-renderer/enoki.Google Scholar
20. Yu Ji, Jinwei Ye, and Jingyi Yu. 2013. Reconstructing gas flows using light-path approximation. IEEE CVPR (2013).Google ScholarDigital Library
21. SeungYeon Kang, Elena Dotsenko, David Amrhein, Christian Theriault, and Craig B Arnold. 2018. Ultra-high-speed variable focus optics for novel applications in advanced imaging. In Photonic Instrumentation Engineering V.Google Scholar
22. Liliya Kharevych, W Wei, Yiying Tong, Eva Kanso, Jerrold E Marsden, Peter Schröder, and Matthieu Desbrun. 2006. Geometric, variational integrators for computer animation. Eurographics Association.Google Scholar
23. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980 (2014).Google Scholar
24. Yu A Kravtsov and Yu I Orlov. 1990. Geometrical optics of inhomogeneous media. Springer.Google Scholar
25. Shingyu Leung, Jianliang Qian, et al. 2006. An adjoint state method for three-dimensional transmission traveltime tomography using first-arrivals. Communications in Mathematical Sciences (2006).Google Scholar
26. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo ray tracing through edge sampling. ACM TOG (2018).Google Scholar
27. Zongling Li, Qingyu Hou, Zhipeng Wang, Fanjiao Tan, Jin Liu, and Wei Zhang. 2021. End-to-end learned single lens design using fast differentiable ray tracing. Optics Letters (2021).Google Scholar
28. Rudolf K Luneberg. 1944. Mathematical Theory of Optics. Providence. Brown Univ. Press.Google Scholar
29. Matthew MacKay, Paul Vicol, Jimmy Ba, and Roger B Grosse. 2018. Reversible recurrent neural networks. NeurIPS (2018).Google Scholar
30. James Clerk Maxwell. 1854. Solutions of problems. Cambridge Dublin Math. J. (1854).Google Scholar
31. Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. 2004. Fluid control using the adjoint method. ACM TOG (2004).Google Scholar
32. Jocelyn Meyron, Quentin Mérigot, and Boris Thibert. 2018. Light in Power: A General and Parameter-Free Algorithm for Caustic Design. ACM TOG (2018).Google ScholarDigital Library
33. Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. ACM TOG (2020).Google ScholarDigital Library
34. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. ACM TOG (2019).Google ScholarDigital Library
35. Marios Papas, Wojciech Jarosz, Wenzel Jakob, Szymon Rusinkiewicz, Wojciech Matusik, and Tim Weyrich. 2011. Goal-based caustics. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 503–511.Google Scholar
36. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. NeurIPS (2019).Google Scholar
37. Adithya Pediredla, Yasin Karimi Chalmiani, Matteo Giuseppe Scopelliti, Maysamreza Chamanzar, Srinivasa Narasimhan, and Ioannis Gkioulekas. 2020. Path tracing estimators for refractive radiative transfer. ACM TOG (2020).Google Scholar
38. Rene-Edouard Plessix. 2006. A review of the adjoint-state method for computing the gradient of a functional with geophysical applications. Geophysical Journal International (2006).Google Scholar
39. Udo Schröder and Thomas Schuster. 2016. An iterative method to reconstruct the refractive index of a medium from time-of-flight measurements. Inverse Problems 32, 8 (jun 2016), 085009. Google ScholarCross Ref
40. Yuliy Schwartzburg, Romain Testuz, Andrea Tagliasacchi, and Mark Pauly. 2014. High-contrast computational caustic design. ACM TOG (2014).Google Scholar
41. Matteo Giuseppe Scopelliti and Maysamreza Chamanzar. 2019. Ultrasonically sculpted virtual relay lens for in situ microimaging. Light: Science & Applications (2019).Google ScholarCross Ref
42. Matteo Giuseppe Scopelliti, Hengji Huang, Adithya Pediredla, Srinivasa G Narasimhan, Ioannis Gkioulekas, and Maysamreza Chamanzar. 2020. Overcoming the tradeoff between confinement and focal distance using virtual ultrasonic optical waveguides. Optics Express (2020).Google ScholarCross Ref
43. James A Sethian. 1999. Fast marching methods. SIAM review (1999).Google Scholar
44. Anurag Sharma, Dhanwada Vizia Kumar, and Ajoy K. Ghatak. 1982. Tracing rays through graded-index media: a new method. Applied Optics (1982).Google Scholar
45. Nicholas Sharp and Keenan Crane. 2018. Variational surface cutting. ACM TOG (2018).Google Scholar
46. Jos Stam. 2020. Computing Light Transport Gradients using the Adjoint Method. arXiv:2006.15059 (2020).Google Scholar
47. Jos Stam and Eric Languénou. 1996. Ray tracing in non-constant media. In EGSR.Google Scholar
48. Qilin Sun, Congli Wang, Fu Qiang, Dun Xiong, and Heidrich Wolfgang. 2021. End-to-End Complex Lens Design with Differentiable Ray Tracing. ACM TOG (2021).Google Scholar
49. Jeremy Teichman, Jenny Holzer, Bohdan Balko, Brent Fisher, and Leonard Buckley. 2013. Gradient index optics at DARPA. Technical Report. Institute for Defense Analyses.Google Scholar
50. Ethan Tseng, Ali Mosleh, Fahim Mannan, Karl St-Arnaud, Avinash Sharma, Yifan Peng, Alexander Braun, Derek Nowrouzezahrai, Jean-Francois Lalonde, and Felix Heide. 2021. Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design. ACM TOG (2021).Google Scholar
51. Eric Veach. 1998. Robust Monte Carlo methods for light transport simulation. Stanford University.Google ScholarDigital Library
52. Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021. Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time. ACM TOG (2021).Google ScholarDigital Library
53. ShiLi Wei, ZhengBo Zhu, ZiChao Fan, and DingLin Ma. 2020. Least-squares ray mapping method for freeform illumination optics design. Optics express (2020).Google Scholar
54. Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. 2021. Neural Fields in Visual Computing and Beyond. arXiv:2111.11426 (2021).Google Scholar
55. Yonghao Yue, Kei Iwasaki, Bing-Yu Chen, Yoshinori Dobashi, and Tomoyuki Nishita. 2014. Poisson-based continuous surface generation for goal-based caustics. ACM Transactions on Graphics (TOG) 33, 3 (2014), 1–7.Google ScholarDigital Library
56. Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM TOG (2020).Google Scholar