“Differentiable time-gated rendering” by Wu, Cai, Ramamoorthi and Zhao
Conference:
Type(s):
Title:
- Differentiable time-gated rendering
Session/Category Title: Light Interactions and Differentiable Rendering
Presenter(s)/Author(s):
Abstract:
The continued advancements of time-of-flight imaging devices have enabled new imaging pipelines with numerous applications. Consequently, several forward rendering techniques capable of accurately and efficiently simulating these devices have been introduced. However, general-purpose differentiable rendering techniques that estimate derivatives of time-of-flight images are still lacking. In this paper, we introduce a new theory of differentiable time-gated rendering that enjoys the generality of differentiating with respect to arbitrary scene parameters. Our theory also allows the design of advanced Monte Carlo estimators capable of handling cameras with near-delta or discontinuous time gates.We validate our theory by comparing derivatives generated with our technique and finite differences. Further, we demonstrate the usefulness of our technique using a few proof-of-concept inverse-rendering examples that simulate several time-of-flight imaging scenarios.
References:
1. Sai Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased Warped-Area Sampling for Differentiable Rendering. ACM Trans. Graph. 39, 6 (2020), 245:1–245:18.
2. Subrahmanyan Chandrasekhar. 1960. Radiative transfer. Courier Corporation.
3. Wenzheng Chen, Fangyin Wei, Kiriakos N. Kutulakos, Szymon Rusinkiewicz, and Felix Heide. 2020. Learned feature embeddings for non-line-of-sight imaging and recognition. ACM Trans. Graph. 39, 6 (2020).
4. BB Das, KM Yoo, and RR Alfano. 1993. Ultrafast time-gated imaging in thick tissues: a step toward optical mammography. Optics letters 18, 13 (1993), 1092–1094.
5. Ofer David, Norman S Kopeika, and Boaz Weizer. 2006. Range gated active night vision system for automobiles. Applied optics 45, 28 (2006), 7248–7254.
6. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An evaluation of computational imaging techniques for heterogeneous inverse scattering. In European Conference on Computer Vision. Springer, 685–701.
7. Yoav Grauer and Ezri Sonn. 2015. Active gated imaging for automotive safety applications. In Video Surveillance and Transportation Imaging Applications 2015, Vol. 9407. International Society for Optics and Photonics, 94070F.
8. Tobias Gruber, Frank Julca-Aguilar, Mario Bijelic, and Felix Heide. 2019. Gated2depth: Real-time dense lidar from gated images. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1506–1516.
9. Felix Heide, Matthew O’Toole, Kai Zang, David B. Lindell, Steven Diamond, and Gordon Wetzstein. 2019. Non-line-of-sight imaging with partial occluders and surface normals. ACM Trans. Graph. 38, 3 (2019), 22:1–22:10.
10. Felix Heide, Lei Xiao, Wolfgang Heidrich, and Matthias B Hullin. 2014. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3222–3229.
11. Julian Iseringhausen and Matthias B. Hullin. 2020. Non-line-of-sight reconstruction using efficient transient rendering. ACM Trans. Graph. 39, 1 (2020), 8:1–8:14.
12. Adrian Jarabo. 2012. Femto-photography: Visualizing light in motion. Universidad de Zaragoza (2012).
13. Adrian Jarabo and Victor Arellano. 2018. Bidirectional Rendering of Vector Light Transport. Computer Graphics Forum (2018).
14. Adrian Jarabo, Julio Marco, Adolfo Muñoz, Raul Buisan, Wojciech Jarosz, and Diego Gutierrez. 2014. A Framework for Transient Rendering. ACM Trans. Graph. 33, 6 (2014), 177:1–177:10.
15. Adrian Jarabo, Belen Masia, Julio Marco, and Diego Gutierrez. 2017. Recent advances in transient imaging: A computer graphics and vision perspective. Visual Informatics 1, 1 (2017), 65–79.
16. James T. Kajiya. 1986. The rendering equation. SIGGRAPH Comput. Graph. 20, 4 (1986), 143–150.
17. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
18. Ahmed Kirmani, Tyler Hutchison, James Davis, and Ramesh Raskar. 2009. Looking around the corner using transient imaging. In 2009 IEEE 12th International Conference on Computer Vision. IEEE, 159–166.
19. Ahmed Kirmani, Tyler Hutchison, James Davis, and Ramesh Raskar. 2011. Looking around the corner using ultrafast transient imaging. International journal of computer vision 95, 1 (2011), 13–28.
20. Joseph R Lakowicz, Henryk Szmacinski, Kazimierz Nowaczyk, Klaus W Berndt, and Michael Johnson. 1992. Fluorescence lifetime imaging. Analytical biochemistry 202, 2 (1992), 316–330.
21. Martin Laurenzis and Andreas Velten. 2014. Nonline-of-sight laser gated viewing of scattered photons. Optical Engineering 53, 2 (2014), 023102.
22. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo ray tracing through edge sampling. ACM Trans. Graph. 37, 6 (2018), 222:1–222:11.
23. David B. Lindell, Gordon Wetzstein, and Matthew O’Toole. 2019. Wave-based non-line-of-sight imaging using fast f-k migration. ACM Trans. Graph. 38, 4 (2019), 116.
24. Xiaochun Liu, Sebastian Bauer, and Andreas Velten. 2020. Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems. Nature Communications 11, 1 (2020), 1–13.
25. Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jarabo, Diego Gutierrez, and Andreas Velten. 2019. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature 572, 7771 (2019), 620–623.
26. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. 38, 6 (2019), 228:1–228:14.
27. Julio Marco, Ibón Guillén, Wojciech Jarosz, Diego Gutierrez, and Adrian Jarabo. 2019. Progressive transient photon beams. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 19–30.
28. Nikhil Naik, Shuang Zhao, Andreas Velten, Ramesh Raskar, and Kavita Bala. 2011. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Trans. Graph. 30, 6 (2011), 171:1–171:10.
29. Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering. ACM Trans. Graph. 39, 4 (2020), 146:1–146:15.
30. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A retargetable forward and inverse renderer. ACM Trans. Graph. 38, 6 (2019), 203:1–203:17.
31. Matthew O’Toole, Felix Heide, Lei Xiao, Matthias B. Hullin, Wolfgang Heidrich, and Kiriakos N. Kutulakos. 2014. Temporal frequency probing for 5D transient analysis of global light transport. ACM Trans. Graph. 33, 4 (2014), 87:1–87:11.
32. Matthew O’Toole, David B Lindell, and Gordon Wetzstein. 2018. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 7696 (2018), 338–341.
33. Xian Pan, Victor Arellano, and Adrian Jarabo. 2019. Transient instant radiosity for efficient time-resolved global illumination. Computers & Graphics 83 (2019), 107–113.
34. Mark Pauly, Thomas Kollig, and Alexander Keller. 2000. Metropolis light transport for participating media. In Rendering Techniques 2000. Springer, 11–22.
35. Adithya Pediredla, Akshat Dave, and Ashok Veeraraghavan. 2019a. Snlos: Non-line-of-sight scanning through temporal focusing. In 2019 IEEE International Conference on Computational Photography (ICCP). IEEE, 1–13.
36. Adithya Pediredla, Ashok Veeraraghavan, and Ioannis Gkioulekas. 2019b. Ellipsoidal path connections for time-gated rendering. ACM Trans. Graph. 38, 4 (2019), 38:1–38:12.
37. Adithya Kumar Pediredla, Mauro Buttafava, Alberto Tosi, Oliver Cossairt, and Ashok Veeraraghavan. 2017. Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner. In 2017 IEEE International Conference on Computational Photography (ICCP). 1–12.
38. Ramesh Raskar, Andreas Velten, Sebastian Bauer, and Tristan Swedish. 2020. Seeing around corners using time of flight. In ACM SIGGRAPH 2020 Courses. 1–97.
39. Osborne Reynolds. 1903. Papers on mechanical and physical subjects: the sub-mechanics of the universe. Vol. 3. The University Press.
40. Guy Satat, Matthew Tancik, and Ramesh Raskar. 2018. Towards photography through realistic fog. In 2018 IEEE International Conference on Computational Photography (ICCP). IEEE, 1–10.
41. Charles Saunders, John Murray-Bruce, and Vivek K Goyal. 2019. Computational periscopy with an ordinary digital camera. Nature 565, 7740 (2019), 472–475.
42. Adam Smith, James Skorupski, and James Davis. 2008. Transient rendering. (2008).
43. Simon Telezhkin. 2021. Low poly tree 3D model. https://done3d.com/low-poly-tree/
44. Christos Thrampoulidis, Gal Shulkind, Feihu Xu, William T Freeman, Jeffrey H Shapiro, Antonio Torralba, Franco NC Wong, and Gregory W Wornell. 2018. Exploiting occlusion in non-line-of-sight active imaging. IEEE Transactions on Computational Imaging 4, 3 (2018), 419–431.
45. Chia-Yin Tsai, Kiriakos N Kutulakos, Srinivasa G Narasimhan, and Aswin C Sankaranarayanan. 2017. The geometry of first-returning photons for non-line-of-sight imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7216–7224.
46. Chia-Yin Tsai, Aswin C Sankaranarayanan, and Ioannis Gkioulekas. 2019. Beyond volumetric albedo-a surface optimization framework for non-line-of-sight imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1545–1555.
47. Eric Veach. 1997. Robust Monte Carlo methods for light transport simulation. Vol. 1610. Stanford University PhD thesis.
48. Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021. Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time. ACM Trans. Graph. 40, 4 (2021), 108:1–108:14.
49. Shumian Xin, Sotiris Nousias, Kiriakos N Kutulakos, Aswin C Sankaranarayanan, Srinivasa G Narasimhan, and Ioannis Gkioulekas. 2019. A theory of Fermat paths for non-line-of-sight shape reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6800–6809.
50. Sean I. Young, David B. Lindell, Bernd Girod, David Taubman, and Gordon Wetzstein. 2020. Non-line-of-sight Surface Reconstruction Using the Directional Light-cone Transform. In Proc. CVPR.
51. Tizian Zeltner, Sébastien Speierer, Iliyan Georgiev, and Wenzel Jakob. 2021. Monte Carlo Estimators for Differential Light Transport. ACM Trans. Graph. 40, 4 (2021).
52. Cheng Zhang, Zhao Dong, Michael Doggett, and Shuang Zhao. 2021a. Antithetic sampling for Monte Carlo differentiable rendering. ACM Trans. Graph. 40, 4 (2021), 77:1–77:12.
53. Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-space differentiable rendering. ACM Trans. Graph. 39, 4 (2020), 143:1–143:19.
54. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A differential theory of radiative transfer. ACM Trans. Graph. 38, 6 (2019), 227:1–227:16.
55. Cheng Zhang, Zihan Yu, and Shuang Zhao. 2021b. Path-space differentiable rendering of participating media. ACM Trans. Graph. 40, 4 (2021), 76:1–76:15.
