“Ellipsoidal path connections for time-gated rendering” by Pediredla, Veeraraghavan and Gkioulekas
Conference:
Type(s):
Title:
- Ellipsoidal path connections for time-gated rendering
Session/Category Title: Light Science
Presenter(s)/Author(s):
Abstract:
During the last decade, we have been witnessing the continued development of new time-of-flight imaging devices, and their increased use in numerous and varied applications. However, physics-based rendering techniques that can accurately simulate these devices are still lacking: while existing algorithms are adequate for certain tasks, such as simulating transient cameras, they are very inefficient for simulating time-gated cameras because of the large number of wasted path samples. We take steps towards addressing these deficiencies, by introducing a procedure for efficiently sampling paths with a predetermined length, and incorporating it within rendering frameworks tailored towards simulating time-gated imaging. We use our open-source implementation of the above to empirically demonstrate improved rendering performance in a variety of applications, including simulating proximity sensors, imaging through occlusions, depth-selective cameras, transient imaging in dynamic scenes, and non-line-of-sight imaging.
References:
1. 2019 FESCA-100 Femtosecond streak camera. https://www.hamamatsu.com/us/en/product/photometry-systems/streak-camera/fesca-100-femtosecond-streak-camera/index.html. Accessed: 01-12-2019.Google Scholar
2. 2019 Laser Range-Gated Imaging for Imaging at Long Ranges and Through Obscurants. http://www.sensorsinc.com/applications/military/laser-range-gating. Accessed: 01-12-2019.Google Scholar
3. Marco Ament, Christoph Bergmann, and Daniel Weiskopf. 2014. Refractive radiative transfer equation. ACM Transactions on Graphics (TOG) 33, 2 (2014), 17. Google ScholarDigital Library
4. Luke Anderson, Tzu-Mao Li, Jaakko Lehtinen, and Frédo Durand. 2017. Aether: An embedded domain specific sampling language for Monte Carlo rendering. ACM Transactions on Graphics (TOG) 36, 4 (2017), 99. Google ScholarDigital Library
5. Victor Arellano, Diego Gutierrez, and Adrian Jarabo. 2017. Fast back-projection for non-line of sight reconstruction. Optics Express 25, 10 (2017), 11574–11583.Google ScholarCross Ref
6. Ian M Baker, Stuart S Duncan, and Jeremy W Copley. 2004. A low-noise laser-gated imaging system for long-range target identification. In Infrared Technology and Applications XXX, Vol. 5406. International Society for Optics and Photonics, 133–145.Google ScholarCross Ref
7. Mauro Buttafava, Jessica Zeman, Alberto Tosi, Kevin Eliceiri, and Andreas Velten. 2015. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Optics express 23, 16 (2015), 20997–21011.Google Scholar
8. Lucrezia Cester, Ashley Lyons, Maria Braidotti, and Daniele Faccio. 2019. Time-of-Flight Imaging at 10 ps Resolution with an ICCD Camera. Sensors 19, 1 (2019), 180.Google ScholarCross Ref
9. Susan Chan, Ryan E Warburton, Genevieve Gariepy, Jonathan Leach, and Daniele Faccio. 2017. Non-line-of-sight tracking of people at long range. Optics express 25, 9 (2017), 10109–10117.Google Scholar
10. Subrahmanyan Chandrasekhar. 1960. Radiative transfer. Dover Publication.Google Scholar
11. BB Das, KM Yoo, and RR Alfano. 1993. Ultrafast time-gated imaging in thick tissues: a step toward optical mammography. Optics letters 18, 13 (1993), 1092–1094.Google Scholar
12. Ofer David, Norman S Kopeika, and Boaz Weizer. 2006. Range gated active night vision system for automobiles. Applied optics 45, 28 (2006), 7248–7254.Google Scholar
13. Philip Dutre, Philippe Bekaert, and Kavita Bala. 2006. Advanced global illumination. AK Peters/CRC Press. Google ScholarDigital Library
14. Daniele Faccio and Andreas Velten. 2018. A trillion frames per second: the techniques and applications of light-in-flight photography. Reports on Progress in Physics 81, 10 (2018), 105901.Google ScholarCross Ref
15. Genevieve Gariepy, Nikola Krstajić, Robert Henderson, Chunyong Li, Robert R Thomson, Gerald S Buller, Barmak Heshmat, Ramesh Raskar, Jonathan Leach, and Daniele Faccio. 2015. Single-photon sensitive light-in-fight imaging. Nature communications 6 (2015), 6021.Google Scholar
16. Ioannis Gkioulekas, Anat Levin, Frédo Durand, and Todd Zickler. 2015. Micron-scale light transport decomposition using interferometry. ACM Transactions on Graphics (ToG) 34, 4 (2015), 37. Google ScholarDigital Library
17. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An Evaluation of Computational Imaging Techniques for Heterogeneous Inverse Scattering. European Conference on Computer Vision (2016).Google Scholar
18. Yoav Grauer and Ezri Sonn. 2015. Active gated imaging for automotive safety applications. In Video Surveillance and Transportation Imaging Applications 2015, Vol. 9407. International Society for Optics and Photonics, 94070F.Google Scholar
19. Tobias Gruber, Frank Julca-Aguilar, Mario Bijelic, Werner Ritter, Klaus Dietmayer, and Felix Heide. 2019. Gated2Depth: Real-time Dense Lidar from Gated Images. arXiv preprint arXiv.1902.04997 (2019).Google Scholar
20. MohitGupta, Shree K Nayar, Matthias B Hullin, and Jaime Martin. 2015. Phasor imaging: A generalization of correlation-based time-of-flight imaging. ACM Transactions on Graphics (ToG) 34, 5 (2015), 156. Google ScholarDigital Library
21. Mohit Gupta, Andreas Velten, Shree K Nayar, and Eric Breitbach. 2018. What are optimal coding functions for time-of-flight imaging? ACM Transactions on Graphics (TOG) 37, 2 (2018), 13. Google ScholarDigital Library
22. Toshiya Hachisuka, Anton S Kaplanyan, and Carsten Dachsbacher. 2014. Multiplexed metropolis light transport. ACM Transactions on Graphics (TOG) 33, 4 (2014), 100. Google ScholarDigital Library
23. Toshiya Hachisuka, Shinji Ogaki, and Henrik Wann Jensen. 2008. Progressive photon mapping. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 130. Google ScholarDigital Library
24. Felix Heide, Matthias B Hullin, James Gregson, and Wolfgang Heidrich. 2013. Low-budget transient imaging using photonic mixer devices. ACM Transactions on Graphics (ToG) 32, 4 (2013), 45. Google ScholarDigital Library
25. Felix Heide, Matthew O’Toole, Kai Zhang, David Lindell, Steven Diamond, and Gordon Wetzstein. 2019. Non-line-of-sight Imaging with Partial Occluders and Surface Normals. ACM Transactions on Graphics (ToG) (2019). Google ScholarDigital Library
26. Felix Heide, Lei Xiao, Wolfgang Heidrich, and Matthias B Hullin. 2014. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3222–3229. Google ScholarDigital Library
27. Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.Google Scholar
28. Wenzel Jakob and Steve Marschner. 2012. Manifold exploration: A markov chain monte carlo technique for rendering scenes with difficult specular transport. ACM Transactions on Graphics (2012). Google ScholarDigital Library
29. Adrian Jarabo. 2012. Femto-photography: Visualizing light in motion. Master’s thesis, Universidad de Zaragoza (2012).Google Scholar
30. Adrian Jarabo, Julio Marco, Adolfo Muñoz, Raul Buisan, Wojciech Jarosz, and Diego Gutierrez. 2014. A framework for transient rendering. ACM Transactions on Graphics (ToG) 33, 6 (2014), 177. Google ScholarDigital Library
31. Adrian Jarabo, Belen Masia, Julio Marco, and Diego Gutierrez. 2017. Recent advances in transient imaging: A computer graphics and vision perspective. Visual Informatics 1, 1 (2017), 65–79.Google ScholarCross Ref
32. Henrik Wann Jensen. 2001. Realistic image synthesis using photon mapping. AK Peters/CRC Press. Google ScholarDigital Library
33. Ahmed Kirmani, Tyler Hutchison, James Davis, and Ramesh Raskar. 2009. Looking around the corner using transient imaging. In 2009 IEEE 12th International Conference on Computer Vision. IEEE, 159–166.Google ScholarCross Ref
34. Eric P Lafortune and Yves D Willems. 1996. Rendering participating media with bidirectional path tracing. In Rendering Techniques’ 96. Springer, 91–100. Google ScholarDigital Library
35. Joseph R Lakowicz, Henryk Szmacinski, Kazimierz Nowaczyk, Klaus W Berndt, and Michael Johnson. 1992. Fluorescence lifetime imaging. Analytical biochemistry 202, 2 (1992), 316–330.Google Scholar
36. Martin Laurenzis and Andreas Velten. 2014. Nonline-of-sight laser gated viewing of scattered photons. Optical Engineering 53, 2 (2014), 023102.Google ScholarCross Ref
37. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable monte carlo ray tracing through edge sampling. In SIGGRAPH Asia 2018 Technical Papers. ACM, 222. Google ScholarDigital Library
38. Jingyu Lin, Yebin Liu, Matthias B Hullin, and Qionghai Dai. 2014. Fourier analysis on transient imaging with a multifrequency time-of-flight camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3230–3237. Google ScholarDigital Library
39. Julio Marco, Ibón Guillén, Wojciech Jarosz, Diego Gutierrez, and Adrian Jarabo. 2018. Progressive Transient Photon Beams. arXiv preprint arXiv:1805.09562 (2018).Google Scholar
40. Julio Marco, Quercus Hernandez, Adolfo Muñoz, Yue Dong, Adrian Jarabo, Min H Kim, Xin Tong, and Diego Gutierrez. 2017a. DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Transactions on Graphics (TOG) 36, 6 (2017), 219. Google ScholarDigital Library
41. Julio Marco, Wojciech Jarosz, Diego Gutierrez, and Adrian Jarabo. 2017b. Transient photon beams. In ACM SIGGRAPH 2017 Posters. ACM, 52. Google ScholarDigital Library
42. Stephen Robert Marschner and Donald P Greenberg. 1998. Inverse rendering for computer graphics. Cornell University.Google ScholarDigital Library
43. Nikhil Naik, Shuang Zhao, Andreas Velten, Ramesh Raskar, and Kavita Bala. 2011. Single View Reflectance Capture Using Multiplexed Scattering and Time-of-flight Imaging. ACM Trans. Graph. 30, 6 (2011), 171:1–171:10. Google ScholarDigital Library
44. Matthew O’Toole, Felix Heide, Lei Xiao, Matthias B Hullin, Wolfgang Heidrich, and Kiriakos N Kutulakos. 2014. Temporal frequency probing for 5D transient analysis of global light transport. ACM Transactions on Graphics (ToG) 33, 4 (2014), 87. Google ScholarDigital Library
45. Matthew O’Toole, David B Lindell, and Gordon Wetzstein. 2018. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 7696 (2018), 338.Google Scholar
46. Adithya Pediredla, Mauro Buttafava, Alberto Tosi, Oliver Cossairt, and Ashok Veeraraghavan. 2017a. Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner. In 2017 IEEE International Conference on Computational Photography (ICCP). IEEE, 1–12.Google ScholarCross Ref
47. Adithya Pediredla, Akshat Dave, and Ashok Veeraraghavan. 2019a. SNLOS: Non-line-of-sight Scanning through Temporal Focusing. (2019).Google Scholar
48. Adithya Pediredla, Nathan Matsuda, Oliver Cossairt, and Ashok Veeraraghavan. 2017b. Linear systems approach to identifying performance bounds in indirect imaging. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on. IEEE, 6235–6239.Google ScholarCross Ref
49. Adithya Pediredla, Ashok Veeraraghavan, and Gkioulekas Ioannis. 2019b. Mitsuba Time-of-Flight renderer. https://github.com/cmu-ci-lab/MitsubaToFRenderer.Google Scholar
50. Christoph Peters, Jonathan Klein, Matthias B Hullin, and Reinhard Klein. 2015. Solving trigonometric moment problems for fast transient imaging. ACM Transactions on Graphics (TOG) 34, 6 (2015), 220. Google ScholarDigital Library
51. Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically based rendering: From theory to implementation. Morgan Kaufmann. Google ScholarDigital Library
52. Guy Satat, Barmak Heshmat, Dan Raviv, and Ramesh Raskar. 2016. All photons imaging through volumetric scattering. Scientific reports 6 (2016), 33946.Google Scholar
53. Guy Satat, Matthew Tancik, and Ramesh Raskar. 2018. Towards photography through realistic fog. In Computational Photography (ICCP), 2018 IEEE International Conference on. IEEE, 1–10.Google ScholarCross Ref
54. B Schmidt, S Laimgruber, W Zinth, and P Gilch. 2003. A broadband Kerr shutter for femtosecond fluorescence spectroscopy. Applied Physics B 76, 8 (2003), 809–814.Google ScholarCross Ref
55. Shikhar Shrestha, Felix Heide, Wolfgang Heidrich, and Gordon Wetzstein. 2016. Computational imaging with multi-camera time-of-flight systems. ACM Transactions on Graphics (TOG) 35, 4 (2016), 33. Google ScholarDigital Library
56. Adam Smith, James Skorupski, and James Davis. 2008. Transient rendering. (2008).Google Scholar
57. Ryuichi Tadano, Adithya Kumar Pediredla, and Ashok Veeraraghavan. 2015. Depth selective camera: A direct, on-chip, programmable technique for depth selectivity in photography. In Proceedings of the IEEE International Conference on Computer Vision. 3595–3603. Google ScholarDigital Library
58. Akira Takahashi, Mitsunori Nishizawa, Yoshinori Inagaki, Musubu Koishi, and Katsuyuki Kinoshita. 1994. New femtosecond streak camera with temporal resolution of 180 fs. In Generation, amplification, and measurement of ultrashort laser pulses, Vol. 2116. International Society for Optics and Photonics, 275–285.Google Scholar
59. Christos Thrampoulidis, Gal Shulkind, Feihu Xu, William T Freeman, Jeffrey Shapiro, Antonio Torralba, Franco Wong, and Gregory Wornell. 2018. Exploiting occlusion in non-line-of-sight active imaging. IEEE Transactions on Computational Imaging (2018).Google ScholarCross Ref
60. Chia-Yin Tsai, Kiriakos N Kutulakos, Srinivasa G Narasimhan, and Aswin C Sankaranarayanan. 2017. The geometry of first-returning photons for non-line-of-sight imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7216–7224.Google ScholarCross Ref
61. Chia-Yin Tsai, Aswin C. Sankaranarayanan, and Ioannis Gkioulekas. 2019. Beyond Volumetric Albedo-A Surface Optimization Framework for Non-Line-of-Sight Imaging. In CVPR.Google Scholar
62. Eric Veach. 1997. Robust monte carlo methods for light transport simulation. Number 1610. Stanford University PhD thesis. Google ScholarDigital Library
63. Eric Veach and Leonidas Guibas. 1995a. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques. Springer, 145–167.Google Scholar
64. Eric Veach and Leonidas J Guibas. 1995b. Optimally combining sampling techniques for Monte Carlo rendering. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. ACM, 419–428. Google ScholarDigital Library
65. Eric Veach and Leonidas J Guibas. 1997. Metropolis light transport. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 65–76. Google ScholarDigital Library
66. Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G Bawendi, and Ramesh Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature communications 3 (2012), 745.Google Scholar
67. Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Chinmaya Joshi, Everett Lawson, Moungi Bawendi, Diego Gutierrez, and Ramesh Raskar. 2013. Femto-photography: capturing and visualizing the propagation of light. ACM Transactions on Graphics (ToG) 32, 4 (2013), 44. Google ScholarDigital Library
68. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet models for refraction through rough surfaces. In Proceedings of the 18th Eurographics conference on Rendering Techniques. Eurographics Association, 195–206. Google ScholarDigital Library
69. Di Wu, Andreas Velten, Matthew O’Toole, Belen Masia, Amit Agrawal, Qionghai Dai, and Ramesh Raskar. 2014a. Decomposing global light transport using time of flight imaging. International journal of computer vision 107, 2 (2014), 123–138. Google ScholarDigital Library
70. Di Wu, Gordon Wetzstein, Christopher Barsi, Thomas Willwacher, Qionghai Dai, and Ramesh Raskar. 2014b. Ultra-fast lensless computational imaging through 5D frequency analysis of time-resolved light transport. International journal of computer vision 110, 2 (2014), 128–140. Google ScholarDigital Library
71. Shumian Xin, Sotiris Nousias, Kyriakos N. Kutulakos, Aswin C. Sankaranarayanan, Srinivasa G. Narasimhan, and Ioannis Gkioulekas. 2019. A Theory of Fermat Paths for Non-Line-of-Sight Shape Reconstruction. In CVPR.Google Scholar
72. Pingping Zhan, Junyi Tong, Wenjiang Tan, and Shichao Xu. 2016. Comparison between traditional and heterodyned optical Kerr gated imaging for CS 2 and Te glass Kerr media. Optical Engineering 55, 11 (2016), 110502.Google ScholarCross Ref
73. Matthias Zwicker, Wojciech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and S-E Yoon. 2015. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. In Computer graphics forum, Vol. 34. Wiley Online Library, 667–681. Google ScholarDigital Library
