“Non-Line-of-Sight Imaging With Partial Occluders and Surface Normals” by Heide, O’Toole, Zang, Lindell, Diamond, et al. …

  • ©Felix Heide, Matthew O'Toole, Kai Zang, David B. Lindell, Steven Diamond, and Gordon Wetzstein

Conference:


Type:


Title:

    Non-Line-of-Sight Imaging With Partial Occluders and Surface Normals

Session/Category Title: Computational Imaging

Presenter(s)/Author(s):


Abstract:


    Imaging objects obscured by occluders is a significant challenge for many applications. A camera that could “see around corners” could help improve navigation and mapping capabilities of autonomous vehicles or make search and rescue missions more effective. Time-resolved single-photon imaging systems have recently been demonstrated to record optical information of a scene that can lead to an estimation of the shape and reflectance of objects hidden from the line of sight of a camera. However, existing non-line-of-sight (NLOS) reconstruction algorithms have been constrained in the types of light transport effects they model for the hidden scene parts. We introduce a factored NLOS light transport representation that accounts for partial occlusions and surface normals. Based on this model, we develop a factorization approach for inverse time-resolved light transport and demonstrate high-fidelity NLOS reconstructions for challenging scenes both in simulation and with an experimental NLOS imaging system.

References:


    1. Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand. 2015. Capturing the human figure through a wall. ACM Trans. Graph. 34, 6 (2015).
    2. Yoann Altmann, Stephen McLaughlin, Miles J. Padgett, Vivek K. Goyal, Alfred O. Hero, and Daniele Faccio. 2018. Quantum-inspired computational imaging. Science 361, 6403 (2018).
    3. Victor Arellano, Diego Gutierrez, and Adrian Jarabo. 2017. Fast back-projection for non-line of sight reconstruction. Opt. Express 25, 10 (2017), 11574–11583.
    4. Katherine Bouman, Vickie Ye, Adam Yedidia, Fredo Durand, Gregory W. Wornell, Antonio Torralba, and William T. Freeman. 2017. Turning corners into cameras: Principles and methods. In Proc. CVPR. 2270–2278.
    5. Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. 2011. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1 (2011), 1–122.
    6. Samuel Burri, Harald Homulle, Claudio Bruschini, and Edoardo Charbon. 2016. LinoSPAD: A time-resolved 256×1 CMOS SPAD line sensor system featuring 64 FPGA-based TDC channels running at up to 8.5 giga-events per second. Proc. SPIE 9899, 98990D–10 (2016).
    7. Mauro Buttafava, Jessica Zeman, Alberto Tosi, Kevin Eliceiri, and Andreas Velten. 2015. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Opt. Express 23, 16 (2015).
    8. Susan Chan, Ryan E. Warburton, Genevieve Gariepy, Jonathan Leach, and Daniele Faccio. 2017. Non-line-of-sight tracking of people at long range. Opt. Express 25, 9 (2017), 10109–10117.
    9. Quercus Hernandez, Diego Gutierrez, and Adrian Jarabo. 2017. A computational model of a single-photon avalanche diode sensor for transient imagingarXiv preprint arXiv:1703.02635 (2017).
    10. Adithya Kumar Pediredla, Mauro Buttafava, Alberto Tosi, Oliver Cossairt, and Ashok Veeraraghavan. 2017. Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner. In Proc. ICCP. 1–12.
    11. Genevieve Gariepy, Nikola Krstajic, Robert Henderson, Chunyong Li, Robert R. Thomson, Gerald S. Buller, Barmak Heshmat, Ramesh Raskar, Jonathan Leach, and Daniele Faccio. 2015. Single-photon sensitive light-in-fight imaging. Nat. Commun. 6, 6021 (2015).
    12. Genevieve Gariepy, Francesco Tonolini, Robert Henderson, Jonathan Leach, and Daniele Faccio. 2016. Detection and tracking of moving objects hidden from view. Nat. Photon. 10, 1 (2016).
    13. Otkrist Gupta, Thomas Willwacher, Andreas Velten, Ashok Veeraraghavan, and Ramesh Raskar. 2012. Reconstruction of hidden 3D shapes using diffuse reflections. OSA Opt. Express 20, 17 (2012).
    14. Felix Heide, Steven Diamond, David B. Lindell, and Gordon Wetzstein. 2018. Sub-picosecond photon-efficient 3D imaging using single-photon sensors. Sci. Rep. 8, 1 (2018).
    15. Felix Heide, Lei Xiao, Wolfgang Heidrich, and Matthias B. Hullin. 2014. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proc. CVPR.
    16. Adrian Jarabo, Julio Marco, Adolfo Muñoz, Raul Buisan, Wojciech Jarosz, and Diego Gutierrez. 2014. A framework for transient rendering. ACM Trans. Graph. 33, 6, Article 177 (Nov. 2014), 10 pages.
    17. Achuta Kadambi, Rafael Whyte, Ayush Bhandari, Lee Streeter, Christopher Barsi, Adrian Dorrington, and Ramesh Raskar. 2013. Coded time of flight cameras: Sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. (SIGGRAPH Asia) 32, 6 (2013).
    18. Avinash Kak and Malcolm Slaney. 1988. Principles of Computerized Tomographic Imaging. IEEE Press.
    19. Ori Katz, Pierre Heidmann, Mathias Fink, and Sylvain Gigan. 2014. Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations. Nat. Photon. 8, 10 (2014).
    20. Ahmed Kirmani, Tyler Hutchison, James Davis, and Ramesh Raskar. 2009. Looking around the corner using transient imaging. Proc. ICCV (2009), 159–166.
    21. Ahmed Kirmani, Dheera Venkatraman, Dongeek Shin, Andrea Colaco, Franco N. C. Wong, Jeffrey H. Shapiro, and Vivek K. Goyal. 2014. First-photon imaging. Science 343, 6166 (2014), 58–61.
    22. Jonathan Klein, Christoph Peters, Jaime Martin, Martin Laurenzis, and Matthias B. Hullin. 2016. Tracking objects outside the line of sight using 2D intensity images. Sci. Rep. 6 (2016), 177–188.
    23. David B. Lindell, Matthew O’Toole, and Gordon Wetzstein. 2018a. Single-photon 3D imaging with deep sensor fusion. ACM Trans. Graph. 4, 37 (2018).
    24. David B. Lindell, Matthew O’Toole, and Gordon Wetzstein. 2018b. Towards transient imaging at interactive rates with single-photon detectors. Proc. IEEE ICCP (2018).
    25. Nelson Max. 1995. Optical models for direct volume rendering. IEEE Trans. Vis. Comput. Graph. 1, 2 (1995), 99–108.
    26. Aongus McCarthy, Nils J. Krichel, Nathan R. Gemmell, Ximing Ren, Michael G. Tanner, Sander N. Dorenbos, Val Zwiller, Robert H. Hadfield, and Gerald S. Buller. 2013. Kilometer-range, high resolution depth imaging via 1560 nm wavelength single-photon detection. Opt. Express 21, 7 (2013), 8904–8915.
    27. Desmond O’Connor. 2012. Time-correlated Single Photon Counting. Academic Press.
    28. Matthew O’Toole, Felix Heide, David Lindell, Kai Zang, Stephen Diamond, and Gordon Wetzstein. 2017. Reconstructing transient images from single-photon sensors. Proc. CVPR (2017).
    29. Matthew O’Toole, David B. Lindell, and Gordon Wetzstein. 2018. Confocal non-line-of-sight imaging based on the light cone transform. Nature 555, 7696 (2018), 338–341.
    30. Joshua Rapp and Vivek K. Goyal. 2017. A few photons among many: Unmixing signal and noise for photon-efficient active imaging. IEEE Trans. Comput. Imag. 3, 3 (2017), 445–459.
    31. Albert Redo-Sanchez, Barmak Heshmat, Alireza Aghasi, Salman Naqvi, Mingjie Zhang, Justin Romberg, and Ramesh Raskar. 2016. Terahertz time-gated spectral imaging for content extraction through layered structures. Nat. Commun. 7 (2016).
    32. Brent Schwarz. 2010. LIDAR: Mapping the world in 3D. Nat. Photon. 4, 7 (2010), 1749–4885.
    33. Pradeep Sen, Billy Chen, Gaurav Garg, Stephen R. Marschner, Mark Horowitz, Marc Levoy, and Hendrik P. A. Lensch. 2005. Dual photography. ACM Trans. Graph. 24, 3 (2005), 745–755.
    34. Dongeek Shin, Feihu Xu, Dheera Venkatraman, Rudi Lussana, Federica Villa, Franco Zappa, Vivek K. Goyal, Franco N. C. Wong, and Jeffrey H. Shapiro. 2016. Photon-efficient imaging with a single-photon camera. Nat. Commun. 7, 12046 (2016), 1–7.
    35. Christos Thrampoulidis, Gal Shulkind, Feihu Xu, William T. Freeman, Jeffrey H. Shapiro, Antonio Torralba, Franco N. C. Wong, and Gregory W. Wornell. 2017. Exploiting occlusion in non-line-of-sight active imaging. (2017). arxiv:1711.06297.
    36. Chia-Yin Tsai, Kiriakos N. Kutulakos, Srinivasa G. Narasimhan, and Aswin C. Sankaranarayanan. 2017. The geometry of first-returning photons for non-line-of-sight imaging. In Proc. CVPR. 1–9.
    37. Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G. Bawendi, and Ramesh Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat. Commun. 3 (2012).
    38. Di Wu, Gordon Wetzstein, Christopher Barsi, Thomas Willwacher, Matthew O’Toole, Nikhil Naik, Qionghai Dai, Kyros Kutulakos, and Ramesh Raskar. 2012. Frequency analysis of transient light transport with applications in bare sensor imaging. In Proc. ECCV.
    39. Feihu Xu, Gal Shulkind, Christos Thrampoulidis, Jeffrey H. Shapiro, Antonio Torralba, Franco N. C. Wong, and Gregory W. Wornell. 2018. Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging. OSA Opt. Express 26, 8 (2018), 9945–9962.

ACM Digital Library Publication:


Overview Page: