“Single-photon 3D imaging with deep sensor fusion” by Lindell, O’Toole and Wetzstein

  • ©David B. Lindell, Matthew O’Toole, and Gordon Wetzstein



Entry Number: 113

Session Title:

    Computational Cameras


    Single-photon 3D imaging with deep sensor fusion




    Sensors which capture 3D scene information provide useful data for tasks in vehicle navigation, gesture recognition, human pose estimation, and geometric reconstruction. Active illumination time-of-flight sensors in particular have become widely used to estimate a 3D representation of a scene. However, the maximum range, density of acquired spatial samples, and overall acquisition time of these sensors is fundamentally limited by the minimum signal required to estimate depth reliably. In this paper, we propose a data-driven method for photon-efficient 3D imaging which leverages sensor fusion and computational reconstruction to rapidly and robustly estimate a dense depth map from low photon counts. Our sensor fusion approach uses measurements of single photon arrival times from a low-resolution single-photon detector array and an intensity image from a conventional high-resolution camera. Using a multi-scale deep convolutional network, we jointly process the raw measurements from both sensors and output a high-resolution depth map. To demonstrate the efficacy of our approach, we implement a hardware prototype and show results using captured data. At low signal-to-background levels, our depth reconstruction algorithm with sensor fusion outperforms other methods for depth estimation from noisy measurements of photon arrival times.


    1. E. Abreu, M. Lightstone, S.K. Mitra, and K. Arakawa. 1996. A new efficient approach for the removal of impulse noise from highly corrupted images. IEEE Trans. Image Process. 5, 6 (1996), 1012–1025. Google ScholarDigital Library
    2. S. Achar, J.R. Bartels, W.L. Whittaker, K.N. Kutulakos, and S.G. Narasimhan. 2017. Epipolar time-of-flight imaging. ACM Trans. Graph. (SIGGRAPH) 36, 4 (2017), 37. Google ScholarDigital Library
    3. Y. Altmann, R. Aspden, M. Padgett, and S. McLaughlin. 2017. A Bayesian Approach to Denoising of Single-Photon Binary Images. IEEE Trans. Computat. Imaging 3, 3 (Sept 2017), 460–471.Google Scholar
    4. A. Bleiweiss and M. Werman. 2009. Fusing time-of-flight depth and color for real-time segmentation and tracking. In Dynamic 3D Imaging. 58–69. Google ScholarDigital Library
    5. S. Burri, H. Homulle, C. Bruschini, and E. Charbon. 2016. LinoSPAD: A time-resolved 256X1 CMOS SPAD line sensor system featuring 64 FPGA-based TDC channels running at up to 8.5 giga-events per second. In Proc. SPIE, Vol. 9899. 98990D.Google Scholar
    6. D. Chan, H. Buisman, C. Theobalt, and S. Thrun. 2008. A noise-aware filter for real-time depth upsampling. In Workshop on Multi-Camera and Multi-Modal Sensor Fusion Algorithms and Applications.Google Scholar
    7. Q. Chen and V. Koltun. 2013. A simple model for intrinsic image decomposition with depth cues. In Proc. ICCV. 241–248. Google ScholarDigital Library
    8. H. Dautet, P. Deschamps, B. Dion, A.D. MacGregor, D. MacSween, R.J. McIntyre, C. Trottier, and P.P. Webb. 1993. Photon counting techniques with silicon avalanche photodiodes. Applied optics 32, 21 (1993), 3894–3900.Google Scholar
    9. J. Diebel and S. Thrun. 2006. An application of Markov Random Fields to range sensing. In Prac. NIPS. 291–298. Google ScholarDigital Library
    10. D. Ferstl, C. Reinbacher, R. Ranftl, M. Rüther, and H. Bischof. 2013. Image guided depth upsampling using anisotropic total generalized variation. In Proc. CVPR. 993–1000. Google ScholarDigital Library
    11. P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox. 2012. RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments. The International Journal of Robotics Research 31, 5 (2012), 647–663. Google ScholarDigital Library
    12. R. Horaud, M. Hansard, G. Evangelidis, and C. Ménier. 2016. An overview of depth cameras and range scanners based on time-of-flight technologies. Machine Vision and Applications 27, 7 (2016), 1005–1020. Google ScholarDigital Library
    13. T. Hui, C.C. Loy, and X. Tang. 2016. Depth map super-resolution by deep multi-scale guidance. In Proc. ECCV. 353–369.Google Scholar
    14. D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
    15. A. Kirmani, D. Venkatraman, D. Shin, A. Colaço, F.N.C. Wong, J.H. Shapiro, and V.K. Goyal. 2014. First-photon imaging. Science 343, 6166 (2014), 58–61.Google Scholar
    16. A. Kolb, E. Barth, R. Koch, and R. Larsen. 2009. Time-of-flight sensors in computer graphics. In Eurographics (STARs). 119–134.Google Scholar
    17. J. Kopf, M.F. Cohen, D. Lischinski, and M. Uyttendaele. 2007. Joint bilateral upsampling. In ACM Trans. Graph. (SIGGRAPH), Vol. 26. 96. Google ScholarDigital Library
    18. M. Koskinen, J.T. Kostamovaara, and R.A. Myllylae. 1992. Comparison of continuous-wave and pulsed time-of-flight laser range-finding techniques. In Proc. SPIE 1614. 296–305.Google Scholar
    19. Y. Li, J. Huang, N. Ahuja, and M. Yang. 2016. Deep joint image filtering. In Proc. ECCV. 154–169.Google Scholar
    20. G. Lin, A. Milan, C. Shen, and I. Reid. 2017. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In Proc. CVPR.Google Scholar
    21. D.B. Lindell, M. O’Toole, and G. Wetzstein. 2018. Towards transient imaging at interactive rates with single-photon detectors. In Proc. ICCP.Google Scholar
    22. J. Marco, Q. Hernandez, A. Muñoz, Y. Dong, A. Jarabo, M.H. Kim, X. Tong, and D. Gutierrez. 2017. DeepToF: Off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6 (2017), 219:1–219:12. Google ScholarDigital Library
    23. A. McCarthy, X. Ren, A. Della Frera, N.R. Gemmell, N.J. Krichel, C. Scarcella, A. Ruggeri, A. Tosi, and G.S. Buller. 2013. Kilometer-range depth imaging at 1550 nm wavelength using an InGaAs/InP single-photon avalanche diode detector. Optics express 21, 19 (2013), 22098–22113.Google Scholar
    24. D. O’Connor and D. Philips. 1984. Time-correlated single photon counting. Academic Press.Google Scholar
    25. M. O’Toole, S. Achar, S.G. Narasimhan, and K.N. Kutulakos. 2015. Homogeneous codes for energy-efficient illumination and imaging. ACM Trans. Graph. (SIGGRAPH) 34, 4, Article 35 (2015). Google ScholarDigital Library
    26. M. O’Toole, F. Heide, D.B. Lindell, K. Zang, S. Diamond, and G. Wetzstein. 2017. Reconstructing transient images from single-photon sensors. In Proc. CVPR.Google Scholar
    27. J. Park, H. Kim, Y. Tai, M.S. Brown, and I. Kweon. 2011. High quality depth map upsampling for 3D-TOF cameras. In Proc. ICCV 1623–1630. Google ScholarDigital Library
    28. A.M. Pawlikowska, A. Halimi, R.A. Lamb, and G.S. Buller. 2017. Single-photon three-dimensional imaging at up to 10 kilometers range. Optics Express 25, 10 (2017), 11919–11931.Google ScholarCross Ref
    29. C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun. 2017. Large kernel matters- Improve semantic segmentation by global convolutional network. In Proc. CVPR. 1743–1751.Google Scholar
    30. G. Petschnigg, R. Szeliski, M. Agrawala, M. Cohen, H. Hoppe, and K. Toyama. 2004. Digital photography with flash and no-flash image pairs. ACM Trans. Graph. (SIGGRAPH) 23, 3 (2004), 664–672. Google ScholarDigital Library
    31. E.F. Pettersen, T.D. Goddard, C.C. Huang, G.S. Couch, D.M. Greenblatt, E.C. Meng, and T.E. Ferrin. 2004. UCSF Chimera-a visualization system for exploratory research and analysis. Journal of computational chemistry 25, 13 (2004), 1605–1612.Google ScholarCross Ref
    32. J. Rapp and V.K. Goyal. 2017. A few photons among many: Unmixing signal and noise for photon-efficient active imaging. IEEE Trans. Computat. Imaging 3 (2017), 445–459. Issue 3.Google ScholarCross Ref
    33. D. Renker. 2006. Geiger-mode avalanche photodiodes, history, properties and problems. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 567, 1 (2006), 48–56.Google ScholarCross Ref
    34. D. Scharstein and C. Pal. 2007. Learning conditional random fields for stereo. In Proc. CVPR. 1–8.Google Scholar
    35. D. Shin, A. Kirmani, V.K. Goyal, and J.H. Shapiro. 2015. Photon-efficient computational 3-D and reflectivity imaging with single-photon detectors. IEEE Trans. Computat. Imaging 1, 2 (2015), 112–125.Google ScholarCross Ref
    36. D. Shin, F. Xu, D. Venkatraman, R. Lussana, F. Villa, F. Zappa, V.K. Goyal, F.N.C. Wong, and J.H. Shapiro. 2016. Photon-efficient imaging with a single-photon camera. Nature Communications 7 (2016).Google Scholar
    37. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. 2012. Indoor segmentation and support inference from RGBD images. In Proc. ECCV. Google ScholarDigital Library
    38. S. Su, F. Heide, G. Wetzstein, and W. Heidrich. 2018. Deep end-to-end time-of-flight imaging. In Proc. CVPR.Google Scholar
    39. R. Tobin, A. Halimi, A. McCarthy, X. Ren, K.J. McEwan, S. McLaughlin, and G.S. Buller, 2017. Long-range depth profiling of camouflaged targets using single-photon detection. Optical Engineering 57 (2017).Google Scholar
    40. Q. Yang, R. Yang, J. Davis, and D. Nistér. 2007. Spatial-depth super resolution for range images. In Proc. CVPR. 1–8.Google Scholar

ACM Digital Library Publication: