“Single-photon 3D imaging with deep sensor fusion” by Lindell, O’Toole and Wetzstein
Conference:
Type(s):
Entry Number: 113
Title:
- Single-photon 3D imaging with deep sensor fusion
Session/Category Title: Computational Cameras
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Sensors which capture 3D scene information provide useful data for tasks in vehicle navigation, gesture recognition, human pose estimation, and geometric reconstruction. Active illumination time-of-flight sensors in particular have become widely used to estimate a 3D representation of a scene. However, the maximum range, density of acquired spatial samples, and overall acquisition time of these sensors is fundamentally limited by the minimum signal required to estimate depth reliably. In this paper, we propose a data-driven method for photon-efficient 3D imaging which leverages sensor fusion and computational reconstruction to rapidly and robustly estimate a dense depth map from low photon counts. Our sensor fusion approach uses measurements of single photon arrival times from a low-resolution single-photon detector array and an intensity image from a conventional high-resolution camera. Using a multi-scale deep convolutional network, we jointly process the raw measurements from both sensors and output a high-resolution depth map. To demonstrate the efficacy of our approach, we implement a hardware prototype and show results using captured data. At low signal-to-background levels, our depth reconstruction algorithm with sensor fusion outperforms other methods for depth estimation from noisy measurements of photon arrival times.
References:
1. E. Abreu, M. Lightstone, S.K. Mitra, and K. Arakawa. 1996. A new efficient approach for the removal of impulse noise from highly corrupted images. IEEE Trans. Image Process. 5, 6 (1996), 1012–1025. Google ScholarDigital Library
2. S. Achar, J.R. Bartels, W.L. Whittaker, K.N. Kutulakos, and S.G. Narasimhan. 2017. Epipolar time-of-flight imaging. ACM Trans. Graph. (SIGGRAPH) 36, 4 (2017), 37. Google ScholarDigital Library
3. Y. Altmann, R. Aspden, M. Padgett, and S. McLaughlin. 2017. A Bayesian Approach to Denoising of Single-Photon Binary Images. IEEE Trans. Computat. Imaging 3, 3 (Sept 2017), 460–471.Google Scholar
4. A. Bleiweiss and M. Werman. 2009. Fusing time-of-flight depth and color for real-time segmentation and tracking. In Dynamic 3D Imaging. 58–69. Google ScholarDigital Library
5. S. Burri, H. Homulle, C. Bruschini, and E. Charbon. 2016. LinoSPAD: A time-resolved 256X1 CMOS SPAD line sensor system featuring 64 FPGA-based TDC channels running at up to 8.5 giga-events per second. In Proc. SPIE, Vol. 9899. 98990D.Google Scholar
6. D. Chan, H. Buisman, C. Theobalt, and S. Thrun. 2008. A noise-aware filter for real-time depth upsampling. In Workshop on Multi-Camera and Multi-Modal Sensor Fusion Algorithms and Applications.Google Scholar
7. Q. Chen and V. Koltun. 2013. A simple model for intrinsic image decomposition with depth cues. In Proc. ICCV. 241–248. Google ScholarDigital Library
8. H. Dautet, P. Deschamps, B. Dion, A.D. MacGregor, D. MacSween, R.J. McIntyre, C. Trottier, and P.P. Webb. 1993. Photon counting techniques with silicon avalanche photodiodes. Applied optics 32, 21 (1993), 3894–3900.Google Scholar
9. J. Diebel and S. Thrun. 2006. An application of Markov Random Fields to range sensing. In Prac. NIPS. 291–298. Google ScholarDigital Library
10. D. Ferstl, C. Reinbacher, R. Ranftl, M. Rüther, and H. Bischof. 2013. Image guided depth upsampling using anisotropic total generalized variation. In Proc. CVPR. 993–1000. Google ScholarDigital Library
11. P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox. 2012. RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments. The International Journal of Robotics Research 31, 5 (2012), 647–663. Google ScholarDigital Library
12. R. Horaud, M. Hansard, G. Evangelidis, and C. Ménier. 2016. An overview of depth cameras and range scanners based on time-of-flight technologies. Machine Vision and Applications 27, 7 (2016), 1005–1020. Google ScholarDigital Library
13. T. Hui, C.C. Loy, and X. Tang. 2016. Depth map super-resolution by deep multi-scale guidance. In Proc. ECCV. 353–369.Google Scholar
14. D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
15. A. Kirmani, D. Venkatraman, D. Shin, A. Colaço, F.N.C. Wong, J.H. Shapiro, and V.K. Goyal. 2014. First-photon imaging. Science 343, 6166 (2014), 58–61.Google Scholar
16. A. Kolb, E. Barth, R. Koch, and R. Larsen. 2009. Time-of-flight sensors in computer graphics. In Eurographics (STARs). 119–134.Google Scholar
17. J. Kopf, M.F. Cohen, D. Lischinski, and M. Uyttendaele. 2007. Joint bilateral upsampling. In ACM Trans. Graph. (SIGGRAPH), Vol. 26. 96. Google ScholarDigital Library
18. M. Koskinen, J.T. Kostamovaara, and R.A. Myllylae. 1992. Comparison of continuous-wave and pulsed time-of-flight laser range-finding techniques. In Proc. SPIE 1614. 296–305.Google Scholar
19. Y. Li, J. Huang, N. Ahuja, and M. Yang. 2016. Deep joint image filtering. In Proc. ECCV. 154–169.Google Scholar
20. G. Lin, A. Milan, C. Shen, and I. Reid. 2017. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In Proc. CVPR.Google Scholar
21. D.B. Lindell, M. O’Toole, and G. Wetzstein. 2018. Towards transient imaging at interactive rates with single-photon detectors. In Proc. ICCP.Google Scholar
22. J. Marco, Q. Hernandez, A. Muñoz, Y. Dong, A. Jarabo, M.H. Kim, X. Tong, and D. Gutierrez. 2017. DeepToF: Off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6 (2017), 219:1–219:12. Google ScholarDigital Library
23. A. McCarthy, X. Ren, A. Della Frera, N.R. Gemmell, N.J. Krichel, C. Scarcella, A. Ruggeri, A. Tosi, and G.S. Buller. 2013. Kilometer-range depth imaging at 1550 nm wavelength using an InGaAs/InP single-photon avalanche diode detector. Optics express 21, 19 (2013), 22098–22113.Google Scholar
24. D. O’Connor and D. Philips. 1984. Time-correlated single photon counting. Academic Press.Google Scholar
25. M. O’Toole, S. Achar, S.G. Narasimhan, and K.N. Kutulakos. 2015. Homogeneous codes for energy-efficient illumination and imaging. ACM Trans. Graph. (SIGGRAPH) 34, 4, Article 35 (2015). Google ScholarDigital Library
26. M. O’Toole, F. Heide, D.B. Lindell, K. Zang, S. Diamond, and G. Wetzstein. 2017. Reconstructing transient images from single-photon sensors. In Proc. CVPR.Google Scholar
27. J. Park, H. Kim, Y. Tai, M.S. Brown, and I. Kweon. 2011. High quality depth map upsampling for 3D-TOF cameras. In Proc. ICCV 1623–1630. Google ScholarDigital Library
28. A.M. Pawlikowska, A. Halimi, R.A. Lamb, and G.S. Buller. 2017. Single-photon three-dimensional imaging at up to 10 kilometers range. Optics Express 25, 10 (2017), 11919–11931.Google ScholarCross Ref
29. C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun. 2017. Large kernel matters- Improve semantic segmentation by global convolutional network. In Proc. CVPR. 1743–1751.Google Scholar
30. G. Petschnigg, R. Szeliski, M. Agrawala, M. Cohen, H. Hoppe, and K. Toyama. 2004. Digital photography with flash and no-flash image pairs. ACM Trans. Graph. (SIGGRAPH) 23, 3 (2004), 664–672. Google ScholarDigital Library
31. E.F. Pettersen, T.D. Goddard, C.C. Huang, G.S. Couch, D.M. Greenblatt, E.C. Meng, and T.E. Ferrin. 2004. UCSF Chimera-a visualization system for exploratory research and analysis. Journal of computational chemistry 25, 13 (2004), 1605–1612.Google ScholarCross Ref
32. J. Rapp and V.K. Goyal. 2017. A few photons among many: Unmixing signal and noise for photon-efficient active imaging. IEEE Trans. Computat. Imaging 3 (2017), 445–459. Issue 3.Google ScholarCross Ref
33. D. Renker. 2006. Geiger-mode avalanche photodiodes, history, properties and problems. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 567, 1 (2006), 48–56.Google ScholarCross Ref
34. D. Scharstein and C. Pal. 2007. Learning conditional random fields for stereo. In Proc. CVPR. 1–8.Google Scholar
35. D. Shin, A. Kirmani, V.K. Goyal, and J.H. Shapiro. 2015. Photon-efficient computational 3-D and reflectivity imaging with single-photon detectors. IEEE Trans. Computat. Imaging 1, 2 (2015), 112–125.Google ScholarCross Ref
36. D. Shin, F. Xu, D. Venkatraman, R. Lussana, F. Villa, F. Zappa, V.K. Goyal, F.N.C. Wong, and J.H. Shapiro. 2016. Photon-efficient imaging with a single-photon camera. Nature Communications 7 (2016).Google Scholar
37. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. 2012. Indoor segmentation and support inference from RGBD images. In Proc. ECCV. Google ScholarDigital Library
38. S. Su, F. Heide, G. Wetzstein, and W. Heidrich. 2018. Deep end-to-end time-of-flight imaging. In Proc. CVPR.Google Scholar
39. R. Tobin, A. Halimi, A. McCarthy, X. Ren, K.J. McEwan, S. McLaughlin, and G.S. Buller, 2017. Long-range depth profiling of camouflaged targets using single-photon detection. Optical Engineering 57 (2017).Google Scholar
40. Q. Yang, R. Yang, J. Davis, and D. Nistér. 2007. Spatial-depth super resolution for range images. In Proc. CVPR. 1–8.Google Scholar
