“Low-cost SPAD sensing for non-line-of-sight tracking, material classification and depth imaging” by Callenberg, Shi, Heide and Hullin

Time-correlated imaging is an emerging sensing modality that has been shown to enable promising application scenarios, including lidar ranging, fluorescence lifetime imaging, and even non-line-of-sight sensing. A leading technology for obtaining time-correlated light measurements are single-photon avalanche diodes (SPADs), which are extremely sensitive and capable of temporal resolution on the order of tens of picoseconds. However, the rare and expensive optical setups used by researchers have so far prohibited these novel sensing techniques from entering the mass market. Fortunately, SPADs also exist in a radically cheaper and more power-efficient version that has been widely deployed as proximity sensors in mobile devices for almost a decade. These commodity SPAD sensors can be obtained at a mere few cents per detector pixel. However, their inferior data quality and severe technical drawbacks compared to their high-end counterparts necessitate the use of additional optics and suitable processing algorithms. In this paper, we adopt an existing evaluation platform for commodity SPAD sensors, and modify it to unlock time-of-flight (ToF) histogramming and hence computational imaging. Based on this platform, we develop and demonstrate a family of hardware/software systems that, for the first time, implement applications that had so far been limited to significantly more advanced, higher-priced setups: direct ToF depth imaging, non-line-of-sight object tracking, and material classification.

1. Nils Abramson. 1978. Light-in-flight recording by holography. Optics letters 3, 4 (1978), 121–123.Google Scholar
2. Yoann Altmann, Stephen McLaughlin, Miles J Padgett, Vivek K Goyal, Alfred O Hero, and Daniele Faccio. 2018. Quantum-inspired computational imaging. Science 361, 6403 (2018).Google Scholar
3. Victor Arellano, Diego Gutierrez, and Adrian Jarabo. 2017. Fast back-projection for non-line of sight reconstruction. Opt. Express 25, 10 (2017), 11574–11583.Google ScholarCross Ref
4. David Patrick Baxter. 2015. Application using a single photon avalanche diode (SPAD). (June 16 2015). US Patent 9,058,081.Google Scholar
5. Katherine L Bouman, Vickie Ye, Adam B Yedidia, Frédo Durand, Gregory W Wornell, Antonio Torralba, and William T Freeman. 2017. Turning corners into cameras: Principles and methods. In IEEE International Conference on Computer Vision (ICCV). 2289–2297.Google ScholarCross Ref
6. Samuel Burri, Harald Homulle, Claudio Bruschini, and Edoardo Charbon. 2016. LinoSPAD: a time-resolved 256×1 CMOS SPAD line sensor system featuring 64 FPGA-based TDC channels running at up to 8.5 giga-events per second. Proc. SPIE 9899 (2016), 98990D-10.Google Scholar
7. Mauro Buttafava, Jessica Zeman, Alberto Tosi, Kevin Eliceiri, and Andreas Velten. 2015. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Optics express 23, 16 (2015), 20997–21011.Google Scholar
8. Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. 1995. A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing 16, 5 (1995), 1190–1208.Google Scholar
9. C. Callenberg, A. Lyons, D. den Brok, A. Fatima, A. Turpin, V. Zickus, L. Machesky, J. Whitelaw, D. Faccio, and M. B. Hullin. 2021. Super-resolution time-resolved imaging using computational sensor fusion. Scientific Reports 11, 1 (Jan. 2021), 1689.Google ScholarCross Ref
10. Barbara Caputo, Eric Hayman, and P Mallikarjuna. 2005. Class-specific material categorisation. In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, Vol. 2. IEEE, 1597–1604.Google ScholarDigital Library
11. Edoardo Charbon. 2008. Towards large scale CMOS single-photon detector arrays for lab-on-chip applications. Journal of Physics D: Applied Physics 41, 9 (2008), 094010.Google ScholarCross Ref
12. Wenzheng Chen, Simon Daneau, Fahim Mannan, and Felix Heide. 2019. Steady-state non-line-of-sight imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6790–6799.Google ScholarCross Ref
13. Wenzheng Chen, Fangyin Wei, Kiriakos N Kutulakos, Szymon Rusinkiewicz, and Felix Heide. 2020. Learned feature embeddings for non-line-of-sight imaging and recognition. ACM Trans. Graph. 39, 6 (2020), 1–18.Google ScholarDigital Library
14. Liang Gao, Jinyang Liang, Chiye Li, and Lihong V Wang. 2014. Single-shot compressed ultrafast photography at one hundred billion frames per second. Nature 516, 7529 (2014), 74.Google Scholar
15. Genevieve Gariepy, Nikola Krstajić, Robert Henderson, Chunyong Li, Robert R Thomson, Gerald S Buller, Barmak Heshmat, Ramesh Raskar, Jonathan Leach, and Daniele Faccio. 2015. Single-photon sensitive light-in-fight imaging. Nature communications 6 (2015), 6021.Google Scholar
16. Javier Grau Chopite, Matthias B. Hullin, Michael Wand, and Julian Iseringhausen. 2020. Deep Non-Line-of-Sight Reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
17. Felix Heide, Matthias B. Hullin, James Gregson, and Wolfgang Heidrich. 2013. Low-Budget Transient Imaging using Photonic Mixer Devices. ACM Trans. Graph. 32, 4 (2013), 45:1–45:10.Google ScholarDigital Library
18. Felix Heide, Matthew O’Toole, Kai Zang, David B Lindell, Steven Diamond, and Gordon Wetzstein. 2019. Non-line-of-sight imaging with partial occluders and surface normals. ACM Trans. Graph. 38, 3 (2019), 1–10.Google ScholarDigital Library
19. Felix Heide, Lei Xiao, Wolfgang Heidrich, and Matthias B Hullin. 2014. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3222–3229.Google ScholarDigital Library
20. Robert K. Henderson, Nick Johnston, Haochang Chen, David Day Uei Li, Graham Hungerford, Richard Hirsch, David McLoskey, Philip Yip, and David J.S. Birch. 2018. A 192X128 Time Correlated Single Photon Counting Imager in 40nm CMOS Technology. ESSCIRC 2018 – IEEE 44th European Solid State Circuits Conference (2018), 54–57. Google ScholarCross Ref
21. Quercus Hernandez, Diego Gutierrez, and Adrian Jarabo. 2017. A Computational Model of a Single-Photon Avalanche Diode Sensor for Transient Imaging. (2017). arXiv:physics.ins-det/1703.02635Google Scholar
22. Atul Ingle, Andreas Velten, and Mohit Gupta. 2019. High Flux Passive Imaging With Single-Photon Sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
23. Julian Iseringhausen and Matthias Hullin. 2020. Non-line-of-sight reconstruction using efficient transient rendering. ACM Trans. Graph. 39, 1 (2020), 1–14.Google ScholarDigital Library
24. Adrian Jarabo, Belen Masia, Julio Marco, and Diego Gutierrez. 2017. Recent advances in transient imaging: A computer graphics and vision perspective. Visual Informatics 1, 1 (2017), 65–79.Google ScholarCross Ref
25. A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar. 2013. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. 32, 6 (2013).Google ScholarDigital Library
26. Jonathan Klein, Christoph Peters, Jaime Martín, Martin Laurenzis, and Matthias B Hullin. 2016. Tracking objects outside the line of sight using 2D intensity images. Scientific reports 6 (2016), 32491.Google Scholar
27. Dilip Krishnan and Rob Fergus. 2009. Fast Image Deconvolution using Hyper-Laplacian Priors. In Advances in Neural Information Processing Systems 22, Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta (Eds.). Curran Associates, Inc., 1033–1041.Google Scholar
28. Jinyang Liang, Cheng Ma, Liren Zhu, Yujia Chen, Liang Gao, and Lihong V Wang. 2017. Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse. Science advances 3, 1 (2017), e1601814.Google Scholar
29. David B Lindell, Matthew O’Toole, and Gordon Wetzstein. 2018. Single-photon 3D imaging with deep sensor fusion. ACM Trans. Graph. 37, 4 (2018), 113.Google ScholarDigital Library
30. David B. Lindell, Gordon Wetzstein, and Matthew O’Toole. 2019. Wave-based non-line-of-sight imaging using fast f-k migration. ACM Trans. Graph. 38, 4 (2019), 116.Google ScholarDigital Library
31. Ce Liu, Lavanya Sharan, Edward H Adelson, and Ruth Rosenholtz. 2010. Exploring features in a bayesian framework for material recognition. In 2010 ieee computer society conference on computer vision and pattern recognition. IEEE, 239–246.Google Scholar
32. Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jarabo, Diego Gutierrez, and Andreas Velten. 2019. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature (2019), 1–4.Google Scholar
33. Andreas Meuleman, Seung-Hwan Baek, Felix Heide, and Min H Kim. 2020. Single-Shot Monocular RGB-D Imaging Using Uneven Double Refraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2465–2474.Google ScholarCross Ref
34. Nikhil Naik, Shuang Zhao, Andreas Velten, Ramesh Raskar, and Kavita Bala. 2011. Single View Reflectance Capture Using Multiplexed Scattering and Time-of-flight Imaging. ACM Trans. Graph. 30, 6 (2011), 171:1–171:10.Google ScholarDigital Library
35. Desmond O’Connor. 2012. Time-correlated single photon counting. Academic Press.Google Scholar
36. Matthew O’Toole, Felix Heide, Lei Xiao, Matthias B. Hullin, Wolfgang Heidrich, and Kiriakos N. Kutulakos. 2014. Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport. ACM Trans. Graph. 33, 4 (Aug. 2014).Google ScholarDigital Library
37. Matthew O’Toole, David B. Lindell, and Gordon Wetzstein. 2018. Confocal Non-line-of-sight imaging based on the light cone transform. Nature (2018), 338–341. Issue 555.Google Scholar
38. Christoph Peters, Jonathan Klein, Matthias B. Hullin, and Reinhard Klein. 2015. Solving Trigonometric Moment Problems for Fast Transient Imaging. ACM Trans. Graph. 34, 6 (Nov. 2015). Google ScholarDigital Library
39. Corneliu Rablau. 2019. LIDAR-A new (self-driving) vehicle for introducing optics to broader engineering and non-engineering audiences. In Education and Training in Optics and Photonics. Optical Society of America, 11143_138.Google Scholar
40. Sabbir Rangwala. 2020. The iPhone 12 – LiDAR At Your Fingertips. Forbes (12 November 2020). https://www.forbes.com/sites/sabbirrangwala/2020/11/12/the-iphone-12lidar-at-your-fingertips/Google Scholar
41. Justin A. Richardson, Lindsay A. Grant, and Robert K. Henderson. 2009. Low dark count single-photon avalanche diode structure compatible with standard nanometer scale CMOS technology. IEEE Photonics Technology Letters 21, 14 (2009), 1020–1022.Google ScholarCross Ref
42. N. Scheiner, F. Kraus, F. Wei, B. Phan, F. Mannan, N. Appenrodt, W. Ritter, J. Dickmann, K. Dietmayer, B. Sick, et al. 2020. Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2068–2077.Google ScholarCross Ref
43. Brent Schwarz. 2010. LIDAR: Mapping the world in 3D. Nat. Photonics 4, 7 (2010), 1749–4885.Google ScholarCross Ref
44. Nikolai Smolyanskiy, Alexey Kamenev, and Stan Birchfield. 2018. On the importance of stereo for accurate depth estimation: An efficient semi-supervised deep neural network approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1007–1015.Google ScholarCross Ref
45. STMicroelectronics. 2019. STMicroelectronics Ships 1 Billionth Time-of-Flight Module. (26 November 2019). https://www.st.com/content/st_com/en/about/media-center/press-item.html/t4210.html Press release.Google Scholar
46. Shuochen Su, Felix Heide, Robin Swanson, Jonathan Klein, Clara Callenberg, Matthias Hullin, and Wolfgang Heidrich. 2016. Material Classification Using Raw Time-Of-Flight Measurements. In Proc. IEEE CVPR.Google ScholarCross Ref
47. Kenichiro Tanaka, Yasuhiro Mukaigawa, Takuya Funatomi, Hiroyuki Kubo, Yasuyuki Matsushita, and Yasushi Yagi. 2017. Material classification using frequency-and depth-dependent time-of-flight distortion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 79–88.Google ScholarCross Ref
48. Chia-Yin Tsai, Aswin C Sankaranarayanan, and Ioannis Gkioulekas. 2019. Beyond Volumetric Albedo-A Surface Optimization Framework for Non-Line-Of-Sight Imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1545–1555.Google ScholarCross Ref
49. Manik Varma and Andrew Zisserman. 2008. A statistical approach to material classification using image patch exemplars. IEEE transactions on pattern analysis and machine intelligence 31, 11 (2008), 2032–2047.Google Scholar
50. Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G. Bawendi, and Ramesh Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications 3 (2012), 745. Google ScholarCross Ref
51. Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Chinmaya Joshi, Everett Lawson, Moungi Bawendi, Diego Gutierrez, and Ramesh Raskar. 2013. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graph. 32, 4 (2013), 44.Google ScholarDigital Library
52. Junko Yoshida. 2018. ST & Apple, through Thick and Thin. EETimes (22 August 2018). https://www.eetimes.com/st-apple-through-thick-and-thin/Google Scholar
53. Franco Zappa, Simone Tisa, Alberto Tosi, and Sergio Cova. 2007. Principles and features of single-photon avalanche diode arrays. Sensors and Actuators A: Physical 140, 1 (2007), 103–112.Google ScholarCross Ref
54. Vytautas Zickus, Ming-Lo Wu, Kazuhiro Morimoto, Valentin Kapitany, Areeba Fatima, Alex Turpin, Robert Insall, Jamie Whitelaw, Laura Machesky, Claudio Bruschini, et al. 2020. Fluorescence lifetime imaging with a megapixel SPAD camera and neural network lifetime estimation. Scientific Reports 10, 1 (2020), 1–10.Google ScholarCross Ref