“Seeing Photons in Color” by Ma, Sundar, Mos, Bruschini, Charbon, et al. …
Conference:
Type:
Title:
- Seeing Photons in Color
Session/Category Title: Colorful Topics in Imaging
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Megapixel single-photon avalanche diode (SPAD) arrays have been developed recently, opening up the possibility of deploying SPADs as generalpurpose passive cameras for photography and computer vision. However, most previous work on SPADs has been limited to monochrome imaging. We propose a computational photography technique that reconstructs high-quality color images from mosaicked binary frames captured by a SPAD array, even for high-dyanamic-range (HDR) scenes with complex and rapid motion. Inspired by conventional burst photography approaches, we design algorithms that jointly denoise and demosaick single-photon image sequences. Based on the observation that motion effectively increases the color sample rate, we design a blue-noise pseudorandom RGBW color filter array for SPADs, which is tailored for imaging dark, dynamic scenes. Results on simulated data, as well as real data captured with a fabricated color SPAD hardware prototype shows that the proposed method can reconstruct high-quality images with minimal color artifacts even for challenging low-light, HDR and fast-moving scenes. We hope that this paper, by adding color to computational single-photon imaging, spurs rapid adoption of SPADs for real-world passive imaging applications.
References:
1. J. Adams, K. Parulski, and K. Spaulding. 1998. Color Processing in Digital Cameras. IEEE Micro 18, 6 (1998), 20–30.
2. David Alleysson, Sabine Susstrunk, and Jeanny Herault. 2005. Linear Demosaicing Inspired by the Human Visual System. IEEE Transactions on Image Processing 14, 4 (April 2005), 439–449.
3. Ivan Michel Antolovic, Samuel Burri, Claudio Bruschini, Ron Hoebe, and Edoardo Charbon. 2016. Nonuniformity Analysis of a 65-Kpixel CMOS SPAD Imager. IEEE Transactions on Electron Devices 63, 1 (Jan. 2016), 57–64.
4. Chenyan Bai and Jia Li. 2019. Convolutional Sparse Coding for Demosaicking with Panchromatic Pixels. Signal Processing: Image Communication 77 (2019), 20–27.
5. Bryce E Bayer. 1976. Color imaging array. US Patent 3,971,065.
6. Goutam Bhat, Martin Danelljan, Fisher Yu, Luc Van Gool, and Radu Timofte. 2021. Deep Reparametrization of Multi-Frame Super-Resolution and Denoising. In IEEE/CVF International Conference on Computer Vision (ICCV). 2460–2470. arXiv:2108.08286 [cs, eess]
7. Folkmar Bornemann and Tom März. 2007. Fast Image Inpainting Based on Coherence Transport. Journal of Mathematical Imaging and Vision 28, 3 (Oct. 2007), 259–278.
8. Claudio Bruschini, Ivan Michel Antolovic, Frédéric Zanella, Arin C Ulku, Scott Lindner, Alexander Kalyanov, Tommaso Milanese, Ermanno Bernasconi, Vladimir Pešić, and Edoardo Charbon. 2023. Challenges and Prospects for Multi-chip Microlens Imprints on Front-Side Illuminated SPAD Imagers. Optica Open (2023).
9. Claudio Bruschini, Harald Homulle, Ivan Michel Antolovic, Samuel Burri, and Edoardo Charbon. 2019. Single-Photon Avalanche Diode Imagers in Biophotonics: Review and Outlook. Light: Science & Applications 8, 1 (Dec. 2019), 87.
10. Mauro Buttafava, Jessica Zeman, Alberto Tosi, Kevin Eliceiri, and Andreas Velten. 2015. Non-Line-of-Sight Imaging Using a Time-Gated Single Photon Avalanche Diode. Optics Express 23, 16 (Aug. 2015), 20997.
11. Ayan Chakrabarti. 2016. Learning Sensor Multiplexing Design through Back-propagation. In International Conference on Neural Information Processing Systems (NIPS’16). Curran Associates Inc., Barcelona, Spain, 3089–3097.
12. Ayan Chakrabarti, William T. Freeman, and Todd Zickler. 2014. Rethinking Color Cameras. In IEEE International Conference on Computational Photography (ICCP). IEEE, Santa Clara, CA, USA, 1–8.
13. Stanley Chan, Omar Elgendy, and Xiran Wang. 2016. Images from Bits: Non-Iterative Image Reconstruction for Quanta Image Sensors. Sensors 16, 11 (Nov. 2016), 1961.
14. Paramanand Chandramouli, Samuel Burri, Claudio Bruschini, Edoardo Charbon, and Andreas Kolb. 2019. A Bit Too Much? High Speed Imaging from Sparse Photon Counts. In IEEE International Conference on Computational Photography (ICCP). Tokyo, Japan, 1–9. arXiv:1811.02396
15. Chen Chen, Qifeng Chen, Minh N Do, and Vladlen Koltun. 2019. Seeing Motion in the Dark. In International Conference on Computer Vision (ICCV). 3185–3194.
16. Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. 2018. Learning to see in the dark. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3291–3300.
17. Yiheng Chi, Abhiram Gnanasambandam, Vladlen Koltun, and Stanley H. Chan. 2020. Dynamic Low-Light Imaging with Quanta Image Sensors. In European Conference on Computer Vision (ECCV), Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 122–138.
18. Joon Hee Choi, Omar A. Elgendy, and Stanley H. Chan. 2018. Image Reconstruction for Quanta Image Sensors Using Deep Neural Networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, Calgary, AB, 6543–6547.
19. Laurent Condat. 2009. A Generic Variational Approach for Demosaicking from an Arbitrary Color Filter Array. In IEEE International Conference on Image Processing (ICIP). IEEE, Cairo, Egypt, 1625–1628.
20. Laurent Condat. 2010. Color Filter Array Design Using Random Patterns with Blue Noise Chromatic Spectra. Image and Vision Computing 28, 8 (Aug. 2010), 1196–1202.
21. Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007. Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering. IEEE Transactions on Image Processing 16, 8 (Aug. 2007), 2080–2095.
22. Mark A. Z. Dippe and Erling Henry Wold. 1985. Antialiasing Through Stochastic Sampling. SIGGRAPH Comput. Graph. 19, 3 (July 1985), 69–78.
23. Xingbo Dong, Wanyan Xu, Zhihui Miao, Lan Ma, Chao Zhang, Jiewen Yang, Zhe Jin, Andrew Beng Jin Teoh, and Jiajun Shen. 2022. Abandoning the Bayer-filter to see in the dark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17431–17440.
24. Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022. Burst Image Restoration and Enhancement. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5759–5768. arXiv:2110.03680 [cs]
25. Omar Elgendy, Abhiram Gnanasambandam, Stanley H. Chan, and Jiaju Ma. 2021. Low-Light Demosaicking and Denoising for Small Pixels Using Learned Frequency Selection. IEEE Transactions on Computational Imaging 7 (2021), 137–150.
26. Omar A. Elgendy and Stanley H. Chan. 2020. Color Filter Arrays for Quanta Image Sensors. IEEE Transactions on Computational Imaging 6 (2020), 652–665. arXiv:1903.09823
27. Eric R. Fossum. 2005. What To Do With Sub-Diffraction Limit (SDL) Pixels?—A Proposal for a Gigapixel Digital Film Sensor (DFS). In IEEE Workshop on Charge-Coupled Devices and Advanced Image Sensors. 214–217.
28. Eric R. Fossum. 2013. Modeling the Performance of Single-Bit and Multi-Bit Quanta Image Sensors. IEEE Journal of the Electron Devices Society 1, 9 (Sept. 2013), 166–174.
29. Edward B Gindele and Andrew C Gallagher. 2002. Sparsely sampled image sensing device with color and luminance photosites. US Patent 6,476,865.
30. Abhiram Gnanasambandam and Stanley H. Chan. 2020. Image Classification in the Dark Using Quanta Image Sensors. In European Conference on Computer Vision (ECCV). Springer, 484–501. arXiv:2006.02026
31. Abhiram Gnanasambandam, Omar Elgendy, Jiaju Ma, and Stanley H. Chan. 2019. Megapixel Photon-Counting Color Imaging Using Quanta Image Sensor. Optics Express 27, 12 (June 2019), 17298.
32. Clément Godard, Kevin Matzen, and Matt Uyttendaele. 2018. Deep Burst Denoising. In European Conference on Computer Vision (ECCV), Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer International Publishing, Cham, 538–554.
33. Alexander D Griffiths, Haochang Chen, David Day-Uei Li, Robert K Henderson, Johannes Herrnsdorf, Martin D Dawson, and Michael J Strain. 2019. Multispectral time-of-flight imaging using light-emitting diodes. Optics express 27, 24 (2019), 35485–35498.
34. Istvan Gyongy, Neale Dutton, and Robert Henderson. 2018. Single-Photon Tracking for High-Speed Vision. Sensors 18, 2 (Jan. 2018), 323.
35. Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst Photography for High Dynamic Range and Low-Light Imaging on Mobile Cameras. ACM Transactions on Graphics 35, 6 (Nov. 2016), 1–12.
36. Felix Heide, Steven Diamond, Matthias Nießner, Jonathan Ragan-Kelley, Wolfgang Heidrich, and Gordon Wetzstein. 2016. ProxImaL: Efficient Image Optimization Using Proximal Algorithms. ACM Transactions on Graphics 35, 4 (July 2016), 1–15.
37. Felix Heide, Karen Egiazarian, Jan Kautz, Kari Pulli, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pająk, Dikpal Reddy, Orazio Gallo, Jing Liu, and Wolfgang Heidrich. 2014. FlexISP: A Flexible Camera Image Processing Framework. ACM Transactions on Graphics 33, 6 (Nov. 2014), 1–13.
38. Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. 2022. Real-Time Intermediate Flow Estimation for Video Frame Interpolation. In European Conference on Computer Vision (ECCV). Springer, 624–642. arXiv:2011.06294 [cs]
39. Atul Ingle, Trevor Seets, Mauro Buttafava, Shantanu Gupta, Alberto Tosi, Mohit Gupta, and Andreas Velten. 2021. Passive Inter-Photon Imaging. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8585–8595.
40. Atul Ingle, Andreas Velten, and Mohit Gupta. 2019. High Flux Passive Imaging with Single-Photon Sensors. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 6760–6769.
41. ITU-R. 2011. Studio Encoding Parameters of Digital Television for Standard 4:3 and Wide-Screen 16:9 Aspect Ratios. (2011).
42. Kiyotaka Iwabuchi, Yusuke Kameda, and Takayuki Hamamoto. 2021. Image Quality Improvements Based on Motion-Based Deblurring for Single-Photon Imaging. IEEE Access 9 (2021), 30080–30094.
43. Kiyotaka Iwabuchi, Tomohiro Yamazaki, and Takayuki Hamamoto. 2019. Iterative Image Reconstruction for Quanta Image Sensor by Using Variance-based Motion Estimation. In International Image Sensor Workshop (IISW).
44. Haiyang Jiang and Yinqiang Zheng. 2019. Learning to see moving objects in the dark. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7324–7333.
45. Linan Jiang, Kyung-Jo Kim, Francis M. Reininger, Sebastien Jiguet, and Stanley Pau. 2020. Microfabrication of a Color Filter Array Utilizing Colored SU-8 Photoresists. Applied Optics 59, 22 (Aug. 2020), G137.
46. Ahmet Serdar Karadeniz, Erkut Erdem, and Aykut Erdem. 2021. Burst photography for learning to enhance extremely dark images. IEEE Transactions on Image Processing 30 (2021), 9372–9385.
47. Fahad Shahbaz Khan, Rao Muhammad Anwer, Joost Van de Weijer, Andrew D. Bagdanov, Maria Vanrell, and Antonio M. Lopez. 2012. Color Attributes for Object Detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Providence, RI, 3306–3313.
48. Adam G Kirk and James F O’Brien. 2011. Perceptually Based Tone Mapping for Low-Light Conditions. ACM Transactions on Graphics 30, 4 (2011), 10.
49. Filippos Kokkinos and Stamatis Lefkimmiatis. 2019. Iterative residual cnns for burst photography applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5929–5938.
50. Hideyasu Kuniba and Roy S. Berns. 2009. Spectral Sensitivity Optimization of Color Image Sensors Considering Photon Shot Noise. Journal of Electronic Imaging 18, 2 (2009), 023002.
51. Chiman Kwan, Jude Larkin, and Bulent Ayhan. 2020. Demosaicing of CFA 3.0 with Applications to Low Lighting Images. Sensors 20, 12 (June 2020), 3423.
52. Jia Li, Chenyan Bai, Zhouchen Lin, and Jian Yu. 2017. Automatic Design of High-Sensitivity Color Filter Arrays With Panchromatic Pixels. IEEE Transactions on Image Processing 26, 2 (Feb. 2017), 870–883.
53. Zhetong Liang, Shi Guo, Hong Gu, Huaqi Zhang, and Lei Zhang. 2020. A Decoupled Learning Scheme for Real-world Burst Denoising from Raw Images. In European Conference on Computer Vision (ECCV). 17.
54. Orly Liba, Ryan Geiss, Samuel W. Hasinoff, Yael Pritch, Marc Levoy, Kiran Murthy, Yun-Ta Tsai, Tim Brooks, Tianfan Xue, Nikhil Karnad, Qiurui He, Jonathan T. Barron, and Dillon Sharlet. 2019. Handheld Mobile Photography in Very Low Light. ACM Transactions on Graphics 38, 6 (Nov. 2019), 1–16.
55. Ziwei Liu, Lu Yuan, Xiaoou Tang, Matt Uyttendaele, and Jian Sun. 2014. Fast Burst Images Denoising. ACM Transactions on Graphics 33, 6 (Nov. 2014), 1–9.
56. Jiaju Ma, Saleh Masoodian, Dakota A. Starkey, and Eric R. Fossum. 2017. Photon-Number-Resolving Megapixel Image Sensor at Room Temperature without Avalanche Gain. Optica 4, 12 (Dec. 2017), 1474.
57. Jiaju Ma, Dexue Zhang, Dakota Robledo, Leo Anzagira, and Saleh Masoodian. 2022. Ultra-High-Resolution Quanta Image Sensor with Reliable Photon-Number-Resolving and High Dynamic Range Capabilities. Scientific Reports 12, 1 (Aug. 2022), 13869.
58. Sizhuo Ma, Shantanu Gupta, Arin C. Ulku, Claudio Bruschini, Edoardo Charbon, and Mohit Gupta. 2020. Quanta Burst Photography. ACM Transactions on Graphics 39, 4 (July 2020), 1–16.
59. Matteo Maggioni, Giacomo Boracchi, Alessandro Foi, and Karen Egiazarian. 2012. Video Denoising, Deblocking, and Enhancement Through Separable 4-D Nonlocal Spatiotemporal Transforms. IEEE Transactions on Image Processing 21, 9 (Sept. 2012), 3952–3966.
60. Rafal Mantiuk, Karol Myszkowski, and Hans-Peter Seidel. 2006. A Perceptual Framework for Contrast Processing of High Dynamic Range Images. ACM Transactions on Applied Perception 3, 3 (July 2006), 286–308.
61. Ben Mildenhall, Jonathan T. Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll. 2018. Burst Denoising with Kernel Prediction Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Salt Lake City, UT, 2502–2510.
62. Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, and Jonathan T. Barron. 2021. NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images. arXiv:2111.13679 [cs, eess] (Nov. 2021). arXiv:2111.13679 [cs, eess]
63. Kazuhiro Morimoto, Andrei Ardelean, Ming-Lo Wu, Arin Can Ulku, Ivan Michel Antolovic, Claudio Bruschini, and Edoardo Charbon. 2020. Megapixel Time-Gated SPAD Image Sensor for 2D and 3D Imaging Applications. Optica 7, 4 (April 2020), 346–354.
64. K Morimoto, J Iwata, M Shinohara, H Sekine, A Abdelghafar, H Tsuchiya, Y Kuroda, K Tojima, W Endo, Y Maehashi, Y Ota, T Sasago, S Maekawa, S Hikosaka, T Kanou, A Kato, T Tezuka, S Yoshizaki, T Ogawa, K Uehira, A Ehara, F Inui, Y Matsuno, K Sakurai, and T Ichikawa. 2021. 3.2 Megapixel 3D-Stacked Charge Focusing SPAD for Low-Light Imaging and Depth Sensing. In IEEE International Electron Devices Meeting (IEDM). 20.2.1–20.2.4.
65. Jun Ogi, Takafumi Takatsuka, Kazuki Hizu, Yutaka Inaoka, Hongbo Zhu, Yasuhisa Tochigi, Yoshiaki Tashiro, Fumiaki Sano, Yusuke Murakawa, Makoto Nakamura, and Yusuke Oike. 2021. A 124-dB Dynamic-Range SPAD Photon-Counting Image Sensor Using Subframe Sampling and Extrapolating Photon Count. IEEE Journal of Solid-State Circuits 56, 11 (Nov. 2021), 3220–3227.
66. Paul Oh, Sukho Lee, and Moon Kang. 2017. Colorization-Based RGB-White Color Interpolation Using Color Filter Array with Randomly Sampled Pattern. Sensors 17, 7 (June 2017), 1523.
67. Yasuharu Ota, Kazuhiro Morimoto, Tomoya Sasago, Mahito Shinohara, Yukihiro Kuroda, Wataru Endo, Yu Maehashi, Shintaro Maekawa, Hiroyuki Tsuchiya, Aymantarek Abdelahafar, Shingo Hikosaka, Masanao Motoyama, Kenzo Tojima, Kosei Uehira, Junji Iwata, Fumihiro Inui, Yasushi Matsuno, Katsuhito Sakurai, and Takeshi Ichikawa. 2022. A 0.37W 143dB-Dynamic-Range 1Mpixel Backside-Illuminated Charge-Focusing SPAD Image Sensor with Pixel-Wise Exposure Control and Adaptive Clocked Recharging. In IEEE International Solid- State Circuits Conference (ISSCC). IEEE, San Francisco, CA, USA, 94–96.
68. Manu Parmar and Stanley J. Reeves. 2006. Selection of Optimal Spectral Sensitivity Functions for Color Filter Arrays. In International Conference on Image Processing. IEEE, Atlanta, GA, 1005–1008.
69. Manu Parmar and Brian A. Wandell. 2009. Interleaved Imaging: An Imaging System Design Inspired by Rod-Cone Vision. In IS&T/SPIE Electronic Imaging, Brian G. Rodricks and Sabine E. Süsstrunk (Eds.). San Jose, CA, 725008.
70. Naama Pearl, Tali Treibitz, and Simon Korman. 2022. Nan: Noise-aware nerfs for burst-denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12672–12681.
71. Ximing Ren, Yoann Altmann, Rachael Tobin, Aongus Mccarthy, Stephen Mclaughlin, and Gerald S Buller. 2018. Wavelength-time coding for multispectral 3D imaging using single-photon LiDAR. Optics express 26, 23 (2018), 30146–30161.
72. Trevor Seets, Atul Ingle, Martin Laurenzis, and Andreas Velten. 2021. Motion Adaptive Deblurring with Single-Photon Cameras. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 1945–1954. arXiv:2012.07931
73. Yash D. Shah, Peter W. R. Connolly, James P. Grant, Danni Hao, Claudio Accarino, Ximing Ren, Mitchell Kenney, Valerio Annese, Kirsty G. Rew, Zoë M. Greener, Yoann Altmann, Daniele Faccio, Gerald S. Buller, and David R. S. Cumming. 2020. Ultralow-Light-Level Color Image Reconstruction Using High-Efficiency Plasmonic Metasurface Mosaic Filters. Optica 7, 6 (June 2020), 632.
74. S. M. A. Sharif and Yong Ju Jung. 2019. Deep Color Reconstruction for a Sparse Color Sensor. Optics Express 27, 17 (Aug. 2019), 23661.
75. Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. 2021. XVFI: eXtreme Video Frame Interpolation. In IEEE/CVF International Conference on Computer Vision (ICCV). 14489–14498.
76. Masayuki Tachi. 2012. Image processing device, image processing method, and program pertaining to image correction. US Patent 8,314,863.
77. Hiroyuki Takeda, Sina Farsiu, and Peyman Milanfar. 2007. Kernel Regression for Image Processing and Reconstruction. IEEE Transactions on Image Processing 16, 2 (Feb. 2007), 349–366.
78. Arin Can Ulku, Claudio Bruschini, Ivan Michel Antolovic, Yung Kuo, Rinat Ankri, Shimon Weiss, Xavier Michalet, and Edoardo Charbon. 2019. A 512 × 512 SPAD Image Sensor With Integrated Gating for Widefield FLIM. IEEE Journal of Selected Topics in Quantum Electronics 25, 1 (Jan. 2019), 1–12.
79. Bartlomiej Wronski, Ignacio Garcia-Dorado, Manfred Ernst, Damien Kelly, Michael Krainin, Chia-Kai Liang, Marc Levoy, and Peyman Milanfar. 2019. Handheld MultiFrame Super-Resolution. ACM Transactions on Graphics 38, 4 (July 2019), 1–18. arXiv:1905.03277
80. Zhihao Xia, Federico Perazzi, Michaël Gharbi, Kalyan Sunkavalli, and Ayan Chakrabarti. 2019. Basis Prediction Networks for Effective Burst Denoising with Large Kernels. arXiv:1912.04421 [cs] (Dec. 2019). arXiv:1912.04421 [cs]
81. Feng Yang, Y. M. Lu, L. Sbaiz, and M. Vetterli. 2012. Bits From Photons: Oversampled Image Acquisition Using Binary Poisson Statistics. IEEE Transactions on Image Processing 21, 4 (April 2012), 1421–1436.
82. F. Zappa, S. Tisa, A. Tosi, and S. Cova. 2007. Principles and Features of Single-Photon Avalanche Diode Arrays. Sensors and Actuators A: Physical 140, 1 (Oct. 2007), 103–112.
83. Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. 2021. Spk2ImgNet: Learning to Reconstruct Dynamic Scene from Continuous Spike Stream. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Nashville, TN, USA, 11991–12000.
