“Compact snapshot hyperspectral imaging with diffracted rotation” by Jeon, Baek, Yi, Fu, Dun, et al. …
Conference:
Type:
Title:
- Compact snapshot hyperspectral imaging with diffracted rotation
Session/Category Title: Computational Imaging
Presenter(s)/Author(s):
Abstract:
Traditional snapshot hyperspectral imaging systems include various optical elements: a dispersive optical element (prism), a coded aperture, several relay lenses, and an imaging lens, resulting in an impractically large form factor. We seek an alternative, minimal form factor of snapshot spectral imaging based on recent advances in diffractive optical technology. We thereupon present a compact, diffraction-based snapshot hyperspectral imaging method, using only a novel diffractive optical element (DOE) in front of a conventional, bare image sensor. Our diffractive imaging method replaces the common optical elements in hyperspectral imaging with a single optical element. To this end, we tackle two main challenges: First, the traditional diffractive lenses are not suitable for color imaging under incoherent illumination due to severe chromatic aberration because the size of the point spread function (PSF) changes depending on the wavelength. By leveraging this wavelength-dependent property alternatively for hyperspectral imaging, we introduce a novel DOE design that generates an anisotropic shape of the spectrally-varying PSF. The PSF size remains virtually unchanged, but instead the PSF shape rotates as the wavelength of light changes. Second, since there is no dispersive element and no coded aperture mask, the ill-posedness of spectral reconstruction increases significantly. Thus, we propose an end-to-end network solution based on the unrolled architecture of an optimization procedure with a spatial-spectral prior, specifically designed for deconvolution-based spectral reconstruction. Finally, we demonstrate hyperspectral imaging with a fabricated DOE attached to a conventional DSLR sensor. Results show that our method compares well with other state-of-the-art hyperspectral imaging methods in terms of spectral accuracy and spatial resolution, while our compact, diffraction-based spectral imaging method uses only a single optical element on a bare image sensor.
References:
1. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng. 2016.Google Scholar
2. TensorFlow: A System for Large-scale Machine Learning. In Proc. USENIX Conf. Operating Systems Design and Implementation (OSDI’16). 265–283. Google ScholarDigital Library
3. Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. 2018. DiffuserCam: lensless single-exposure 3D imaging. Optica 5, 1 (2018), 1–9.Google ScholarCross Ref
4. Nicholas Antipa, Sylvia Necula, Ren Ng, and Laura Waller. 2016. Single-shot diffuser-encoded light field imaging. In Proc. IEEE Int. Conf. Computational Photography (ICCP 2016). IEEE, 1–11.Google ScholarCross Ref
5. Boaz Arad and Ohad Ben-Shahar. 2016. Sparse Recovery of Hyperspectral Signal from Natural RGB Images. In Proc. European Conference on Computer Vision (ECCV 2016). Springer, 19–34.Google ScholarCross Ref
6. M Salman Asif, Ali Ayremlou, Aswin Sankaranarayanan, Ashok Veeraraghavan, and Richard G Baraniuk. 2017. FlatCam: Thin, lensless cameras using coded aperture and computation. IEEE Transactions on Computational Imaging (TCI) 3, 3 (2017), 384–397.Google ScholarCross Ref
7. Seung-Hwan Baek, Incheol Kim, Diego Gutierrez, and Min H. Kim. 2017. Compact Single-Shot Hyperspectral Imaging Using a Prism. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017) 36, 6 (2017). Google ScholarDigital Library
8. Jose M. Bioucas-Dias and Mario A. T. Figueiredo. 2007. A new TwIST: two-step iterative shrinkage/thresholding for image restoration. IEEE Trans. Image Processing (TIP) 16, 12 (2007), 2992–3004. Google ScholarDigital Library
9. Nicola Brusco, S Capeleto, M Fedel, Anna Paviotti, Luca Poletto, Guido Maria Cortelazzo, and G Tondello. 2006. A system for 3D modeling frescoed historical buildings with multispectral texture information. Machine Vision and Applications 17, 6 (2006), 373–393. Google ScholarDigital Library
10. Ayan Chakrabarti and Todd Zickler. 2011. Statistics of real-world hyperspectral images. In Proc. Conference on Computer Vision and Pattern Recognition (CVPR 2011). IEEE, 193–200. Google ScholarDigital Library
11. Inchang Choi, Daniel S. Jeon, Giljoo Nam, Diego Gutierrez, and Min H. Kim. 2017. High-Quality Hyperspectral Reconstruction Using a Spectral Prior. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017) 36, 6 (2017). Google ScholarDigital Library
12. B. Choudhury, R. Swanson, F. Heide, G. Wetzstein, and W. Heidrich. 2017. Consensus Convolutional Sparse Coding. In Proc. International Conference on Computer Vision (ICCV 2017). 4290–4298.Google Scholar
13. Weisheng Dong, Peiyao Wang, Wotao Yin, and Guangming Shi. 2018. Denoising Prior Driven Deep Neural Network for Image Restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2018), 1–1.Google Scholar
14. M E Gehm, R John, D J Brady, R M Willett, and T J Schulz. 2007. Single-shot compressive spectral imaging with a dual-disperser architecture. OSA OE 15, 21 (2007), 14013–27.Google ScholarCross Ref
15. Adam Greengard, Yoav Y Schechner, and Rafael Piestun. 2006. Depth from diffracted rotation. Optics letters 31, 2 (2006), 181–183.Google Scholar
16. Ralf Habel, Michael Kudenov, and Michael Wimmer. 2012. Practical spectral photography. In Computer graphics forum, Vol. 31. Wiley Online Library, 449–458. Google ScholarDigital Library
17. Felix Heide, Qiang Fu, Yifan Peng, and Wolfgang Heidrich. 2016. Encoded diffractive optics for full-spectrum computational imaging. Scientific Reports 6 (2016), 33543.Google ScholarCross Ref
18. Daniel S Jeon, Inchang Choi, and Min H Kim. 2016. Multisampling Compressive Video Spectroscopy. Computer Graphics Forum 35, 2 (2016), 467–477.Google ScholarCross Ref
19. William R Johnson, Daniel W Wilson, Wolfgang Fink, Mark Humayun, and Greg Bearman. 2007. Snapshot hyperspectral imaging in ophthalmology. Journal of biomedical optics 12, 1 (2007), 014036–014036.Google ScholarCross Ref
20. Min H Kim. 2013. 3D Graphics Techniques for Capturing and Inspecting Hyperspectral Appearance. In Ubiquitous Virtual Reality (ISUVR), 2013 Int. Symp. on. IEEE, 15–18. Google ScholarDigital Library
21. Min H Kim, Todd Alan Harvey, David S Kittle, Holly Rushmeier, Julie Dorsey, Richard O Prum, and David J Brady. 2012a. 3D imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Transactions on Graphics 31, 4 (2012), 38. Google ScholarDigital Library
22. Min H. Kim and Holly Rushmeier. 2011. Radiometric Characterization of Spectral Imaging for Textual Pigment Identification. In Proc. International Symposium on Virtual Reality, Archaeology and Cultural Heritage (VAST 2011). Eurographics, Tuscany, Italy, 57–64. Google ScholarDigital Library
23. Min H Kim, Holly Rushmeier, John ffrench, and Irma Passeri. 2012b. Developing Open-Source Software for Art Conservators. In VAST12: The 13th International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage. Eurographics Association, Brighton, England, 97–104.Google Scholar
24. Min H Kim, Holly Rushmeier, John ffrench, Irma Passeri, and David Tidmarsh. 2014. Hyper3D: 3D Graphics Software for Examining Cultural Artifacts. ACM Journal on Computing and Cultural Heritage 7, 3 (2014), 1:1–19. Google ScholarDigital Library
25. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. In The International Conference on Learning Representations (ICLR).Google Scholar
26. David Kittle, Kerkil Choi, Ashwin Wagadarikar, and David J Brady. 2010. Multiframe image estimation for coded aperture snapshot spectral imagers. Applied Optics 49, 36 (2010), 6824–6833.Google ScholarCross Ref
27. Fred A Kruse, AB Lefkoff, JW Boardman, KB Heidebrecht, AT Shapiro, PJ Barloon, and AFH Goetz. 1993. The spectral image processing system (SIPS)—interactive visualization and analysis of imaging spectrometer data. Remote sensing of environment 44, 2–3 (1993), 145–163.Google Scholar
28. Haebom Lee and Min H. Kim. 2014. Building a Two-Way Hyperspectral Imaging System with Liquid Crystal Tunable Filters. In Proc. Int. Conf. Image and Signal Processing (ICISP 2014) (Lecture Notes in Computer Science (LNCS)), Vol. 8509. Springer, Normandy, France, 26–34.Google Scholar
29. Chengbo Li, Wotao Yin, and Yin Zhang. 2009. User’s guide for TVAL3: TV minimization by augmented lagrangian and alternating direction algorithms. CAAM report 20, 46–47 (2009), 4.Google Scholar
30. Xing Lin, Yebin Liu, Jiamin Wu, and Qionghai Dai. 2014. Spatial-spectral encoded compressive hyperspectral imaging. ACM Transactions on Graphics 33, 6 (2014), 233. Google ScholarDigital Library
31. Giljoo Nam and Min H. Kim. 2014. Multispectral Photometric Stereo for Acquiring High-Fidelity Surface Normals. IEEE Computer Graphics and Applications 34, 6 (2014), 57–68.Google ScholarCross Ref
32. Takayuki Okamoto, Akinori Takahashi, and Ichirou Yamaguchi. 1993. Simultaneous Acquisition of Spectral and Spatial Intensity Distribution. Appl. Spectrosc. 47, 8 (Aug 1993), 1198–1202.Google ScholarCross Ref
33. Donald C. O’Shea, Thomas J. Suleski, Alan D. Kathman, and Dennis W. Prather. 2003. Diffractive Optics: Design, Fabrication, and Test. SPIE Press. Google ScholarDigital Library
34. Yifan Peng, Xiong Dun, Qilin Sun, Felix Heide, and Wolfgang Heidrich. 2018. Focal sweep imaging with multi-focal diffractive optics. In IEEE Proc. Int. Conf. Computational Photography (ICCP). IEEE, 1–8.Google ScholarCross Ref
35. Yifan Peng, Qiang Fu, Hadi Amata, Shuochen Su, Felix Heide, and Wolfgang Heidrich. 2015. Computational imaging using lightweight diffractive-refractive optics. Optics Express 23, 24 (2015), 31393–31407.Google ScholarCross Ref
36. Yifan Peng, Qiang Fu, Felix Heide, and Wolfgang Heidrich. 2016. The diffractive achromat full spectrum computational imaging with diffractive optics. ACM Transactions on Graphics (Proc. SIGGRAPH 2016) (2016), 1–11. Google ScholarDigital Library
37. Wallace M Porter and Harry T Enmark. 1987. A system overview of the airborne visible/infrared imaging spectrometer (AVIRIS). In 31st Annual Technical Symposium. International Society for Optics and Photonics, 22–31.Google ScholarCross Ref
38. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241.Google ScholarCross Ref
39. K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. Int. Conf. Learning Representation (ICLR).Google Scholar
40. Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. 2018. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (Proc. SIGGRAPH 2018) 37, 4 (2018), 114. Google ScholarDigital Library
41. Gary J Swanson. 1991. Binary optics technology: theoretical limits on the diffraction efficiency of multilevel diffractive optical elements. Technical Report. MASSACHUSETTS INST OF TECH LEXINGTON LINCOLN LAB.Google Scholar
42. Kazuyuki Tajima, Takeshi Shimano, Yusuke Nakamura, Mayu Sao, and Taku Hoshizawa. 2017. Lensless light-field imaging with multi-phased Fresnel zone aperture. In Proc. IEEE Int. Conf. Computational Photography (ICCP). IEEE, 1–7.Google ScholarCross Ref
43. Ashwin Wagadarikar, Renu John, Rebecca Willett, and David Brady. 2008. Single disperser design for coded aperture snapshot spectral imaging. Applied optics 47, 10 (2008), B44–B51.Google Scholar
44. Lizhi Wang, Chen Sun, Ying Fu, Min H. Kim, and Huang Hua. 2019. Hyperspectral Image Reconstruction Using a Deep Spatial-Spectral Prior. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019). TBD.Google ScholarCross Ref
45. Peng Wang and Rajesh Menon. 2015. Ultra-high-sensitivity color imaging via a transparent diffractive-filter array and computational optics. Optica 2, 11 (Nov 2015), 933–939.Google ScholarCross Ref
46. Peng Wang and Rajesh Menon. 2018. Computational multispectral video imaging. J. Opt. Soc. Am. A 35, 1 (Jan 2018), 189–199.Google ScholarCross Ref
47. Jian Zhang and Bernard Ghanem. 2018. ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR 2018). 1828–1837.Google ScholarCross Ref
48. Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. 2017. Learning deep CNN denoiser prior for image restoration. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Vol. 2.Google ScholarCross Ref
