“High-quality hyperspectral reconstruction using a spectral prior” by Choi, Jeon, Nam, Gutierrez and Kim
Conference:
Type(s):
Title:
- High-quality hyperspectral reconstruction using a spectral prior
Session/Category Title: High Performance Imaging
Presenter(s)/Author(s):
Abstract:
We present a novel hyperspectral image reconstruction algorithm, which overcomes the long-standing tradeoff between spectral accuracy and spatial resolution in existing compressive imaging approaches. Our method consists of two steps: First, we learn nonlinear spectral representations from real-world hyperspectral datasets; for this, we build a convolutional autoencoder which allows reconstructing its own input through its encoder and decoder networks. Second, we introduce a novel optimization method, which jointly regularizes the fidelity of the learned nonlinear spectral representations and the sparsity of gradients in the spatial domain, by means of our new fidelity prior. Our technique can be applied to any existing compressive imaging architecture, and has been thoroughly tested both in simulation, and by building a prototype hyperspectral imaging system. It outperforms the state-of-the-art methods from each architecture, both in terms of spectral accuracy and spatial resolution, while its computational complexity is reduced by two orders of magnitude with respect to sparse coding techniques. Moreover, we present two additional applications of our method: hyperspectral interpolation and demosaicing. Last, we have created a new high-resolution hyperspectral dataset containing sharper images of more spectral variety than existing ones, available through our project website.
References:
1. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat. G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng. 2016. TensorFlow: A System for Large-scale Machine Learning. In Proc. USENIX Conf. Operating Systems Design and Implementation (OSDI’16). 265–283.
2. Manya V. Afonso, José M. Bioucas-Dias, and Mário A. T. Figueiredo. 2011. An Augmented Lagrangian Approach to the Constrained Optimization Formulation of Imaging Inverse Problems. IEEE Trans. Image Processing (TIP) 20, 3 (2011), 681–695.
3. Sara Alvarez, Timo Kunkel, and Belen Masia. 2016. Practical Low-Cost Recovery of Spectral Power Distributions. Computer Graphics Forum 35, 1 (2016), 166–178.
4. Michael Attas, Edward Cloutis, Catherine Collins, Douglas Goltz, Claudine Majzels. James R Mansfield, and Henry H Mantsch. 2003. Near-infrared spectroscopic imaging in art conservation: investigation of drawing constituents. J. Cultural Heritage 4, 2 (2003), 127–136. Cross Ref
5. Seung-Hwan Baek, Incheol Kim, Diego Gutierrez, and Min H. Kim. 2017. Compact Single-Shot Hyperspectral Imaging Using a Prism. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017) 36, 6 (2017).
6. Jose M. Bioucas-Dias and Mario A. T. Figueiredo. 2007. A new TwIST: two-step iterative shrinkage/thresholding for image restoration. IEEE Trans. Image Processing (TIP) 16, 12 (2007), 2992–3004.
7. N. Brusco, S. Capeleto, M. Fedel, A. Paviotti, L. Poletto, G. M. Cortelazzo, and G. Tondello. 2006. A system for 3D modeling frescoed historical buildings with multispectral texture information. Machine Vision and Applications 17, 6 (2006), 373–393.
8. Xun Cao, Hao Du, Xin Tong, Qionghai Dai, and Stephen Lin. 2011. A prism-mask system for multispectral video acquisition. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 33, 12 (2011), 2423–2435.
9. A. Chakrabarti and T. Zickler. 2011. Statistics of Real-World Hyperspectral Images. In Proc. IEEE Conf. Comput. Vis. Pattern. Recognit. (CVPR). 193–200.
10. Yushi Chen, Zhouhan Lin, Xing Zhao, Gang Wang, and Yanfeng Gu. 2014. Deep Learning-Based Classification of Hyperspectral Data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 7, 6 (2014), 2094–2107. Cross Ref
11. B. Du, W. Xiong, J. Wu, L. Zhang, L. Zhang, and D. Tao. 2016. Stacked Convolutional Denoising Auto-Encoders for Feature Representation. IEEE Trans. Cybernetics PP. 99 (2016), 1–11.
12. Ying Fu, Yinqiang Zheng, Imari Sato, and Yoichi Sato. 2016. Exploiting Spectral-Spatial Correlation for Coded Hyperspectral Image Restoration. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
13. Nahum Gat. 2000. Imaging spectroscopy using tunable filters: a review. In Proc. AeroSense 2000. 50–64.
14. M E Gehm, R John, D J Brady, R M Willett, and T J Schulz. 2007. Single-shot compressive spectral imaging with a dual-disperser architecture. OSA Optics Express (OE) 15, 21 (2007), 14013–27. Cross Ref
15. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Aistats, Vol. 9. 249–256.
16. Mohammad Golbabaee, Simon Arberet, and Pierre Vandergheynst. 2013. Compressive source separation: Theory and methods for hyperspectral imaging. IEEE Transactions on Image Processing (TIP) 22, 12 (2013), 5096–5110.
17. Ralf Habel, Michael Kudenov, and Michael Wimmer. 2012. Practical spectral photography. In Computer Graphics Forum (Proc. EUROGRAPHICS 2012), Vol. 31. Wiley Online Library, 449–458.
18. Geoffrey E Hinton and Ruslan R Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504–507.
19. Gudrun Hoye and Andrei Fridman. 2013. Mixel camera – a new push-broom camera concept for high spatial resolution keystone-free hyperspectral imaging. OSA Optics Express (OE) 21, 9 (2013), 11057–11077. Cross Ref
20. Francisco Imai, Lawrence Taplin, and Ellen Day. 2002. Comparison of the accuracy of various transformations from multi-band images to reflectance spectra. Technical Report ISO 12233:2000. Rochester Institute of Technology.
21. Daniel S. Jeon, Inchang Choi, and Min H. Kim. 2016. Multisampling Compressive Video Spectroscopy. Computer Graphics Forum 35, 2 (2016), 467–477. Cross Ref
22. Min H Kim. 2013. 3D Graphics Techniques for Capturing and Inspecting Hyperspectral Appearance. In Ubiquitous Virtual Reality (ISUVR), 2013 Int. Symp. on. IEEE, 15–18.
23. Min H. Kim, Todd Alan Harvey, David S. Kittle, Holly Rushmeier, Julie Dorsey, Richard O. Prum, and David J. Brady. 2012a. 3D Imaging Spectroscopy for Measuring Hyperspectral Patterns on Solid Objects. ACM Trans. Graph. (TOG) 31, 4 (2012), 38:1–11.
24. Min H. Kim, Holly Rushmeier, John ffrench, and Irma Passeri. 2012b. Developing Open-Source Software for Art Conservators. In VAST12: The 13th International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage. Eurographics Association, Brighton, England, 97–104.
25. Min H. Kim, Holly Rushmeier, John ffrench, Irma Passeri, and David Tidmarsh. 2014. Hyper3D: 3D Graphics Software for Examining Cultural Artifacts. ACM Journal on Computing and Cultural Heritage 7, 3 (2014), 1:1–19.
26. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. In The International Conference on Learning Representations (ICLR).
27. David Kittle, Kerkil Choi, Ashwin Wagadarikar, and David J Brady. 2010. Multiframe image estimation for coded aperture snapshot spectral imagers. Applied Optics (AO) 49, 36 (2010), 6824–6833. Cross Ref
28. Kuldeep Kulkarni, Suhas Lohit, Pavan Turaga, Ronan Kerviche, and Amit Ashok. 2016. ReconNet: Non-Iterative Reconstruction of Images From Compressively Sensed Measurements. In The IEEE Conf. Computer Vision and Pattern Recognition (CVPR).
29. HaebomLee and Min H. Kim. 2014. Building a Two-Way Hyperspectral Imaging System with Liquid Crystal Tunable Filters. In Proc. Int. Conf. Image and Signal Processing (ICISP). 26–34.
30. Biao Leng, Shuang Guo, Xiangyang Zhang, and Zhang Xiong. 2015. 3D object retrieval with stacked local convolutional autoencoder. Signal Processing 112 (2015), 119–128.
31. Chengbo Li, Ting Sun, Kevin F. Kelly, and Yin Zhang. 2012. A Compressive Sensing and Unmixing Scheme for Hyperspectral Data Processing. IEEE Trans. Image Processing (TIP) 21, 3 (2012), 1200–1210.
32. Yunsong Li, Weiying Xie, and Huaqing Li. 2017. Hyperspectral image reconstruction by deep convolutional neural network for classification. Pattern Recognition (PR) 63 (2017), 371 — 383.
33. Xing Lin, Yebin Liu, Jiamin Wu, and Qionghai Dai. 2014. Spatial-spectral Encoded Compressive Hyperspectral Imaging. ACM Trans. Graph. (TOG) 33, 6 (2014).
34. Zhouhan Lin, Yushi Chen, Xing Zhao, and Gang Wang. 2013. Spectral-spatial classification of hyperspectral image using autoencoders. In Proc. Int. Conf. Signal Process. Commun. 1–5.
35. Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully Convolutional Networks for Semantic Segmentation. In Proc. IEEE Conf. Comput. Vis. Pattern. Recognit. (CVPR). 3431–3440. Cross Ref
36. Xiaorui Ma, Hongyu Wang, and Jie Geng. 2016. Spectral-Spatial Classification of Hyperspectral Image Based on Deep Auto-Encoder. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 9, 9 (2016), 4073–4085. Cross Ref
37. Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pierre Alliez. 2016. Convolutional Neural Networks for Large-Scale Remote Sensing Image Classification. IEEE Trans. Geosci. Remote Sens. (2016), 645 — 657.
38. Alkhazur Manakov, John F Restrepo, Oliver Klehm, Ramon Hegedüs, Elmar Eisemann, Hans-Peter Seidel, and Ivo Ihrke. 2013. A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging. ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 4 (2013), 47.
39. Julio Marco, Quercus Hernandez, Adolfo MuÃśoz, Yue Dong, Adrian Jarabo, Min Kim, Xin Tong, and Diego Gutierrez. 2017. DeepToF: Off-the-Shelf Real-time Correction of Multipath Interference in Time-of-Flight Imaging. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017) 36, 6 (2017).
40. Gabriel Martin, Jose M. Bioucas-Dias, and Antonio J. Plaza. 2015. HYCA: A New Technique for Hyperspectral Compressive Sensing. IEEE Trans. Geoscience and Remote Sensing 53, 5 (2015), 2819–2831. Cross Ref
41. Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber. 2011. Stacked convolutional auto-encoders for hierarchical feature extraction. In Proc. Int. Conf. Artificial Neural Networks (ICANN). 52–59.
42. Yi Peng, Deyu Meng, Zongben Xu, Chenqiang Gao, Yi Yang, and Biao Zhang. 2014. Decomposable Nonlocal Tensor Dictionary Learning for Multispectral Image De-noising. In Proc. IEEE Conf. Comput. Vis. Pattern. Recognit. (CVPR).
43. Wallace M Porter and Harry T Enmark. 1987. A system overview of the airborne visible/infrared imaging spectrometer (AVIRIS). In Technical Symposium. International Society for Optics and Photonics, 22–31. Cross Ref
44. Ajit Rajwade, David Kittle, Tsung-Han Tsai, David Brady, and Lawrence Carin. 2013. Coded Hyperspectral Imaging and Blind Compressive Sensing. SIAM Journal on Imaging Sciences 6, 2 (2013). Cross Ref
45. Konstantinos Rapantzikos and Costas Balas. 2005. Hyperspectral imaging: potential in non-destructive analysis of palimpsests. In Proc. IEEE Int. Conf. Image Processing (ICIP), Vol. 2. IEEE, II–618. Cross Ref
46. K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. Int. Conf. Learning Representation (ICLR).
47. Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. 2014. DeepFace: Closing the Gap to Human-Level Performance in Face Verification. In Proc. IEEE Conf. Comput. Vis. Pattern. Recognit. (CVPR).
48. Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11 (2010), 3371–3408. Cross Ref
49. J. J. Vos. 1978. Colorimetric and photometric properties of a 2-deg fundamental observer. Color Res. Appl. 3 (1978), 125–128. Cross Ref
50. Ashwin Wagadarikar, Renu John, Rebecca Willett, and David Brady. 2008. Single disperser design for coded aperture snapshot spectral imaging. OSA Applied Optics (AO) 47, 10 (2008), B44–B51. Cross Ref
51. Lizhi Wang, Zhiwei Xiong, Dahua Gao, Guangming Shi, Wenjun Zeng, and Feng Wu. 2015. High-Speed Hyperspectral Video Acquisition With a Dual-Camera Architecture. In Proc. IEEE Conf. Comput. Vis. Pattern. Recognit. (CVPR). Cross Ref
52. S.J. Wright, R.D. Nowak, and M.A.T. Figueiredo. 2009. Sparse reconstruction by separable approximation. IEEE TSP 57, 7 (2009), 2479–2493.
53. Yuehao Wu, Iftekhar O Mirza, Gonzalo R Arce, and Dennis W Prather. 2011. Development of a digital-micromirror-device-based multishot snapshot spectral imaging system. OSA Optics Letters (OL) 36, 14 (2011), 2692–4. Cross Ref
54. Chen Xing, Li Ma, and Xiaoquan Yang. 2016. Stacked Denoise Autoencoder Based Feature Extraction and Classification for Hyperspectral Images. Journal of Sensors 2016 (2016), 1–10.
55. F. Yasuma, T. Mitsunaga, D. Iso, and S. K. Nayar. 2010. Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum. IEEE Trans. Image Processing (TIP) 19, 9 (Sept 2010), 2241–2253.
56. Jaime Zabalza, Jinchang Ren, Jiangbin Zheng, Huimin Zhao, Chunmei Qing, Zhijing Yang, Peijun Du, and Stephen Marshall. 2016. Novel Segmented Stacked Autoencoder for Effective Dimensionality Reduction and Feature Extraction in Hyperspectral Imaging. Neurocomputing 214 (2016), 1–10.
57. Qiang Zhang, Robert Plemmons, David Kittle, David Brady, and Sudhakar Prasad. 2011. Joint segmentation and reconstruction of hyperspectral data with compressed measurements. OSA Applied Optics (AO) 50, 22 (2011), 4417–4435. Cross Ref
58. Yuting Zhang, Kibok Lee, and Honglak Lee. 2016. Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification. In Proc. International Conference on Machine Learning (ICML).
