“Image super-resolution via deterministic-stochastic synthesis and local statistical rectification”
Conference:
Type(s):
Title:
- Image super-resolution via deterministic-stochastic synthesis and local statistical rectification
Session/Category Title: Low-level imaging
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Single image superresolution has been a popular research topic in the last two decades and has recently received a new wave of interest due to deep neural networks. In this paper, we approach this problem from a different perspective. With respect to a downsampled low resolution image, we model a high resolution image as a combination of two components, a deterministic component and a stochastic component. The deterministic component can be recovered from the low-frequency signals in the downsampled image. The stochastic component, on the other hand, contains the signals that have little correlation with the low resolution image. We adopt two complementary methods for generating these two components. While generative adversarial networks are used for the stochastic component, deterministic component reconstruction is formulated as a regression problem solved using deep neural networks. Since the deterministic component exhibits clearer local orientations, we design novel loss functions tailored for such properties for training the deep regression network. These two methods are first applied to the entire input image to produce two distinct high-resolution images. Afterwards, these two images are fused together using another deep neural network that also performs local statistical rectification, which tries to make the local statistics of the fused image match the same local statistics of the groundtruth image. Quantitative results and a user study indicate that the proposed method outperforms existing state-of-the-art algorithms with a clear margin.
References:
1. Il Jun Ahn and Woo Hyun Nam. 2016. Texture Enhancement via High-Resolution Style Transfer for Single-Image Super-Resolution. arXiv preprint arXiv:1612.00085 (2016).Google Scholar
2. S. Baker and T. Kanade. 2000. Hallucinating faces. In 4th IEEE International Conference on Automatic Face and Gesture Recognition. 83–88. Google ScholarDigital Library
3. T. Bengtsson, I.Y-H. Gu, M. Viberg, and K. Lindstrom. 2012. Regularized optimization for joint super-resolution and high dynamic range image reconstruction in a perceptually uniform domain. In IEEE International Conference on Acoustics, Speech and Signal Processing. 1097–1100.Google Scholar
4. Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. (2012).Google Scholar
5. D. Capel and A. Zisserman. 1998. Automated mosaicing with superresolution zoom. In IEEE Conference on Computer Vision and Pattern Recognition. 885–891. Google ScholarDigital Library
6. Hong Chang, Dit-Yan Yeung, and Yimin Xiong. 2004. Super-resolution through neighbor embedding. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, Vol. 1. IEEE, I–I.Google ScholarCross Ref
7. Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision. Springer, 184–199.Google ScholarCross Ref
8. Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2016. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence 38, 2 (2016), 295–307. Google ScholarDigital Library
9. Raanan Fattal. 2007. Image Upsampling via Imposed Edge Statistics. In ACM SIGGRAPH 2007 Papers (SIGGRAPH ’07). ACM, New York, NY, USA, Article 95. Google ScholarDigital Library
10. William T Freeman, Thouis R Jones, and Egon C Pasztor. 2002. Example-based super-resolution. IEEE Computer graphics and Applications 22, 2 (2002), 56–65. Google ScholarDigital Library
11. Leon Gatys, Alexander S Ecker, and Matthias Bethge. 2015. Texture synthesis using convolutional neural networks. In Advances in Neural Information Processing Systems. 262–270. Google ScholarDigital Library
12. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2414–2423.Google ScholarCross Ref
13. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672–2680. Google ScholarDigital Library
14. B.K. Gunturk, Y. Altunbasak, and R.M. Mersereau. 2004. Superresolution reconstruction of compressed video using transform domain statistics. IEEE Transactions on Image Processing 13, 1 (2004), 33–43. Google ScholarDigital Library
15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision. 1026–1034. Google ScholarDigital Library
16. Y. Hu, K.M. Lam, G. Qiu, and T. Shen. 2010. From local pixel structure to global image super-resolution: a new face hallucination framework. IEEE Transactions on Image Processing 20, 2 (2010), 433–445. Google ScholarDigital Library
17. Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5197–5206.Google ScholarCross Ref
18. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093 (2014).Google Scholar
19. Yang Jianchao, John Wright, Thomas Huang, and Yi Ma. 2008. Image super-resolution as sparse representation of raw image patches. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition. 1–8.Google ScholarCross Ref
20. Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1646–1654.Google ScholarCross Ref
21. Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017a. Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
22. Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017b. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 624–632.Google ScholarCross Ref
23. Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
24. Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Vol. 1. 3.Google ScholarCross Ref
25. Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Pablo Arbeláez, and Luc Van Gool. 2018. Convolutional oriented boundaries: From image segmentation to high-level tasks. IEEE transactions on pattern analysis and machine intelligence 40, 4 (2018), 819–833.Google Scholar
26. David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on, Vol. 2. IEEE, 416–423.Google ScholarCross Ref
27. Viorica Pătrăucean, Pierre Gurdjos, and Rafael Grompone Von Gioi. 2012. A parameterless line segment and elliptical arc detector with enhanced ellipse fitting. In Computer Vision-ECCV 2012. Springer, 572–585.Google ScholarCross Ref
28. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (2015), 211–252. Google ScholarDigital Library
29. C.A. Segall, R. Molina, and A.K. Katsaggelos. 2003. High-resolution images from low-resolution compressed video. IEEE Signal Processing Mag. 20, 3 (2003), 37–48.Google ScholarCross Ref
30. Omry Sendik and Daniel Cohen-Or. 2017. Deep correlations for texture synthesis. ACM Transactions on Graphics (TOG) 36, 5 (2017), 161. Google ScholarDigital Library
31. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
32. Libin Sun and James Hays. 2017. Super-resolution Using Constrained Deep Texture Synthesis. arXiv preprint arXiv:1701.07604 (2017).Google Scholar
33. Ying Tai, Jian Yang, and Xiaoming Liu. 2017a. Image super-resolution via deep recursive residual network. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1.Google ScholarCross Ref
34. Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. 2017b. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4539–4547.Google ScholarCross Ref
35. Yu-Wing Tai, Shuaicheng Liu, Michael S Brown, and Stephen Lin. 2010. Super resolution using edge prior and single image detail synthesis. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR), Vol. 00. 2400–2407.Google ScholarCross Ref
36. Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee, et al. 2017. Ntire 2017 challenge on single image super-resolution: Methods and results. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on. IEEE, 1110–1121.Google ScholarCross Ref
37. Radu Timofte, Vincent De Smet, and Luc Van Gool. 2013. Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of the IEEE International Conference on Computer Vision. 1920–1927. Google ScholarDigital Library
38. Radu Timofte, Vincent De Smet, and Luc Van Gool. 2014. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Asian Conference on Computer Vision. Springer, 111–126.Google Scholar
39. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612. Google ScholarDigital Library
40. Saining Xie and Zhuowen Tu. 2015. Holistically-nested edge detection. In Proceedings of the IEEE international conference on computer vision. 1395–1403. Google ScholarDigital Library
41. D. Yldrm and O. Gungor. 2012. A novel image fusion method using IKONOS satellite images. Journal of Geodesy and Geoinformation 1, 1 (2012), 27–34.Google ScholarCross Ref
42. Roman Zeyde, Michael Elad, and Matan Protter. 2010. On single image scale-up using sparse-representations. In International conference on curves and surfaces. Springer, 711–730. Google ScholarDigital Library
43. Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual Dense Network for Image Super-Resolution. In CVPR.Google Scholar
44. Qiang Zhou, Shifeng Chen, Jianzhuang Liu, and Xiaoou Tang. 2011. Edge-preserving Single Image Super-resolution. In Proceedings of the 19th ACM International Conference on Multimedia (MM ’11). ACM, New York, NY, USA, 1037–1040. Google ScholarDigital Library
