“Blind image super-resolution with spatially variant degradations” by Cornillère, Djelouah, Wang, Sorkine-Hornung and Schroers
Conference:
Type(s):
Title:
- Blind image super-resolution with spatially variant degradations
Session/Category Title: Photography in the Field
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Existing deep learning approaches to single image super-resolution have achieved impressive results but mostly assume a setting with fixed pairs of high resolution and low resolution images. However, to robustly address realistic upscaling scenarios where the relation between high resolution and low resolution images is unknown, blind image super-resolution is required. To this end, we propose a solution that relies on three components: First, we use a degradation aware SR network to synthesize the HR image given a low resolution image and the corresponding blur kernel. Second, we train a kernel discriminator to analyze the generated high resolution image in order to predict errors present due to providing an incorrect blur kernel to the generator. Finally, we present an optimization procedure that is able to recover both the degradation kernel and the high resolution image by minimizing the error predicted by our kernel discriminator. We also show how to extend our approach to spatially variant degradations that typically arise in visual effects pipelines when compositing content from different sources and how to enable both local and global user interaction in the upscaling process.
References:
1. Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. 2010. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence 33, 5 (2010), 898–916.Google Scholar
2. Isabelle Begin and FR Ferrie. 2004. Blind super-resolution using a learning-based approach. In ICPR.Google Scholar
3. Isabelle Begin and Frank P Ferrie. 2007. PSF recovery from examples for blind super-resolution. In ICIP.Google Scholar
4. Siavash Arjomand Bigdeli, Matthias Zwicker, Paolo Favaro, and Meiguang Jin. 2017. Deep mean-shift priors for image restoration. In Advances in Neural Information Processing Systems.Google Scholar
5. Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Real-time video super-resolution with spatiotemporal networks and motion compensation. In CVPR.Google Scholar
6. Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. In ECCV.Google Scholar
7. Chao Dong, Chen Change Loy, and Xiaoou Tang. 2016. Accelerating the super-resolution convolutional neural network. In ECCV.Google Scholar
8. Netalee Efrat, Daniel Glasner, Alexander Apartsin, Boaz Nadler, and Anat Levin. 2013. Accurate blur models vs. image priors in single image super-resolution. In ICCV.Google Scholar
9. Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. 2019. Blind super-resolution with iterative kernel correction. In CVPR.Google Scholar
10. Neel Joshi, Richard Szeliski, and David J Kriegman. 2008. PSF estimation using sharp edge prediction. In CVPR.Google Scholar
11. Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In CVPR.Google Scholar
12. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
13. Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR.Google Scholar
14. Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In CVPR Workshops.Google ScholarCross Ref
15. Tomer Michaeli and Michal Irani. 2013. Nonparametric blind super-resolution. In ICCV.Google Scholar
16. Kamal Nasrollahi and Thomas B Moeslund. 2014. Super-resolution: a comprehensive survey. Machine vision and applications (2014).Google Scholar
17. Jianping Qiao, Ju Liu, and Caihua Zhao. 2006. A novel SVM-based blind super-resolution algorithm. In The 2006 IEEE International Joint Conference on Neural Network Proceedings.Google Scholar
18. JH Rick Chang, Chun-Liang Li, Barnabas Poczos, BVK Vijaya Kumar, and Aswin C Sankaranarayanan. 2017. One Network to Solve Them All-Solving Linear Inverse Problems Using Deep Projection Models. In ICCV.Google Scholar
19. Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018. Frame-recurrent video super-resolution. In CVPR.Google Scholar
20. Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR.Google Scholar
21. Assaf Shocher, Nadav Cohen, and Michal Irani. 2018. “Zero-Shot” Super-Resolution using Deep Internal Learning. In CVPR.Google Scholar
22. Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. 2017. Ntire 2017 challenge on single image super-resolution: Methods and results. In CVPR Workshops.Google ScholarCross Ref
23. Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine-Hornung, Olga Sorkine-Hornung, and Christopher Schroers. 2018. A Fully Progressive Approach to Single-Image Super-Resolution. (2018).Google Scholar
24. Chih-Yuan Yang, Chao Ma, and Ming-Hsuan Yang. 2014. Single-image super-resolution: A benchmark. In ECCV.Google Scholar
25. Roman Zeyde, Michael Elad, and Matan Protter. 2010. On single image scale-up using sparse-representations. In International conference on curves and surfaces. Springer, 711–730.Google Scholar
26. Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. 2017. Learning deep CNN denoiser prior for image restoration. In CVPR.Google Scholar
27. Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2018d. Learning a single convolutional super-resolution network for multiple degradations. In CVPR.Google Scholar
28. Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2019. Deep Plug-and-Play Super-Resolution for Arbitrary Blur Kernels. CVPR (2019).Google Scholar
29. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018b. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.Google Scholar
30. Xinyi Zhang, Hang Dong, Zhe Hu, Wei-Sheng Lai, Fei Wang, and Ming-Hsuan Yang. 2018a. Gated Fusion Network for Joint Image Deblurring and Super-Resolution. In BMVC.Google Scholar
31. Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018c. Image super-resolution using very deep residual channel attention networks. In ECCV.Google Scholar
