“Image and Video Upscaling From Local Self-Examples” by Feedman and Fattal

We propose a new high-quality and efficient single-image upscaling technique that extends existing example-based super-resolution frameworks. In our approach we do not rely on an external example database or use the whole input image as a source for example patches. Instead, we follow a local self-similarity assumption on natural images and extract patches from extremely localized regions in the input image. This allows us to reduce considerably the nearest-patch search time without compromising quality in most images. Tests, that we perform and report, show that the local self-similarity assumption holds better for small scaling factors where there are more example patches of greater relevance. We implement these small scalings using dedicated novel nondyadic filter banks, that we derive based on principles that model the upscaling process. Moreover, the new filters are nearly biorthogonal and hence produce high-resolution images that are highly consistent with the input image without solving implicit back-projection equations. The local and explicit nature of our algorithm makes it simple, efficient, and allows a trivial parallel implementation on a GPU. We demonstrate the new method ability to produce high-quality resolution enhancement, its application to video sequences with no algorithmic modification, and its efficiency to perform real-time enhancement of low-resolution video standard into recent high-definition formats.

1. Aly, H. and Dubois, E. 2005. Image up-sampling using total-variation regularization with a new observation model. IEEE Trans. Image Process. 14, 10, 1647–1659.
2. Barnsley, M. 1988. Fractal modelling of real world images. In The Science of Fractal Images, H.-O. Peitgen and D. Saupe, Eds. Springer, Berlin.
3. Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Curless, B., Cohen, M., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Proceedings of the Conference on Rendering Techniques, J. Kautz and S. Pattanaik, Eds. Eurographics, 327–338.
4. Ebrahimi, M. and Vrscay, E. R. 2007. Solving the inverse problem of image zooming using self-examples. In Proceedings of the International Conference on Image Analysis and Recognition ICIAR. Lecture Notes in Computer Science, vol. 4633. Springer, 117–130.
5. Farsiu, S., Robinson, M., Elad, M., and Milanfar, P. 2004. Fast and robust multiframe super resolution. IEEE Trans. Image Process. 13, 10, 1327–1344.
6. Fattal, R. 2007. Image upsampling via imposed edge statistics. ACM Trans. Graph. 26, 3, 95.
7. Freeman, W. T., Jones, T. R., and Pasztor, E. C. 2002. Example-based super-resolution. IEEE Comput. Graph. Appl. 22, 2, 56–65.
8. Freeman, W. T., Pasztor, E. C., and Carmichael, O. T. 2000. Learning low-level vision. Int. J. Comput. Vision 40, 1, 25–47.
9. Glasner, D., Bagon, S., and Irani, M. 2009. Super-Resolution from a single image. In Proceedings of the IEEE International Conference on Computer Vision (ICCV09). 349–356.
10. Li, X. and Orchard, M. T. 2001. New edge-directed interpolation. IEEE Trans. Image Process. 10, 10, 1521–1527.
11. Lin, Z. and Shum, H.-Y. 2004. Fundamental limits of reconstruction-based superresolution algorithms under local translation. IEEE Trans. Patt. Anal. Mach. Intell. 26, 1, 83–97.
12. Mallat, S. 1999. A Wavelet Tour of Signal Processing, 2nd ed. (Wavelet Analysis & Its Applications). Academic Press.
13. Polidori, E. and Dugelay, J.-L. 1995. Zooming using iterated function systems. In NATO ASI on Image Coding and Analysis.
14. Pollock, S. and Cascio, I. 2007. Non-Dyadic wavelet analysis. In Optimisation, Econometric and Financial Analysis, 167–203.
15. Pratt, W. K. 2001. Digital Image Processing: PIKS Inside. John Wiley & Sons, New York.
16. Reusens, E. 1994. Overlapped adaptive partitioning for image coding based on the theory of iterated functions systems. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing. (ICASSP 94). vol. 5.
17. Robert, M. G.-A., Denardo, R., Tenda, Y., and Huang, T. S. 1997. Resolution enhancement of images using fractal coding. In Proceedings of the Conference on Visual Communications and Image Processing ’97. 1089–1100.
18. Shan, Q., Li, Z., Jia, J., and Tang, C.-K. 2008. Fast image/video upsampling. ACM Trans. Graph. 27, 5, 1–7.
19. Su, D. and Willis, P. 2004. Image interpolation by pixel-level data-dependent triangulation. Comput. Graph. Forum 23, 2, 189–202.
20. Suetake, N., Sakano, M., and Uchino, E. 2008. Image super-resolution based on local self-similarity. J. Optic. Rev. 15, 1 (January), 26–30.
21. Sun, J., Ning Zheng, N., Tao, H., and Yeung Shum, H. 2003. Image hallucination with primal sketch priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 729–736.
22. Sun, J., Xu, Z., and Shum, H.-Y. 2008. Image super-resolution using gradient profile prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1–8.
23. Tappen, M. F., Russell, B. C., and Freeman, W. T. 2004. Efficient graphical models for processing images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 673–680.
24. Thvenaz, P., Blu, T., and Unser, M. 2000. Image interpolation and resampling. In Handbook of Medical Imaging, Processing and Analysis, I. Bankman, Ed. Academic Press, San Diego CA, 393–420.
25. Vrscay, E. R. 2002. From Fractal Image Compression to Fractal-Based Methods in Mathematics. The IMA Volumes in Mathematics and Its Applications. Springer, Berlin.
26. Xiong, R., Xu, J., and Wu, F. 2006. A lifting-based wavelet transform supporting non-dyadic spatial scalability. In Proceedings of the IEEE International Conference on Image Processing. 1861–1864.