“Perceptually based downscaling of images” by Fattal

  • ©Cengiz Öztireli and Markus Gross



Session Title:

    Image Processing


    Perceptually based downscaling of images




    We propose a perceptually based method for downscaling images that provides a better apparent depiction of the input image. We formulate image downscaling as an optimization problem where the difference between the input and output images is measured using a widely adopted perceptual image quality metric. The downscaled images retain perceptually important features and details, resulting in an accurate and spatio-temporally consistent representation of the high resolution input. We derive the solution of the optimization problem in closed-form, which leads to a simple, efficient and parallelizable implementation with sums and convolutions. The algorithm has running times similar to linear filtering and is orders of magnitude faster than the state-of-the-art for image downscaling. We validate the effectiveness of the technique with extensive tests on many images, video, and by performing a user study, which indicates a clear preference for the results of the new algorithm.


    1. Banterle, F., Artusi, A., Aydin, T., Didyk, P., Eisemann, E., Gutierrez, D., Mantiuk, R., and Myszkowski, K. 2011. Multidimensional image retargeting. In ACM Siggraph Asia 2011 Courses, ACM, ACM Siggraph Asia. Google ScholarDigital Library
    2. Bonnier, N., Schmitt, F., Brettel, H., and Berche, S. 2006. Evaluation of spatial gamut mapping algorithms. In Proc. 14th Color Imag. Conf., 56–61.Google Scholar
    3. Brunet, D., Vrscay, E., and Wang, Z. 2010. Structural similarity-based approximation of signals and images using orthogonal bases. In Image Analysis and Recognition, A. Campilho and M. Kamel, Eds., vol. 6111 of Lecture Notes in Computer Science. Springer Berlin Heidelberg, 11–22. Google ScholarDigital Library
    4. Brunet, D., Vrscay, E., and Wang, Z. 2012. On the mathematical properties of the structural similarity index. Image Processing, IEEE Trans. on 21, 4 (April), 1488–1499. Google ScholarDigital Library
    5. Brunet, D. 2012. A Study of the Structural Similarity Image Quality Measure with Applications to Image Processing. PhD thesis, University of Waterloo.Google Scholar
    6. Chai, L., Sheng, Y., and Zhang, J. 2014. Ssim performance limitation of linear equalizers. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, 1220–1224.Google Scholar
    7. Channappayya, S., Bovik, A., and Heath, R. 2006. A linear estimator optimized for the structural similarity index and its application to image denoising. In Image Processing, 2006 IEEE International Conference on, 2637–2640.Google Scholar
    8. Channappayya, S., Bovik, A., Caramanis, C., and Heath, R. 2008. Ssim-optimal linear image restoration. In Acoustics, Speech and Signal Processing (ICASSP), 2008. IEEE International Conference on, 765–768.Google Scholar
    9. Channappayya, S., Bovik, A., and Heath, R. 2008. Rate bounds on ssim index of quantized images. Image Processing, IEEE Trans. on 17, 9 (Sept), 1624–1639. Google ScholarDigital Library
    10. Channappayya, S. S., Bovik, A. C., Caramanis, C., and Jr., R. W. H. 2008. Design of linear equalizers optimized for the structural similarity index. Image Processing, IEEE Trans. on 17, 6, 857–872. Google ScholarDigital Library
    11. Chen, G.-H., Yang, C.-L., and Xie, S.-L. 2006. Gradient-based structural similarity for image quality assessment. In Image Processing, IEEE International Conference on, 2929–2932.Google Scholar
    12. Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., and Sen, P. 2012. Image Melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph. (Proc. of SIGGRAPH 2012) 31, 4, 82:1–82:10. Google ScholarDigital Library
    13. Demirtas, A., Reibman, A., and Jafarkhani, H. 2014. Full-reference quality estimation for images with different spatial resolutions. Image Processing, IEEE Trans. on 23, 5 (May), 2069–2080.Google ScholarCross Ref
    14. Didyk, P., Ritschel, T., Eisemann, E., and Myszkowski, K. 2012. Perceptual Digital Imaging: Methods and Applications. CRC Press, ch. Exceeding Physical Limitations: Apparent Display Qualities.Google Scholar
    15. Dong, J., and Ye, Y. 2012. Adaptive downsampling for high-definition video coding. In ICIP 2012, 2925–2928.Google Scholar
    16. Gerstner, T., DeCarlo, D., Alexa, M., Finkelstein, A., Gingold, Y., and Nealen, A. 2012. Pixelated image abstraction. In NPAR 2012, Proc. of the 10th International Symposium on Non-photorealistic Animation and Rendering. Google ScholarDigital Library
    17. He, L., Gao, F., Hou, W., and Hao, L. 2014. Objective image quality assessment: A survey. Int. J. Comput. Math. 91, 11 (Nov.), 2374–2388. Google ScholarDigital Library
    18. Kopf, J., Shamir, A., and Peers, P. 2013. Content-adaptive image downscaling. ACM Trans. Graph. 32, 6 (Nov.), 173:1–173:8. Google ScholarDigital Library
    19. Krawczyk, G., Myszkowski, K., and Seidel, H.-P. 2007. Contrast restoration by adaptive countershading. In Proc. of Eurographics 2007, Blackwell, vol. 26 of Computer Graphics Forum.Google Scholar
    20. Lissner, I., Preiss, J., Urban, P., Lichtenauer, M. S., and Zolliker, P. 2013. Image-difference prediction: From grayscale to color. Image Processing, IEEE Trans. on 22, 2, 435–446. Google ScholarDigital Library
    21. Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., and Shum, H.-Y. 2011. Learning to detect a salient object. Pattern Analysis and Machine Intelligence, IEEE Trans. on 33, 2 (Feb), 353–367. Google ScholarDigital Library
    22. Mitchell, D. P., and Netravali, A. N. 1988. Reconstruction filters in computer-graphics. In Proc. of SIGGRAPH ’88, ACM, New York, NY, USA, 221–228. Google ScholarDigital Library
    23. Nehab, D., and Hoppe, H. 2011. Generalized sampling in computer graphics. Tech. Rep. MSR-TR-2011-16, February.Google Scholar
    24. Ogawa, T., and Haseyama, M. 2013. Image inpainting based on sparse representations with a perceptual metric. EURASIP Journal on Advances in Signal Processing 2013, 1.Google ScholarCross Ref
    25. Pang, W.-M., Qu, Y., Wong, T.-T., Cohen-Or, D., and Heng, P.-A. 2008. Structure-aware halftoning. ACM Trans. Graph. 27, 3 (Aug.), 89:1–89:8. Google ScholarDigital Library
    26. Polesel, A., Ramponi, G., and Mathews, V. J. 1997. Adaptive unsharp masking for contrast enhancement. In ICIP ’97 3- Volume Set-Volume 1 – Volume 1, IEEE Computer Society, Washington, DC, USA, 267–. Google ScholarDigital Library
    27. Rehman, A., Wang, Z., Brunet, D., and Vrscay, E. 2011. Ssim-inspired image denoising using sparse representations. In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, 1121–1124.Google Scholar
    28. Ritschel, T., Smith, K., Ihrke, M., Grosch, T., Myszkowski, K., and Seidel, H.-P. 2008. 3D Unsharp Masking for Scene Coherent Enhancement. ACM Trans. Graph. (Proc. of SIGGRAPH 2008) 27, 3. Google ScholarDigital Library
    29. Shannon, C. 1998. Communication in the presence of noise. Proc. of the IEEE 86, 2 (Feb), 447–457.Google ScholarCross Ref
    30. Shao, Y., Sun, F., Li, H., and Liu, Y. 2014. Structural similarity-optimal total variation algorithm for image denoising. In Foundations and Practical Applications of Cognitive Systems and Information Processing, vol. 215. Springer Berlin Heidelberg, 833–843.Google Scholar
    31. Silvestre-Blanes, J. 2011. Structural similarity image quality reliability: Determining parameters and window size. Signal Processing 91, 4, 1012–1020. Google ScholarDigital Library
    32. Smith, K., Landes, P.-E., Thollot, J., and Myszkowski, K. 2008. Apparent greyscale: A simple and fast conversion to perceptually accurate images and video. Computer Graphics Forum (Proc. of Eurographics 2008) 27, 2 (apr).Google Scholar
    33. Thévenaz, P., Blu, T., and Unser, M. 2000. Interpolation revisited. Medical Imaging, IEEE Trans. on 19, 7, 739–758.Google ScholarCross Ref
    34. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Computer Vision, 1998. Sixth International Conference on, 839–846. Google ScholarDigital Library
    35. Trentacoste, M., Mantiuk, R., and Heidrich, W. 2011. Blur-Aware Image Downsizing. In Proc. of Eurographics.Google Scholar
    36. Wang, Z., and Bovik, A. 2009. Mean squared error: Love it or leave it? a new look at signal fidelity measures. Signal Processing Magazine, IEEE 26, 1 (Jan), 98–117.Google ScholarCross Ref
    37. Wang, Z., and Li, Q. 2007. Video quality assessment using a statistical model of human visual speed perception. J. Opt. Soc. Am. A 24, 12, B61B69.Google ScholarCross Ref
    38. Wang, Z., Bovik, A., Sheikh, H., and Simoncelli, E. 2004. Image quality assessment: from error visibility to structural similarity. Image Processing, IEEE Trans. on 13, 4 (April), 600–612. Google ScholarDigital Library
    39. Wang, S., Rehman, A., Wang, Z., Ma, S., and Gao, W. 2011. Rate-ssim optimization for video coding. In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, 833–836.Google Scholar
    40. Wu, X., Zhang, X., and Wang, X. 2009. Low bit-rate image compression via adaptive down-sampling and constrained least squares upconversion. Trans. Img. Proc. 18, 3 (Mar.), 552–561. Google ScholarDigital Library
    41. Yeganeh, H. 2014. Cross Dynamic Range And Cross Resolution Objective Image Quality Assessment With Applications. PhD thesis, University of Waterloo.Google Scholar
    42. Zhang, Y., Zhao, D., Zhang, J., Xiong, R., and Gao, W. 2011. Interpolation-dependent image downsampling. Image Processing, IEEE Trans. on 20, 11 (Nov), 3291–3296. Google ScholarDigital Library
    43. Zhang, L., Zhang, L., Mou, X., and Zhang, D. 2012. A comprehensive evaluation of full reference image quality assessment algorithms. In ICIP 2012, 1477–1480.Google Scholar
    44. Zhou, F., and Liao, Q. 2015. Single-frame image super-resolution inspired by perceptual criteria. Image Processing, IET 9, 1, 1–11.Google ScholarCross Ref

ACM Digital Library Publication: