“New measurements reveal weaknesses of image quality metrics in evaluating graphics artifacts”
Conference:
Type(s):
Title:
- New measurements reveal weaknesses of image quality metrics in evaluating graphics artifacts
Session/Category Title: Materials and Images
Presenter(s)/Author(s):
Abstract:
Reliable detection of global illumination and rendering artifacts in the form of localized distortion maps is important for many graphics applications. Although many quality metrics have been developed for this task, they are often tuned for compression/transmission artifacts and have not been evaluated in the context of synthetic CG-images. In this work, we run two experiments where observers use a brush-painting interface to directly mark image regions with noticeable/objectionable distortions in the presence/absence of a high-quality reference image, respectively. The collected data shows a relatively high correlation between the with-reference and no-reference observer markings. Also, our demanding per-pixel image-quality datasets reveal weaknesses of both simple (PSNR, MSE, sCIE-Lab) and advanced (SSIM, MS-SSIM, HDR-VDP-2) quality metrics. The most problematic are excessive sensitivity to brightness and contrast changes, the calibration for near visibility-threshold distortions, lack of discrimination between plausible/implausible illumination, and poor spatial localization of distortions for multi-scale metrics. We believe that our datasets have further potential in improving existing quality metrics, but also in analyzing the saliency of rendering distortions, and investigating visual equivalence given our with- and no-reference data.
References:
1. Baldi, P., Brunak, S., Chauvin, Y., Andersen, C. A. F., and Nielsen, H. 2000. Assessing the accuracy of prediction algorithms for classification: an overview. Bioinformatics 16, 5 (May), 412–424.
2. Bolin, M., and Meyer, G. 1998. A perceptually based adaptive sampling algorithm. In Proc. of SIGGRAPH, 299–310.
3. Čadík, M., Aydin, T. O., Myszkowski, K., and Seidel, H.-P. 2011. On evaluation of video quality metrics: an HDR dataset for computer graphics applications. In SPIE HVEI XVI.
4. Chandler, D., and Hemami, S. 2007. VSNR: a wavelet-based visual signal-to-noise ratio for natural images. IEEE Trans. on Image Processing 16, 9, 2284–2298.
5. Daly, S. 1993. The Visible Differences Predictor: An algorithm for the assessment of image fidelity. In Digital Images and Human Vision, MIT Press, 179–206.
6. Drago, F., Myszkowski, K., Annen, T., and N. Chiba. 2003. Adaptive logarithmic mapping for displaying high contrast scenes. Proc. of Eurographics 22, 3, 419–426.
7. Fattal, R., Lischinski, D., and Werman, M. 2002. Gradient domain high dynamic range compression. In Proc. of SIGGRAPH, 249–256.
8. Hachisuka, T., Ogaki, S., and Jensen, H. W. 2008. Progressive photon mapping. In Proc. of SIGGRAPH Asia, 130:1–130:8.
9. Herzog, R., Myszkowski, K., and Seidel, H.-P. 2009. Anisotropic radiance-cache splatting for efficiently computing high-quality GI with lightcuts. Proc. of Eurographics, 259–268.
10. Herzog, R., Čadík, M., Aydin, T. O., Kim, K. I., Myszkowski, K., and Seidel, H.-P. 2012. NoRM: no-reference image quality metric for realistic image synthesis. Computer Graphics Forum 31, 2, 545–554.
11. Howell, D. C. 2007. Statistical Methods for Psychology, 6th edition ed. Thomas Wadsworth.
12. ITU-R-BT.500-11, 2002. Methodology for the subjective assessment of the quality of television pictures.
13. ITU-T-P.910. 2008. Subjective audiovisual quality assessment methods for multimedia applications. Tech. rep.
14. Jensen, H. W. 2001. Realistic Image Synthesis Using Photon Mapping. AK, Peters.
15. Keller, A. 1997. Instant radiosity. In Proc. of SIGGRAPH, 49–56.
16. Křivánek, J., Gautron, P., Pattanaik, S., and Bouatouch, K. 2005. Radiance caching for efficient global illumination computation. IEEE TVCG 11, 5, 550–561.
17. Larson, E. C., and Chandler, D. M. 2010. Most apparent distortion: full-reference image quality assessment and the role of strategy. J. Electron. Imaging 19, 1, 011006:1–21.
18. Ledda, P., Chalmers, A., Troscianko, T., and Seetzen, H. 2005. Evaluation of tone mapping operators using a high dynamic range display. Proc. of SIGGRAPH 24, 3, 640–648.
19. Lin, W., and Kuo, C.-C. J. 2011. Perceptual visual quality metrics: A survey. JVCIR, 297–312.
20. Lubin, J. 1995. Vision Models for Target Detection and Recognition. World Scientific, ch. A Visual Discrimination Model for Imaging System Design and Evaluation, 245–283.
21. Mantiuk, R., Daly, S., Myszkowski, K., and Seidel, H.-P. 2005. Predicting visible differences in high dynamic range images – model and its calibration. In SPIE HVEI X.
22. Mantiuk, R., Myszkowski, K., and Seidel, H.-P. 2006. A perceptual framework for contrast processing of high dynamic range images. ACM Trans. on Applied Perception 3, 3, 286–308.
23. Mantiuk, R., Daly, S., and Kerofsky, L. 2008. Display adaptive tone mapping. In Proc. of SIGGRAPH, vol. 27(3), #68.
24. Mantiuk, R., Kim, K. J., Rempel, A. G., and Heidrich, W. 2011. HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. In Proc. of SIGGRAPH, #40.
25. Pedersen, M., and Hardeberg, Jon, Y. 2011. Full-Reference Image Quality Metrics: Classification and Evaluation. Foundations and Trends in Computer Graphics and Vision 7, 1, 1–80.
26. Ponomarenko, N., Lukin, V., Zelensky, A., Egiazarian, K., Carli, M., and Battisti, F. 2009. TID2008 – A database for evaluation of full-reference visual quality assessment metrics. Advances of Modern Radioelectronics 10, 30–45.
27. Ramanarayanan, G., Ferwerda, J., Walter, B., and Bala, K. 2007. Visual equivalence: towards a new standard for image fidelity. In Proc. of SIGGRAPH, #76.
28. Ramasubramanian, M., Pattanaik, S., and Greenberg, D. 1999. A perceptually based physical error metric for realistic image synthesis. In Proc. of SIGGRAPH, 73–82.
29. Reinhard, E., Stark, M. M., Shirley, P., and Ferwerda, J. A. 2002. Photographic tone reproduction for digital images. In Proc. of SIGGRAPH, 267–276.
30. Rushmeier, H., Ward, G., Piatko, C., Sanders, P., and Rust, B. 1995. Comparing real and synthetic images: some ideas about metrics. In Rendering Techniques ’95, 82–91.
31. Salkind, N., Ed. 2007. Encyclopedia of measurement and statistics. A Sage reference publication. SAGE, Thousand Oaks.
32. Sheikh, H., Sabir, M., and Bovik, A. 2006. A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans. on Image Processing 15, 11, 3440–3451.
33. Walter, B., Fernandez, S., Arbree, A., Bala, K., Donikian, M., and Greenberg, D. 2005. Lightcuts: A scalable approach to illumination. Proc. of SIGGRAPH, 1098–1107.
34. Wang, Z., and Bovik, A. C. 2006. Modern Image Quality Assessment. Morgan & Claypool Publishers.
35. Wang, Z., Simoncelli, E. P., and Bovik, A. C. 2003. Multi-scale structural similarity for image quality assessment. In Proc. IEEE Asilomar Conf. on Signals, Systems & Comp., 1398–1402.
36. Ward, G. J., Rubinstein, F. M., and Clear, R. D. 1988. A ray tracing solution for diffuse interreflection. In Proc. of SIGGRAPH, 85–92.
37. Wu, H., and Rao, K. 2005. Digital Video Image Quality and Perceptual Coding. CRC Press.
38. Zhang, X., and Wandell, B. A. 1998. Color image fidelity metrics evaluated using image distortion maps. Signal Proc. 70, 3, 201–214.
