Image Quality Assessment without Reference by Combining Deep Learning-Based Features and Viewing Distance
Source abstract
Abstract
An abundance of objective image quality metrics have been introduced in the literature. One important essential aspect that perceived image quality is dependent on is the viewing distance from the observer to the image. We introduce in this study a novel image quality metric able to estimate the quality of a given image without reference for different viewing distances between the image and the observer. We first select relevant patches from the image using saliency information. For each patch, a feature vector is extracted from a convolutional neural network model and concatenated at the viewing distance, for which the quality is predicted. The resulting vector is fed to fully connected layers to predict subjective scores for the considered viewing distance. The proposed method was evaluated using the Colourlab Image Database: Image Quality and Viewing Distance-changed Image Database. Both databases provide subjective scores at two different viewing distances. In the Colourlab Image Database: Image Quality we obtain a Pearson correlation of 0.87 at both 50 cm and 100 cm viewing distances, while in the Viewing Distance-changed Image Database we obtained a Pearson correlation of 0.93 and 0.94 at viewing distance of four and six times the image height. The results show the efficiency of our method and its generalization ability.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Image and Video Quality Assessment · Advanced Image Fusion Techniques · Visual Attention and Saliency Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
PRISME Laboratory, University of Orleans, 45072 Orleans, France
Department of Computer Science, Norwegian University of Science and Technology, 2802 Gjøvik, Norway
References (100+)
Show source-provided references
1. Full-reference image quality metrics: Classification and evaluation
2. Perceptual visual quality metrics: A survey10.1016/j.jvcir.2011.01.005 ↗
3. Engelke, U., and Zepernick, H.J. (2007, January 21–23). Perceptual-based quality metrics for image and video services: A survey. Proceedings of the 2007 Next Generation Internet Networks, Trondheim, Norway.10.1109/NGI.2007.371215 ↗
4. Thung, K.H., and Raveendran, P. (2009, January 14–15). A survey of image quality measures. Proceedings of the 2009 International Conference for Technical Postgraduates (TECHPOS), Kuala Lumpur, Malaysia.10.1109/TECHPOS.2009.5412098 ↗
5. Image quality measures and their performance10.1109/26.477498 ↗
6. Computational image quality metrics: A review
7. Pedersen, M. (2015, January 27–30). Evaluation of 60 full-reference image quality metrics on the CID:IQ. Proceedings of the IEEE International Conference on Image Processing, Quebec City, QC, Canada.10.1109/ICIP.2015.7351068 ↗
8. Chetouani, A. (2014, January 24–28). Full Reference Image Quality Assessment: Limitation. Proceedings of the 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden.10.1109/ICPR.2014.153 ↗
9. Statistical evaluation of image quality measures10.1117/1.1455011 ↗
10. TID2008-a database for evaluation of full-reference visual quality assessment metrics
11. Zhang, L., Zhang, L., Mou, X., and Zhang, D. (October, January 30). A comprehensive evaluation of full reference image quality assessment algorithms. Proceedings of the IEEE International Conference on Image Processing, Orlando, FL, USA.
12. Full-reference image quality metrics performance evaluation over image quality databases10.1007/s13369-012-0509-6 ↗
13. Objective Image Quality Assessment: Facing The Real-World Challenges
14. Chandler, D.M. (2013). Seven challenges in image quality assessment: Past, present, and future research. ISRN Signal Process., 2013.10.1155/2013/905685 ↗
15. Future Directions in Image Quality10.2352/issn.2169-2629.2019.27.71 ↗
16. Wang, Z., Bovik, A.C., and Lu, L. (2002, January 13–17). Why is image quality assessment so difficult?. Proceedings of the 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, Orlando, FL, USA.10.1109/ICASSP.2002.5745362 ↗
17. Modern image quality assessment10.1007/978-3-031-02238-8 ↗
18. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
19. A spatial extension of CIELAB for digital color-image reproduction10.1889/1.1985127 ↗
20. An image difference metric based on simulation of image detail visibility and total variation10.2352/CIC.2014.22.1.art00005 ↗
21. Ponomarenko, N., Silvestri, F., Egiazarian, K., Carli, M., Astola, J., and Lukin, V. On between-coefficient contrast masking of DCT basis functions. Proceedings of the Third International Workshop on Video Processing and Quality Metrics.
22. Analysis of the difference of gaussians model in image difference metrics10.2352/CGIV.2010.5.1.art00076 ↗
23. Machine learning to design full-reference image quality assessment algorithm10.1016/j.image.2012.01.002 ↗
24. A new spatial filtering based image difference metric based on hue angle weighting10.2352/J.ImagingSci.Technol.12.56.5.050501 ↗
25. Perceptual image quality assessment based on structural similarity and visual masking10.1016/j.image.2012.04.005 ↗
26. Simulation of image detail visibility using contrast sensitivity functions and wavelets10.2352/CIC.2012.20.1.art00013 ↗
27. Evaluation of Image Corrected by Retinex Method Based on S-CIELAB and Gazing Information10.1093/ietfec/e89-a.11.2955 ↗
28. Using gaze information to improve image difference metrics10.1117/12.764468 ↗
29. Pedersen, M., Zheng, Y., and Hardeberg, J.Y. (2011). Evaluation of image quality metrics for color prints. Scandinavian Conference on Image Analysis, Springer.10.1007/978-3-642-21227-7_30 ↗
30. Using image quality metrics to evaluate an icc printer profile10.2352/CIC.2010.18.1.art00043 ↗
31. Spatial pooling for measuring color printing quality attributes10.1016/j.jvcir.2012.03.010 ↗
32. Zhao, P., Cheng, Y., and Pedersen, M. (2015, January 25–26). Objective assessment of perceived sharpness of projection displays with a calibrated camera. Proceedings of the 2015 Colour and Visual Computing Symposium (CVCS), Gjovik, Norway.10.1109/CVCS.2015.7274892 ↗
33. Charrier, C., Knoblauch, K., Maloney, L.T., and Bovik, A.C. (2011, January 11–14). Calibrating MS-SSIM for compression distortions using MLDS. Proceedings of the 2011 18th IEEE International Conference on Image Processing, Brussels, Belgium.10.1109/ICIP.2011.6116381 ↗
34. Brooks, A.C., and Pappas, T.N. (2007, January 15–20). Using structural similarity quality metrics to evaluate image compression techniques. Proceedings of the 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP’07, Honolulu, HI, USA.10.1109/ICASSP.2007.366047 ↗
35. Seybold, T., Keimel, C., Knopp, M., and Stechele, W. (2013, January 9–11). Towards an evaluation of denoising algorithms with respect to realistic camera noise. Proceedings of the 2013 IEEE International Symposium on Multimedia, Anaheim, CA, USA.10.1109/ISM.2013.39 ↗
36. Amirshahi, S.A., Kadyrova, A., and Pedersen, M. (2019, January 28–31). How do image quality metrics perform on contrast enhanced images?. Proceedings of the 2019 8th European Workshop on Visual Information Processing (EUVIP), Roma, Italy.10.1109/EUVIP47703.2019.8946143 ↗
37. Saliency models as gamut-mapping artifact detectors10.2352/CGIV.2010.5.1.art00069 ↗
38. Evaluating colour image difference metrics for gamut-mapped images10.1111/j.1478-4408.2008.00148.x ↗
39. Image Quality Metrics for the Evaluation and Optimization of Capsule Video Endoscopy Enhancement Techniques10.2352/J.ImagingSci.Technol.2017.61.4.040402 ↗
40. Völgyes, D., Martinsen, A., Stray-Pedersen, A., Waaler, D., and Pedersen, M. (2018). A Weighted Histogram-Based Tone Mapping Algorithm for CT Images. Algorithms, 11.10.3390/a11080111 ↗
41. Yao, Z., Le Bars, J., Charrier, C., and Rosenberger, C. Fingerprint Quality Assessment Combining Blind Image Quality, Texture and Minutiae Features. Proceedings of the 1st International Conference on Information Systems Security and Privacy.
42. Performance evaluation of no-reference image quality metrics for face biometric images10.1117/1.JEI.27.2.023001 ↗
43. Jenadeleh, M., Pedersen, M., and Saupe, D. (2020). Blind Quality Assessment of Iris Images Acquired in Visible Light for Biometric Recognition. Sensors, 20.10.3390/s20051308 ↗
44. On the Use of Deep Learning for Blind Image Quality Assessment10.1007/s11760-017-1166-8 ↗
45. Kang, L., Ye, P., Li, Y., and Doermann, D. (2014, January 23–28). Convolutional Neural Networks for No-Reference Image Quality Assessment. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.224 ↗
46. No-reference image quality assessment with shearlet transform and deep neural networks10.1016/j.neucom.2014.12.015 ↗
47. Fully deep blind image quality predictor10.1109/JSTSP.2016.2639328 ↗
48. Lv, Y., Jiang, G., Yu, M., Xu, H., Shao, F., and Liu, S. (2015, January 27–30). Difference of Gaussian statistical features based blind image quality assessment: A deep learning approach. Proceedings of the IEEE International Conference on Image Processing, Quebec City, QC, Canada.10.1109/ICIP.2015.7351221 ↗
49. DeepSim: Deep similarity for image quality assessment10.1016/j.neucom.2017.01.054 ↗
50. Blind image quality assessment via deep learning
51. Image quality assessment by comparing CNN features between images10.2352/J.ImagingSci.Technol.2016.60.6.060410 ↗
52. Liu, X., Pedersen, M., and Hardeberg, J. (2014). CID:IQ-A New Image Quality Database. Image and Signal Processing, Springer.10.1007/978-3-319-07998-1_22 ↗
53. Quality assessment considering viewing distance and image resolution10.1109/TBC.2015.2459851 ↗
54. A hybrid system for distortion classification and image quality evaluation10.1016/j.image.2012.06.001 ↗
55. Sheikh, H. (2021, April 12). LIVE Image Quality Assessment Database Release 2. Available online: http://live.ece.utexas.edu/research/quality.
56. Chetouani, A. (2018, January 7–10). Convolutional Neural Network and Saliency Selection for Blind Image Quality Assessment. Proceedings of the IEEE International Conference on Image Processing, Athens, Greece.10.1109/ICIP.2018.8451654 ↗
57. Most apparent distortion: Full-reference image quality assessment and the role of strategy10.1117/1.3267105 ↗
58. Chetouani, A. (2019, January 22–25). Blind Utility and Quality Assessment Using a Convolutional Neural Network and a Patch Selection. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP.2019.8803829 ↗
59. Estimating the usefulness of distorted natural images using an image contour degradation measure10.1364/JOSAA.28.000157 ↗
60. Ninassi, A., Le Callet, P., and Autrusseau, F. (2018, March 24). Subjective Quality Assessment-IVC Database. Available online: http://www.irccyn.ec-nantes.fr/ivcdb.
61. Horita, Y., Shibata, K., Kawayoke, Y., and Sazzad, Z.P. (2015, July 27). MICT Image Quality Evaluation Database. Available online: http://mict.eng.u-toyama.ac.jp/mictdb.html.
62. Jayaraman, D., Mittal, A., Moorthy, A.K., and Bovik, A.C. (2012, January 4–7). Objective quality assessment of multiply distorted images. Proceedings of the 2012 Conference Record of the Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), Pacific Grove, CA, USA.10.1109/ACSSC.2012.6489321 ↗
63. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014, January 3–7). Caffe: Convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, USA.10.1145/2647868.2654889 ↗
64. Massive online crowdsourced study of subjective and objective picture quality10.1109/TIP.2015.2500021 ↗
65. Image database TID2013: Peculiarities, results and perspectives10.1016/j.image.2014.10.009 ↗
66. No-reference image quality assessment using Prewitt magnitude based on convolutional neural networks10.1007/s11760-015-0784-2 ↗
67. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
68. No reference image quality assessment based on multi-expert convolutional neural networks10.1109/ACCESS.2018.2802498 ↗
69. No-reference image quality assessment based on deep convolutional neural networks
70. Varga, D. (2020). Multi-Pooled Inception Features for No-Reference Image Quality Assessment. Appl. Sci., 10.10.3390/app10062186 ↗
71. Blind Image Quality Assessment With Active Inference10.1109/TIP.2021.3064195 ↗
72. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, Curran Associates, Inc.
73. Beghdadi, A., Qureshi, M.A., Sdiri, B., Deriche, M., and Alaya-Cheikh, F. (2018, January 19–20). Ceed-A Database for Image Contrast Enhancement Evaluation. Proceedings of the 2018 Colour and Visual Computing Symposium (CVCS), Gjovik, Norway.10.1109/CVCS.2018.8496603 ↗
74. Amirshahi, S., Pedersen, M., and Beghdadi, A. (2018). Reviving Traditional Image Quality Metrics Using CNNs. Color and Imaging Conference, Society for Imaging Science and Technology.10.2352/ISSN.2169-2629.201S.26.241 ↗
75. Saccadic model of eye movements for free-viewing condition10.1016/j.visres.2014.12.026 ↗
76. Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment10.1109/TIP.2017.2760518 ↗
77. Vigier, T., Da Silva, M.P., and Le Callet, P. (2016, January 25–28). Impact of visual angle on attention deployment and robustness of visual saliency models in videos: From SD to UHD. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.10.1109/ICIP.2016.7532445 ↗
78. Harel, J., Koch, C., and Perona, P. (2007). Graph-based visual saliency. Advances in Neural Information Processing Systems, MIT Press.10.7551/mitpress/7503.003.0073 ↗
79. Borji, A., Tavakoli, H.R., Sihite, D.N., and Itti, L. (2013, January 1–8). Analysis of scores, datasets, and models in visual saliency prediction. Proceedings of the IEEE International Conference on Computer Vision, 2013, Sydney, NSW, Australia.10.1109/ICCV.2013.118 ↗
80. Chetouani, A. (2018, January 3–7). A Blind Image Quality Metric using a Selection of Relevant Patches based on Convolutional Neural Network. Proceedings of the European Signal Processing Conference, Rome, Italy.10.23919/EUSIPCO.2018.8553127 ↗
81. He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep residual learning for image recognition. arXiv.10.1109/CVPR.2016.90 ↗
82. Classification of engraved pottery sherds mixing deep-learning features by compact bilinear pooling10.1016/j.patrec.2019.12.009 ↗
83. Jiang, R., Li, C.T., Crookes, D., Meng, W., and Rosenberger, C. (2020). Anti-Spoofing in Face Recognition: Deep Learning and Image Quality Assessment-Based Approaches. Deep Biometrics. Unsupervised and Semi-Supervised Learning, Springer.10.1007/978-3-030-32583-1 ↗
84. No-reference mesh visual quality assessment via ensemble of convolutional neural networks and compact multi-linear pooling10.1016/j.patcog.2019.107174 ↗
85. On the use of a scanpath predictor and convolutional neural network for blind image quality assessment10.1016/j.image.2020.115963 ↗
86. Chetouani, A. (2020, January 6–10). Image Quality Assessment Without Reference By Mixing Deep Learning-Based Features. Proceedings of the IEEE International Conference on Multimedia and Expo, ICME 2020, London, UK.10.1109/ICME46284.2020.9102903 ↗
87. An oblique effect in human primary visual cortex10.1038/75702 ↗
88. Transition of Visual Attention Assessment in Stereoscopic Images with Evaluation of Subjective Visual Quality and Discomfort10.1109/TMM.2015.2493367 ↗
89. Ponomarenko, N., Ieremeiev, O., Lukin, V., Egiazarian, K., and Carli, M. (2011, January 23–25). Modified image visual quality metrics for contrast change and mean shift accounting. Proceedings of the 2011 11th International Conference The Experience of Designing and Application of CAD Systems in Microelectronics (CADSM), Polyana, Ukraine.
90. Ponomarenko, M., Egiazarian, K., Lukin, V., and Abramova, V. (2018, January 2–5). Structural similarity index with predictability of image blocks. Proceedings of the 2018 IEEE 17th International Conference on Mathematical Methods in Electromagnetic Theory (MMET), Kyiv, UKraine.10.1109/MMET.2018.8460285 ↗
91. Reenu, M., David, D., Raj, S.A., and Nair, M.S. (2013, January 15–17). Wavelet based sharp features (WASH): An image quality assessment metric based on HVS. Proceedings of the 2013 2nd International Conference on Advanced Computing, Networking and Security, Wavelet Based Sharp Features (WASH): An Image Quality Assessment Metric Based on HVS, Mangalore, India.10.1109/ADCONS.2013.25 ↗
92. Image information and visual quality10.1109/TIP.2005.859378 ↗
93. An information fidelity criterion for image quality assessment using natural scene statistics10.1109/TIP.2005.859389 ↗
94. A universal image quality index10.1109/97.995823 ↗
95. Mitsa, T., and Varkur, K.L. (1993, January 27–30). Evaluation of contrast sensitivity functions for the formulation of quality measures incorporated in halftoning algorithms. Proceedings of the 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing, Minneapolis, MN, USA.10.1109/ICASSP.1993.319807 ↗
96. Image quality assessment based on a degradation model10.1109/83.841940 ↗
97. Wang, Z., Simoncelli, E.P., and Bovik, A.C. (2003, January 9–12). Multiscale structural similarity for image quality assessment. Proceedings of the The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, Pacific Grove, CA, USA.
98. FSIM: A Feature Similarity Index for Image Quality Assessment10.1109/TIP.2011.2109730 ↗
99. Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index10.1109/TIP.2013.2293423 ↗
100. No-reference image quality assessment in the spatial domain10.1109/TIP.2012.2214050 ↗