An Adaptive Image Quality Assessment Algorithm ↔ A Processing Algorithm with some User-Defined Parameters

EARLIER¶0-3

Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) DOI 10.7236/JASC2012.1.1.2 An Adaptive Image Quality Assessment Algorithm Ravi Sankar* and Goran Ivkovic iCONS Research Group, Dept. of Electrical Engineering, University of South Florida, USA Abstract An improved algorithm for image quality assessment is presented. First a simple model of human visual system, consisting of a nonlinear function and a 2-D filter, processes the input images. This filter has one user-defined parameter, whose value depends on the reference image. This way the algorithm can adapt to different scenarios. In the next step the average value of locally computed correlation coefficients between the two processed images is found. This criterion is closely related to the way in which human observer assesses image quality. Finally, image quality measure is computed as the average value of locally computed correlation coefficients, adjusted by the average correlation coefficient between the reference and error images. By this approach the proposed measure differentiates between the random and signal dependant distortions, which have different effects on human observer. Performance of the proposed quality measure is illustrated by examples involving images with different types of degradation. Key words: Brightness perception, Contrast sensitivity function, Human visual system, Image quality measure 1. INTRODUCTION Image and video quality assessment methods can be divided into: subjective and objective. The only meaningful subjective (qualitative) measure is mean opinion score (MOS), where quality of an image is rated on some scale by several individuals according to the level of distortion and the mean value of their scores is used as image quality measure. Since human observer is the ultimate receiver of the information contained in an image, this is the best way for image quality assessment. In the case of video, several subjective test methods such as Single Stimulus Continuous Quality Evaluation (SSCQE) and Quality-Oriented Adaptation Scheme (QOAS) have been described in [1-3] to obtain MOS. Objective (quantitative) measures use intensities from the distorted input image and the reference image to compute a value that indicates image quality. The goal is to find a good measure which produces results which are in good agreement with subjective evaluation. Most widely used objective measures are mean squared error (MSE) and MSE-based measures: peak signal to noise ratio (PSNR) and signal to noise ratio (SNR). These simple measures work well when images with the same type of distortion or degradation are compared. In this case distorted image with smaller MSE will be perceived closer to the original image than the one with greater MSE. However, when images with different types of distortion (additive white Gaussian noise, blurring. contrast stretching, JPEG2000 coding/compression) are compared MSE does not produce results that correlate well with subjective quality assessment. Images with different types of distortions with the same MSE values can have very different subjective visual qualities. One role of these measures is to assess performance of algorithms for image enhancement and coding. Another role is to optimize image processing algorithms. For example, in image coding the goal is to represent images with as few bits as possible with the minimal loss of image quality. Quality measure is used as a criterion in a sense that the bits that contribute more significantly to the used quality measure are coded first. The objective video quality measures have been described recently in [4]. Examples of digital video distortions including tiling, smearing, jerkiness, edge busyness, and object retention that are produced by video processing systems are fundamentally different kinds of impairments than those produced by image processing systems. So both subjective and objective quality assessments are quite different for images and videos. In this paper we focus our attention only on image quality monitoring and assessment.

EARLIER¶44-46

12 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) Table 5. “Couple” Images with Different Distortions Type of Distortion MSE Quality Measure (Q) Contrast stretching 81 0.9893 Additive white Gaussian noise 81 0.8424 Blurring 81 0.7750 JPEG2000 coding 82 0.5419 In these experiments, the proposed image quality measure produces results which are in good agreement with subjective visual quality of corresponding images. In contrast MSE values are the same for images with very different visual quality. 5. CONCLUSION In this paper, a simple image quality assessment algorithm that adapts to various image scenarios by a user-controlled parameter has been presented. First, three main reasons for disagreement between MSE values and subjective visual quality of images have been identified. Then a new quality measure has been introduced. The proposed measure takes into account two HVS properties: nonlinear relationship between intensity and perceived brightness and the presence of spatial filtering in HVS. This measure is based on average value of locally computed correlation coefficients, which is more closely related to the way in which human observer determines quality of an image than a traditional objective quality measure such as MSE. Finally, this value is modified by the average value of locally computed correlation coefficients between the original and error images. This way the proposed measure differentiates between random and signal-dependant distortions, which have different effects on human observer. The proposed image quality measure performs reasonably well. The experimental performance results presented here demonstrate that this measure ranks images according to their visual quality in cases when MSE-based measures fail to do that. However, subjective evaluation is still the best way for image quality assessment. HVS is more sophisticated than any mathematically defined image quality measure that we try hard to mimic.

EARLIER¶20-24

(3) Where, L is the total number of data blocks and the correlation coefficient between two vectors X = [x1 x2 ... xN] and Y = [y1 y2 ... yN] is defined as Y X XY XY σ σ σ ρ / = where XY σ is the covariance and y x σ σ , are the standard Nonlinear Function 2-D Filter 10 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) deviations of X and Y. By computing the correlation coefficient locally we force it to concentrate more on the difference in details, which is essential in image quality assessment. This is also closely related to the way in which human observer assesses image quality by scanning an image piece by piece and determining the difference between corresponding blocks of pixels. Average correlation coefficient given by equation (3) is a good indicator of similarity between the reference image and the distorted image, but it has one downside: it does not differentiate between random and signal dependent distortions. Therefore, it will overestimate the image degradation in the case of random noise and underestimate it in the case of signal dependent noise. This problem can be eliminated by computing the average correlation coefficient between the reference image x(m,n) and the error image e(m,n), which is given by ) , ( } { ) , ( ) , ( _ n m y sign n m x n m e avg xy ρ − = (4) Then local correlation coefficients are computed block by block between images x(m,n) and e(m,n). If the local correlation coefficient for the ith block of data is denoted by ) (i xe ρ , then the average correlation coefficient between images x(m,n) and e(m,n) is: ∑ = L i xe avg xe i L ) ( 1 _ ρ ρ (5) This coefficient will have very small positive or negative value close to zero, when there is little or no correlation between the images x(m,n) and e(m,n) and this will indicate random noise. On the other hand if there is a significant correlation between x(m,n) and e(m,n) the average

EARLIER¶5-9

Manuscript received May 10, 2012; *Corresponding Author: sankar@usf.edu Tel: +1-813-974-4769, Fax: +1-813-974-5250, USF. *Dept. of Electrical Engineering, Univ. of South Florida, USA. JASC 12-1-2 An Adaptive Image Quality Assessment Algorithm 7 intensities and perceived brightness, frequency response of HVS (contrast sensitivity function) and texture masking [5-7]. The algorithm described in [5] can identify the pixels with visible distortion, but it cannot quantify the extent of easily visible distortion, because every spatial location with clearly visible distortion will have detection probability of one in the output map. Also, this algorithm does not produce one number that indicates image quality. In [6], the properties of HVS are used to design an algorithm for image coding. In this approach input image is first decomposed into 16 sub-bands using generalized quadrature mirror filters. Image quality measure in this case is defined as MSE of normalized sub-band coefficients of the original image and distorted image, where each sub-band coefficient is normalized by the perceptual threshold for that sub-band at that spatial location. Another approach presented in [7], assumes that input image is degraded by two sources of degradation: linear distortion (for example as a result of filtering) and additive noise. In the first step linear distortion and additive noise are separated and then two quality measures are computed: distortion and noise quality measures. Distortion measure is computed using discrete Fourier transform (DFT) coefficients of the original image and original image distorted by linear distortion only. Noise quality measure is computed using the original image degraded by linear distortion and noise and the original image degraded by linear distortion only. This approach requires as input not only two images but also the algorithm that causes linear distortion and produces two numbers, which assess image quality. An overview of image quality measures is presented in [8]. Performance of several proposed quality measures is tested in [9]. Distorted images are created using four different image coding algorithms, which produce different types of degradation. Then tested quality measures are computed for each distorted image and these values are compared with subjective assessment. The tested measures included MSE and the measure proposed in [6]. The quality measure from [6] showed best agreement with subjective evaluation compared with other tested measures. However, none of the tested measures was able to predict subjective image quality consistently. Problems with the types of image quality measures similar to those described in [5-7] are that there are no standards for modeling the same properties of HVS. For example, models for brightness perception in [5] and [6] are totally different. Also, there are various contrast sensitivity functions estimated by different authors. Another problem is computation of perceptual threshold, which different authors compute in a totally different way. This sometimes requires computation of local contrast (as in [5] and [7]), which is difficult to define for complex images. A different image quality measure called the universal quality index (UQI) was proposed in [10] and [11]. This measure is computed locally to model any distortion and it is defined as a product of three components: correlation, mean and variance of distortion. Most important of these components is the correlation coefficient, which measures the degree of linear relationship between the corresponding blocks of pixels. Quality measure for the whole image is the average value of locally computed values. Most other approaches try to modify MSE by modeling properties of HVS. This approach defines different criterion which is better related to the way in which human observer assesses image quality. No HVS model is used here. The UQI produced unstable results and to avoid this problem, the measure has been generalized to the structural similarity metric (SSIM) that captures the loss of image structure for quality assessment [12]. As in the case of UQI, the overall quality (Mean SSIM) is obtained by computing the average of the locally computed values in a sliding window. A visual information fidelity criterion that is similar to HVS-based measures that relates visual quality to relative image information has been described in [13]. This is a very complex measure requiring a source model, a distortion model, and a HVS model. In [14], a completely new measure based on singular value decomposition that expressed image quality graphically or numerically was presented. The approach presented here will use the basic idea from [10-12], but it will not use image quality measure as it is defined there (only correlation coefficient will be used). In the first step, the reference and distorted images are processed by a simple model of HVS. It consists of a nonlinear function modeling the brightness perception and a 2-D filter modeling the frequency response of HVS (contrast sensitivity function). The 2-D filter used in this HVS model is not fixed. It contains one user-defined parameter which can be changed depending on the content of the reference image. After this, the correlation coefficient is computed on a block-by-block basis for the processed input images. Finally, the quality measure is computed as the average correlation coefficient between the reference image and distorted image, adjusted by the average correlation coefficient between the reference image and error image. This is a simpler model to those described in [10-14]. The rest of this paper is organized as follows. Problems with MSE-based image quality measures are discussed in Section 2. HVS model used to process input images is described in Section 3. Proposed image quality measure is defined next in Section 4. Performances of the proposed measure are illustrated by examples involving images with different types of distortions in Section 5, followed by the conclusion. 2. WHY MSE IS NOT A GOOD MEASURE? MSE and MSE-based measures such as PSNR and SNR are widely used to assess performances of image enhancement and coding algorithms. MSE-based measures work well when images with the same type of degradation are compared. In a sequence of images obtained by distorting an image by various amounts of the same type of degradation, the image with the smallest MSE will be perceived by a human observer to be the closest to the original image. However, when images with different types of degradation are compared, the image with the smallest MSE will not always be perceived to be the closest to the original image. It is possible to create images with the same 8 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) MSE value but with totally different visual quality. There are three main reasons for disagreement between MSE values and subjective evaluation. First reason for this is presence of spatial filtering in HVS. This can be demonstrated by adding a white Gaussian filtered (2-D FIR filter) noise to the original Lena image for various values of filter cut-off frequencies, fmin and fmax. Each time the filtered noise is scaled so that its standard deviation is 10. The resulting images shown in Figures 2-4 all have MSE values of 100, however, their visual qualities are quite different. Noise at very low and very high frequencies are less noticeable than noise at mid frequencies. This indicates band pass characteristic nature of HVS. One way to solve this problem is by using weighted MSE, where reference and distorted images are filtered by contrast sensitivity function (CSF) and then MSE is computed using filtered images [15]. Second reason for disagreement between MSE values and subjective assessment is in the criterion itself, i.e., MSE treats image intensities as a set of uncorrelated numbers [10-12]. One pixel is never seen in isolation from the surrounding pixels and they always create some structure in an image. This information about the structure created by a group of pixels is more important for a human observer than intensity at some spatial location. However, this is completely ignored by MSE and its computation is not quite related to the way human observers perceive image quality. The images distorted by a wide variety of corruptions such as impulsive salt-pepper noise, additive Gaussian noise, multiplicative speckle noise, contrast stretching, blurring, and JPEG compression yield the same MSE value relative to the original image but very different visual quality. This is illustrated with examples using Lena image in [10-11]. This problem cannot be solved by some modification of MSE, such as weighted MSE and it is necessary to define a different image quality measure. In this paper, we propose a new quality measure, Q and it is described later in Section 4. Third reason for low correlation between MSE values and subjective visual quality is due to the fact that MSE does not differentiate between random and signal dependent distortions. Random noise such as additive white Gaussian noise is much less objectionable for a human viewer than signal dependant distortion such as blurring. The results of these experiments are presented in Tables I-III in Section 5.

EARLIER¶0-5

Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) DOI 10.7236/JASC2012.1.1.2 An Adaptive Image Quality Assessment Algorithm Ravi Sankar* and Goran Ivkovic iCONS Research Group, Dept. of Electrical Engineering, University of South Florida, USA Abstract An improved algorithm for image quality assessment is presented. First a simple model of human visual system, consisting of a nonlinear function and a 2-D filter, processes the input images. This filter has one user-defined parameter, whose value depends on the reference image. This way the algorithm can adapt to different scenarios. In the next step the average value of locally computed correlation coefficients between the two processed images is found. This criterion is closely related to the way in which human observer assesses image quality. Finally, image quality measure is computed as the average value of locally computed correlation coefficients, adjusted by the average correlation coefficient between the reference and error images. By this approach the proposed measure differentiates between the random and signal dependant distortions, which have different effects on human observer. Performance of the proposed quality measure is illustrated by examples involving images with different types of degradation. Key words: Brightness perception, Contrast sensitivity function, Human visual system, Image quality measure 1. INTRODUCTION Image and video quality assessment methods can be divided into: subjective and objective. The only meaningful subjective (qualitative) measure is mean opinion score (MOS), where quality of an image is rated on some scale by several individuals according to the level of distortion and the mean value of their scores is used as image quality measure. Since human observer is the ultimate receiver of the information contained in an image, this is the best way for image quality assessment. In the case of video, several subjective test methods such as Single Stimulus Continuous Quality Evaluation (SSCQE) and Quality-Oriented Adaptation Scheme (QOAS) have been described in [1-3] to obtain MOS. Objective (quantitative) measures use intensities from the distorted input image and the reference image to compute a value that indicates image quality. The goal is to find a good measure which produces results which are in good agreement with subjective evaluation. Most widely used objective measures are mean squared error (MSE) and MSE-based measures: peak signal to noise ratio (PSNR) and signal to noise ratio (SNR). These simple measures work well when images with the same type of distortion or degradation are compared. In this case distorted image with smaller MSE will be perceived closer to the original image than the one with greater MSE. However, when images with different types of distortion (additive white Gaussian noise, blurring. contrast stretching, JPEG2000 coding/compression) are compared MSE does not produce results that correlate well with subjective quality assessment. Images with different types of distortions with the same MSE values can have very different subjective visual qualities. One role of these measures is to assess performance of algorithms for image enhancement and coding. Another role is to optimize image processing algorithms. For example, in image coding the goal is to represent images with as few bits as possible with the minimal loss of image quality. Quality measure is used as a criterion in a sense that the bits that contribute more significantly to the used quality measure are coded first. The objective video quality measures have been described recently in [4]. Examples of digital video distortions including tiling, smearing, jerkiness, edge busyness, and object retention that are produced by video processing systems are fundamentally different kinds of impairments than those produced by image processing systems. So both subjective and objective quality assessments are quite different for images and videos. In this paper we focus our attention only on image quality monitoring and assessment. In order to find a criterion, which agrees with subjective assessment, several other algorithms have been developed. These algorithms try to assess image quality by taking into account the properties of human visual system (HVS). Most of these algorithms are trying to model the following three properties of HVS: nonlinear relationship between image Manuscript received May 10, 2012; *Corresponding Author: sankar@usf.edu Tel: +1-813-974-4769, Fax: +1-813-974-5250, USF. *Dept. of Electrical Engineering, Univ. of South Florida, USA.

EARLIER¶6-10

JASC 12-1-2 An Adaptive Image Quality Assessment Algorithm 7 intensities and perceived brightness, frequency response of HVS (contrast sensitivity function) and texture masking [5-7]. The algorithm described in [5] can identify the pixels with visible distortion, but it cannot quantify the extent of easily visible distortion, because every spatial location with clearly visible distortion will have detection probability of one in the output map. Also, this algorithm does not produce one number that indicates image quality. In [6], the properties of HVS are used to design an algorithm for image coding. In this approach input image is first decomposed into 16 sub-bands using generalized quadrature mirror filters. Image quality measure in this case is defined as MSE of normalized sub-band coefficients of the original image and distorted image, where each sub-band coefficient is normalized by the perceptual threshold for that sub-band at that spatial location. Another approach presented in [7], assumes that input image is degraded by two sources of degradation: linear distortion (for example as a result of filtering) and additive noise. In the first step linear distortion and additive noise are separated and then two quality measures are computed: distortion and noise quality measures. Distortion measure is computed using discrete Fourier transform (DFT) coefficients of the original image and original image distorted by linear distortion only. Noise quality measure is computed using the original image degraded by linear distortion and noise and the original image degraded by linear distortion only. This approach requires as input not only two images but also the algorithm that causes linear distortion and produces two numbers, which assess image quality. An overview of image quality measures is presented in [8]. Performance of several proposed quality measures is tested in [9]. Distorted images are created using four different image coding algorithms, which produce different types of degradation. Then tested quality measures are computed for each distorted image and these values are compared with subjective assessment. The tested measures included MSE and the measure proposed in [6]. The quality measure from [6] showed best agreement with subjective evaluation compared with other tested measures. However, none of the tested measures was able to predict subjective image quality consistently. Problems with the types of image quality measures similar to those described in [5-7] are that there are no standards for modeling the same properties of HVS. For example, models for brightness perception in [5] and [6] are totally different. Also, there are various contrast sensitivity functions estimated by different authors. Another problem is computation of perceptual threshold, which different authors compute in a totally different way. This sometimes requires computation of local contrast (as in [5] and [7]), which is difficult to define for complex images. A different image quality measure called the universal quality index (UQI) was proposed in [10] and [11]. This measure is computed locally to model any distortion and it is defined as a product of three components: correlation, mean and variance of distortion. Most important of these components is the correlation coefficient, which measures the degree of linear relationship between the corresponding blocks of pixels. Quality measure for the whole image is the average value of locally computed values. Most other approaches try to modify MSE by modeling properties of HVS. This approach defines different criterion which is better related to the way in which human observer assesses image quality. No HVS model is used here. The UQI produced unstable results and to avoid this problem, the measure has been generalized to the structural similarity metric (SSIM) that captures the loss of image structure for quality assessment [12]. As in the case of UQI, the overall quality (Mean SSIM) is obtained by computing the average of the locally computed values in a sliding window. A visual information fidelity criterion that is similar to HVS-based measures that relates visual quality to relative image information has been described in [13]. This is a very complex measure requiring a source model, a distortion model, and a HVS model. In [14], a completely new measure based on singular value decomposition that expressed image quality graphically or numerically was presented. The approach presented here will use the basic idea from [10-12], but it will not use image quality measure as it is defined there (only correlation coefficient will be used). In the first step, the reference and distorted images are processed by a simple model of HVS. It consists of a nonlinear function modeling the brightness perception and a 2-D filter modeling the frequency response of HVS (contrast sensitivity function). The 2-D filter used in this HVS model is not fixed. It contains one user-defined parameter which can be changed depending on the content of the reference image. After this, the correlation coefficient is computed on a block-by-block basis for the processed input images. Finally, the quality measure is computed as the average correlation coefficient between the reference image and distorted image, adjusted by the average correlation coefficient between the reference image and error image. This is a simpler model to those described in [10-14]. The rest of this paper is organized as follows. Problems with MSE-based image quality measures are discussed in Section 2. HVS model used to process input images is described in Section 3. Proposed image quality measure is defined next in Section 4. Performances of the proposed measure are illustrated by examples involving images with different types of distortions in Section 5, followed by the conclusion. 2. WHY MSE IS NOT A GOOD MEASURE? MSE and MSE-based measures such as PSNR and SNR are widely used to assess performances of image enhancement and coding algorithms. MSE-based measures work well when images with the same type of degradation are compared. In a sequence of images obtained by distorting an image by various amounts of the same type of degradation, the image with the smallest MSE will be perceived by a human observer to be the closest to the original image. However, when images with different types of degradation are compared, the image with the smallest MSE will not always be perceived to be the closest to the original image. It is possible to create images with the same 8 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) MSE value but with totally different visual quality. There are three main reasons for disagreement between MSE values and subjective evaluation. First reason for this is presence of spatial filtering in HVS. This can be demonstrated by adding a white Gaussian filtered (2-D FIR filter) noise to the original Lena image for various values of filter cut-off frequencies, fmin and fmax. Each time the filtered noise is scaled so that its standard deviation is 10. The resulting images shown in Figures 2-4 all have MSE values of 100, however, their visual qualities are quite different. Noise at very low and very high frequencies are less noticeable than noise at mid frequencies. This indicates band pass characteristic nature of HVS. One way to solve this problem is by using weighted MSE, where reference and distorted images are filtered by contrast sensitivity function (CSF) and then MSE is computed using filtered images [15]. Second reason for disagreement between MSE values and subjective assessment is in the criterion itself, i.e., MSE treats image intensities as a set of uncorrelated numbers [10-12]. One pixel is never seen in isolation from the surrounding pixels and they always create some structure in an image. This information about the structure created by a group of pixels is more important for a human observer than intensity at some spatial location. However, this is completely ignored by MSE and its computation is not quite related to the way human observers perceive image quality. The images distorted by a wide variety of corruptions such as impulsive salt-pepper noise, additive Gaussian noise, multiplicative speckle noise, contrast stretching, blurring, and JPEG compression yield the same MSE value relative to the original image but very different visual quality. This is illustrated with examples using Lena image in [10-11]. This problem cannot be solved by some modification of MSE, such as weighted MSE and it is necessary to define a different image quality measure. In this paper, we propose a new quality measure, Q and it is described later in Section 4. Third reason for low correlation between MSE values and subjective visual quality is due to the fact that MSE does not differentiate between random and signal dependent distortions. Random noise such as additive white Gaussian noise is much less objectionable for a human viewer than signal dependant distortion such as blurring. The results of these experiments are presented in Tables I-III in Section 5. 3. HVS MODEL Image quality measure developed here will use a simple model of HVS consisting of a nonlinear function, modeling brightness perception of HVS and a 2-D filter, modeling frequency response of HVS. This type of HVS model was used in [15] and [16]. However, nonlinear function and 2-D filter used here are not the same as those developed in [15]. Digital images are represented using a finite number of intensity levels. For example, gray scale images with 8 bits/pixel are represented by 256 intensity levels, where 0 corresponds to the darkest level and 255 corresponds to the brightest level. Intensities between these two extremes

EARLIER¶1-5

iCONS Research Group, Dept. of Electrical Engineering, University of South Florida, USA Abstract An improved algorithm for image quality assessment is presented. First a simple model of human visual system, consisting of a nonlinear function and a 2-D filter, processes the input images. This filter has one user-defined parameter, whose value depends on the reference image. This way the algorithm can adapt to different scenarios. In the next step the average value of locally computed correlation coefficients between the two processed images is found. This criterion is closely related to the way in which human observer assesses image quality. Finally, image quality measure is computed as the average value of locally computed correlation coefficients, adjusted by the average correlation coefficient between the reference and error images. By this approach the proposed measure differentiates between the random and signal dependant distortions, which have different effects on human observer. Performance of the proposed quality measure is illustrated by examples involving images with different types of degradation. Key words: Brightness perception, Contrast sensitivity function, Human visual system, Image quality measure 1. INTRODUCTION Image and video quality assessment methods can be divided into: subjective and objective. The only meaningful subjective (qualitative) measure is mean opinion score (MOS), where quality of an image is rated on some scale by several individuals according to the level of distortion and the mean value of their scores is used as image quality measure. Since human observer is the ultimate receiver of the information contained in an image, this is the best way for image quality assessment. In the case of video, several subjective test methods such as Single Stimulus Continuous Quality Evaluation (SSCQE) and Quality-Oriented Adaptation Scheme (QOAS) have been described in [1-3] to obtain MOS. Objective (quantitative) measures use intensities from the distorted input image and the reference image to compute a value that indicates image quality. The goal is to find a good measure which produces results which are in good agreement with subjective evaluation. Most widely used objective measures are mean squared error (MSE) and MSE-based measures: peak signal to noise ratio (PSNR) and signal to noise ratio (SNR). These simple measures work well when images with the same type of distortion or degradation are compared. In this case distorted image with smaller MSE will be perceived closer to the original image than the one with greater MSE. However, when images with different types of distortion (additive white Gaussian noise, blurring. contrast stretching, JPEG2000 coding/compression) are compared MSE does not produce results that correlate well with subjective quality assessment. Images with different types of distortions with the same MSE values can have very different subjective visual qualities. One role of these measures is to assess performance of algorithms for image enhancement and coding. Another role is to optimize image processing algorithms. For example, in image coding the goal is to represent images with as few bits as possible with the minimal loss of image quality. Quality measure is used as a criterion in a sense that the bits that contribute more significantly to the used quality measure are coded first. The objective video quality measures have been described recently in [4]. Examples of digital video distortions including tiling, smearing, jerkiness, edge busyness, and object retention that are produced by video processing systems are fundamentally different kinds of impairments than those produced by image processing systems. So both subjective and objective quality assessments are quite different for images and videos. In this paper we focus our attention only on image quality monitoring and assessment. In order to find a criterion, which agrees with subjective assessment, several other algorithms have been developed. These algorithms try to assess image quality by taking into account the properties of human visual system (HVS). Most of these algorithms are trying to model the following three properties of HVS: nonlinear relationship between image Manuscript received May 10, 2012; *Corresponding Author: sankar@usf.edu Tel: +1-813-974-4769, Fax: +1-813-974-5250, USF. *Dept. of Electrical Engineering, Univ. of South Florida, USA.

EARLIER¶2-6

Key words: Brightness perception, Contrast sensitivity function, Human visual system, Image quality measure 1. INTRODUCTION Image and video quality assessment methods can be divided into: subjective and objective. The only meaningful subjective (qualitative) measure is mean opinion score (MOS), where quality of an image is rated on some scale by several individuals according to the level of distortion and the mean value of their scores is used as image quality measure. Since human observer is the ultimate receiver of the information contained in an image, this is the best way for image quality assessment. In the case of video, several subjective test methods such as Single Stimulus Continuous Quality Evaluation (SSCQE) and Quality-Oriented Adaptation Scheme (QOAS) have been described in [1-3] to obtain MOS. Objective (quantitative) measures use intensities from the distorted input image and the reference image to compute a value that indicates image quality. The goal is to find a good measure which produces results which are in good agreement with subjective evaluation. Most widely used objective measures are mean squared error (MSE) and MSE-based measures: peak signal to noise ratio (PSNR) and signal to noise ratio (SNR). These simple measures work well when images with the same type of distortion or degradation are compared. In this case distorted image with smaller MSE will be perceived closer to the original image than the one with greater MSE. However, when images with different types of distortion (additive white Gaussian noise, blurring. contrast stretching, JPEG2000 coding/compression) are compared MSE does not produce results that correlate well with subjective quality assessment. Images with different types of distortions with the same MSE values can have very different subjective visual qualities. One role of these measures is to assess performance of algorithms for image enhancement and coding. Another role is to optimize image processing algorithms. For example, in image coding the goal is to represent images with as few bits as possible with the minimal loss of image quality. Quality measure is used as a criterion in a sense that the bits that contribute more significantly to the used quality measure are coded first. The objective video quality measures have been described recently in [4]. Examples of digital video distortions including tiling, smearing, jerkiness, edge busyness, and object retention that are produced by video processing systems are fundamentally different kinds of impairments than those produced by image processing systems. So both subjective and objective quality assessments are quite different for images and videos. In this paper we focus our attention only on image quality monitoring and assessment. In order to find a criterion, which agrees with subjective assessment, several other algorithms have been developed. These algorithms try to assess image quality by taking into account the properties of human visual system (HVS). Most of these algorithms are trying to model the following three properties of HVS: nonlinear relationship between image Manuscript received May 10, 2012; *Corresponding Author: sankar@usf.edu Tel: +1-813-974-4769, Fax: +1-813-974-5250, USF. *Dept. of Electrical Engineering, Univ. of South Florida, USA. JASC 12-1-2 An Adaptive Image Quality Assessment Algorithm 7 intensities and perceived brightness, frequency response of HVS (contrast sensitivity function) and texture masking [5-7]. The algorithm described in [5] can identify the pixels with visible distortion, but it cannot quantify the extent of easily visible distortion, because every spatial location with clearly visible distortion will have detection probability of one in the output map. Also, this algorithm does not produce one number that indicates image quality. In [6], the properties of HVS are used to design an algorithm for image coding. In this approach input image is first decomposed into 16 sub-bands using generalized quadrature mirror filters. Image quality measure in this case is defined as MSE of normalized sub-band coefficients of the original image and distorted image, where each sub-band coefficient is normalized by the perceptual threshold for that sub-band at that spatial location. Another approach presented in [7], assumes that input image is degraded by two sources of degradation: linear distortion (for example as a result of filtering) and additive noise. In the first step linear distortion and additive noise are separated and then two quality measures are computed: distortion and noise quality measures. Distortion measure is computed using discrete Fourier transform (DFT) coefficients of the original image and original image distorted by linear distortion only. Noise quality measure is computed using the original image degraded by linear distortion and noise and the original image degraded by linear distortion only. This approach requires as input not only two images but also the algorithm that causes linear distortion and produces two numbers, which assess image quality. An overview of image quality measures is presented in [8]. Performance of several proposed quality measures is tested in [9]. Distorted images are created using four different image coding algorithms, which produce different types of degradation. Then tested quality measures are computed for each distorted image and these values are compared with subjective assessment. The tested measures included MSE and the measure proposed in [6]. The quality measure from [6] showed best agreement with subjective evaluation compared with other tested measures. However, none of the tested measures was able to predict subjective image quality consistently. Problems with the types of image quality measures similar to those described in [5-7] are that there are no standards for modeling the same properties of HVS. For example, models for brightness perception in [5] and [6] are totally different. Also, there are various contrast sensitivity functions estimated by different authors. Another problem is computation of perceptual threshold, which different authors compute in a totally different way. This sometimes requires computation of local contrast (as in [5] and [7]), which is difficult to define for complex images. A different image quality measure called the universal quality index (UQI) was proposed in [10] and [11]. This measure is computed locally to model any distortion and it is defined as a product of three components: correlation, mean and variance of distortion. Most important of these components is the correlation coefficient, which measures the degree of linear relationship between the corresponding blocks of pixels. Quality measure for the whole image is the average

EARLIER¶4-8

In order to find a criterion, which agrees with subjective assessment, several other algorithms have been developed. These algorithms try to assess image quality by taking into account the properties of human visual system (HVS). Most of these algorithms are trying to model the following three properties of HVS: nonlinear relationship between image Manuscript received May 10, 2012; *Corresponding Author: sankar@usf.edu Tel: +1-813-974-4769, Fax: +1-813-974-5250, USF. *Dept. of Electrical Engineering, Univ. of South Florida, USA. JASC 12-1-2 An Adaptive Image Quality Assessment Algorithm 7 intensities and perceived brightness, frequency response of HVS (contrast sensitivity function) and texture masking [5-7]. The algorithm described in [5] can identify the pixels with visible distortion, but it cannot quantify the extent of easily visible distortion, because every spatial location with clearly visible distortion will have detection probability of one in the output map. Also, this algorithm does not produce one number that indicates image quality. In [6], the properties of HVS are used to design an algorithm for image coding. In this approach input image is first decomposed into 16 sub-bands using generalized quadrature mirror filters. Image quality measure in this case is defined as MSE of normalized sub-band coefficients of the original image and distorted image, where each sub-band coefficient is normalized by the perceptual threshold for that sub-band at that spatial location. Another approach presented in [7], assumes that input image is degraded by two sources of degradation: linear distortion (for example as a result of filtering) and additive noise. In the first step linear distortion and additive noise are separated and then two quality measures are computed: distortion and noise quality measures. Distortion measure is computed using discrete Fourier transform (DFT) coefficients of the original image and original image distorted by linear distortion only. Noise quality measure is computed using the original image degraded by linear distortion and noise and the original image degraded by linear distortion only. This approach requires as input not only two images but also the algorithm that causes linear distortion and produces two numbers, which assess image quality. An overview of image quality measures is presented in [8]. Performance of several proposed quality measures is tested in [9]. Distorted images are created using four different image coding algorithms, which produce different types of degradation. Then tested quality measures are computed for each distorted image and these values are compared with subjective assessment. The tested measures included MSE and the measure proposed in [6]. The quality measure from [6] showed best agreement with subjective evaluation compared with other tested measures. However, none of the tested measures was able to predict subjective image quality consistently. Problems with the types of image quality measures similar to those described in [5-7] are that there are no standards for modeling the same properties of HVS. For example, models for brightness perception in [5] and [6] are totally different. Also, there are various contrast sensitivity functions estimated by different authors. Another problem is computation of perceptual threshold, which different authors compute in a totally different way. This sometimes requires computation of local contrast (as in [5] and [7]), which is difficult to define for complex images. A different image quality measure called the universal quality index (UQI) was proposed in [10] and [11]. This measure is computed locally to model any distortion and it is defined as a product of three components: correlation, mean and variance of distortion. Most important of these components is the correlation coefficient, which measures the degree of linear relationship between the corresponding blocks of pixels. Quality measure for the whole image is the average value of locally computed values. Most other approaches try to modify MSE by modeling properties of HVS. This approach defines different criterion which is better related to the way in which human observer assesses image quality. No HVS model is used here. The UQI produced unstable results and to avoid this problem, the measure has been generalized to the structural similarity metric (SSIM) that captures the loss of image structure for quality assessment [12]. As in the case of UQI, the overall quality (Mean SSIM) is obtained by computing the average of the locally computed values in a sliding window. A visual information fidelity criterion that is similar to HVS-based measures that relates visual quality to relative image information has been described in [13]. This is a very complex measure requiring a source model, a distortion model, and a HVS model. In [14], a completely new measure based on singular value decomposition that expressed image quality graphically or numerically was presented. The approach presented here will use the basic idea from [10-12], but it will not use image quality measure as it is defined there (only correlation coefficient will be used). In the first step, the reference and distorted images are processed by a simple model of HVS. It consists of a nonlinear function modeling the brightness perception and a 2-D filter modeling the frequency response of HVS (contrast sensitivity function). The 2-D filter used in this HVS model is not fixed. It contains one user-defined parameter which can be changed depending on the content of the reference image. After this, the correlation coefficient is computed on a block-by-block basis for the processed input images. Finally, the quality measure is computed as the average correlation coefficient between the reference image and distorted image, adjusted by the average correlation coefficient between the reference image and error image. This is a simpler model to those described in [10-14]. The rest of this paper is organized as follows. Problems with MSE-based image quality measures are discussed in Section 2. HVS model used to process input images is described in Section 3. Proposed image quality measure is defined next in Section 4. Performances of the proposed measure are illustrated by examples involving images with different types of distortions in Section 5, followed by the conclusion. 2. WHY MSE IS NOT A GOOD MEASURE? MSE and MSE-based measures such as PSNR and SNR are widely used to assess performances of image enhancement and coding algorithms. MSE-based measures work well when images with the same type of degradation are compared. In a sequence of images obtained by distorting an image by various amounts of the same type of degradation, the image with the smallest MSE will be perceived by a human observer to be the closest to the original image. However, when images with different types of degradation are compared, the image with the smallest MSE will not always be perceived to be the closest to the original image. It is possible to create images with the same

EARLIER¶4-9

In order to find a criterion, which agrees with subjective assessment, several other algorithms have been developed. These algorithms try to assess image quality by taking into account the properties of human visual system (HVS). Most of these algorithms are trying to model the following three properties of HVS: nonlinear relationship between image Manuscript received May 10, 2012; *Corresponding Author: sankar@usf.edu Tel: +1-813-974-4769, Fax: +1-813-974-5250, USF. *Dept. of Electrical Engineering, Univ. of South Florida, USA. JASC 12-1-2 An Adaptive Image Quality Assessment Algorithm 7 intensities and perceived brightness, frequency response of HVS (contrast sensitivity function) and texture masking [5-7]. The algorithm described in [5] can identify the pixels with visible distortion, but it cannot quantify the extent of easily visible distortion, because every spatial location with clearly visible distortion will have detection probability of one in the output map. Also, this algorithm does not produce one number that indicates image quality. In [6], the properties of HVS are used to design an algorithm for image coding. In this approach input image is first decomposed into 16 sub-bands using generalized quadrature mirror filters. Image quality measure in this case is defined as MSE of normalized sub-band coefficients of the original image and distorted image, where each sub-band coefficient is normalized by the perceptual threshold for that sub-band at that spatial location. Another approach presented in [7], assumes that input image is degraded by two sources of degradation: linear distortion (for example as a result of filtering) and additive noise. In the first step linear distortion and additive noise are separated and then two quality measures are computed: distortion and noise quality measures. Distortion measure is computed using discrete Fourier transform (DFT) coefficients of the original image and original image distorted by linear distortion only. Noise quality measure is computed using the original image degraded by linear distortion and noise and the original image degraded by linear distortion only. This approach requires as input not only two images but also the algorithm that causes linear distortion and produces two numbers, which assess image quality. An overview of image quality measures is presented in [8]. Performance of several proposed quality measures is tested in [9]. Distorted images are created using four different image coding algorithms, which produce different types of degradation. Then tested quality measures are computed for each distorted image and these values are compared with subjective assessment. The tested measures included MSE and the measure proposed in [6]. The quality measure from [6] showed best agreement with subjective evaluation compared with other tested measures. However, none of the tested measures was able to predict subjective image quality consistently. Problems with the types of image quality measures similar to those described in [5-7] are that there are no standards for modeling the same properties of HVS. For example, models for brightness perception in [5] and [6] are totally different. Also, there are various contrast sensitivity functions estimated by different authors. Another problem is computation of perceptual threshold, which different authors compute in a totally different way. This sometimes requires computation of local contrast (as in [5] and [7]), which is difficult to define for complex images. A different image quality measure called the universal quality index (UQI) was proposed in [10] and [11]. This measure is computed locally to model any distortion and it is defined as a product of three components: correlation, mean and variance of distortion. Most important of these components is the correlation coefficient, which measures the degree of linear relationship between the corresponding blocks of pixels. Quality measure for the whole image is the average value of locally computed values. Most other approaches try to modify MSE by modeling properties of HVS. This approach defines different criterion which is better related to the way in which human observer assesses image quality. No HVS model is used here. The UQI produced unstable results and to avoid this problem, the measure has been generalized to the structural similarity metric (SSIM) that captures the loss of image structure for quality assessment [12]. As in the case of UQI, the overall quality (Mean SSIM) is obtained by computing the average of the locally computed values in a sliding window. A visual information fidelity criterion that is similar to HVS-based measures that relates visual quality to relative image information has been described in [13]. This is a very complex measure requiring a source model, a distortion model, and a HVS model. In [14], a completely new measure based on singular value decomposition that expressed image quality graphically or numerically was presented. The approach presented here will use the basic idea from [10-12], but it will not use image quality measure as it is defined there (only correlation coefficient will be used). In the first step, the reference and distorted images are processed by a simple model of HVS. It consists of a nonlinear function modeling the brightness perception and a 2-D filter modeling the frequency response of HVS (contrast sensitivity function). The 2-D filter used in this HVS model is not fixed. It contains one user-defined parameter which can be changed depending on the content of the reference image. After this, the correlation coefficient is computed on a block-by-block basis for the processed input images. Finally, the quality measure is computed as the average correlation coefficient between the reference image and distorted image, adjusted by the average correlation coefficient between the reference image and error image. This is a simpler model to those described in [10-14]. The rest of this paper is organized as follows. Problems with MSE-based image quality measures are discussed in Section 2. HVS model used to process input images is described in Section 3. Proposed image quality measure is defined next in Section 4. Performances of the proposed measure are illustrated by examples involving images with different types of distortions in Section 5, followed by the conclusion. 2. WHY MSE IS NOT A GOOD MEASURE? MSE and MSE-based measures such as PSNR and SNR are widely used to assess performances of image enhancement and coding algorithms. MSE-based measures work well when images with the same type of degradation are compared. In a sequence of images obtained by distorting an image by various amounts of the same type of degradation, the image with the smallest MSE will be perceived by a human observer to be the closest to the original image. However, when images with different types of degradation are compared, the image with the smallest MSE will not always be perceived to be the closest to the original image. It is possible to create images with the same 8 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) MSE value but with totally different visual quality. There are three main reasons for disagreement between MSE values and subjective evaluation. First reason for this is presence of spatial filtering in HVS. This can be demonstrated by adding a white Gaussian filtered (2-D FIR filter) noise to the original Lena image for various values of filter cut-off frequencies, fmin and fmax. Each time the filtered noise is scaled so that its standard deviation is 10. The resulting images shown in Figures 2-4 all have MSE values of 100, however, their visual qualities are quite different. Noise at very low and very high frequencies are less noticeable than noise at mid frequencies. This indicates band pass characteristic nature of HVS. One way to solve this problem is by using weighted MSE, where reference and distorted images are filtered by contrast sensitivity function (CSF) and then MSE is computed using filtered images [15]. Second reason for disagreement between MSE values and subjective assessment is in the criterion itself, i.e., MSE treats image intensities as a set of uncorrelated numbers [10-12]. One pixel is never seen in isolation from the surrounding pixels and they always create some structure in an image. This information about the structure created by a group of pixels is more important for a human observer than intensity at some spatial location. However, this is completely ignored by MSE and its computation is not quite related to the way human observers perceive image quality. The images distorted by a wide variety of corruptions such as impulsive salt-pepper noise, additive Gaussian noise, multiplicative speckle noise, contrast stretching, blurring, and JPEG compression yield the same MSE value relative to the original image but very different visual quality. This is illustrated with examples using Lena image in [10-11]. This problem cannot be solved by some modification of MSE, such as weighted MSE and it is necessary to define a different image quality measure. In this paper, we propose a new quality measure, Q and it is described later in Section 4. Third reason for low correlation between MSE values and subjective visual quality is due to the fact that MSE does not differentiate between random and signal dependent distortions. Random noise such as additive white Gaussian noise is much less objectionable for a human viewer than signal dependant distortion such as blurring. The results of these experiments are presented in Tables I-III in Section 5.

EARLIER¶0-3

Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) DOI 10.7236/JASC2012.1.1.2 An Adaptive Image Quality Assessment Algorithm Ravi Sankar* and Goran Ivkovic iCONS Research Group, Dept. of Electrical Engineering, University of South Florida, USA Abstract An improved algorithm for image quality assessment is presented. First a simple model of human visual system, consisting of a nonlinear function and a 2-D filter, processes the input images. This filter has one user-defined parameter, whose value depends on the reference image. This way the algorithm can adapt to different scenarios. In the next step the average value of locally computed correlation coefficients between the two processed images is found. This criterion is closely related to the way in which human observer assesses image quality. Finally, image quality measure is computed as the average value of locally computed correlation coefficients, adjusted by the average correlation coefficient between the reference and error images. By this approach the proposed measure differentiates between the random and signal dependant distortions, which have different effects on human observer. Performance of the proposed quality measure is illustrated by examples involving images with different types of degradation. Key words: Brightness perception, Contrast sensitivity function, Human visual system, Image quality measure 1. INTRODUCTION Image and video quality assessment methods can be divided into: subjective and objective. The only meaningful subjective (qualitative) measure is mean opinion score (MOS), where quality of an image is rated on some scale by several individuals according to the level of distortion and the mean value of their scores is used as image quality measure. Since human observer is the ultimate receiver of the information contained in an image, this is the best way for image quality assessment. In the case of video, several subjective test methods such as Single Stimulus Continuous Quality Evaluation (SSCQE) and Quality-Oriented Adaptation Scheme (QOAS) have been described in [1-3] to obtain MOS. Objective (quantitative) measures use intensities from the distorted input image and the reference image to compute a value that indicates image quality. The goal is to find a good measure which produces results which are in good agreement with subjective evaluation. Most widely used objective measures are mean squared error (MSE) and MSE-based measures: peak signal to noise ratio (PSNR) and signal to noise ratio (SNR). These simple measures work well when images with the same type of distortion or degradation are compared. In this case distorted image with smaller MSE will be perceived closer to the original image than the one with greater MSE. However, when images with different types of distortion (additive white Gaussian noise, blurring. contrast stretching, JPEG2000 coding/compression) are compared MSE does not produce results that correlate well with subjective quality assessment. Images with different types of distortions with the same MSE values can have very different subjective visual qualities. One role of these measures is to assess performance of algorithms for image enhancement and coding. Another role is to optimize image processing algorithms. For example, in image coding the goal is to represent images with as few bits as possible with the minimal loss of image quality. Quality measure is used as a criterion in a sense that the bits that contribute more significantly to the used quality measure are coded first. The objective video quality measures have been described recently in [4]. Examples of digital video distortions including tiling, smearing, jerkiness, edge busyness, and object retention that are produced by video processing systems are fundamentally different kinds of impairments than those produced by image processing systems. So both subjective and objective quality assessments are quite different for images and videos. In this paper we focus our attention only on image quality monitoring and assessment.

EARLIER¶20-24

(3) Where, L is the total number of data blocks and the correlation coefficient between two vectors X = [x1 x2 ... xN] and Y = [y1 y2 ... yN] is defined as Y X XY XY σ σ σ ρ / = where XY σ is the covariance and y x σ σ , are the standard Nonlinear Function 2-D Filter 10 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) deviations of X and Y. By computing the correlation coefficient locally we force it to concentrate more on the difference in details, which is essential in image quality assessment. This is also closely related to the way in which human observer assesses image quality by scanning an image piece by piece and determining the difference between corresponding blocks of pixels. Average correlation coefficient given by equation (3) is a good indicator of similarity between the reference image and the distorted image, but it has one downside: it does not differentiate between random and signal dependent distortions. Therefore, it will overestimate the image degradation in the case of random noise and underestimate it in the case of signal dependent noise. This problem can be eliminated by computing the average correlation coefficient between the reference image x(m,n) and the error image e(m,n), which is given by ) , ( } { ) , ( ) , ( _ n m y sign n m x n m e avg xy ρ − = (4) Then local correlation coefficients are computed block by block between images x(m,n) and e(m,n). If the local correlation coefficient for the ith block of data is denoted by ) (i xe ρ , then the average correlation coefficient between images x(m,n) and e(m,n) is: ∑ = L i xe avg xe i L ) ( 1 _ ρ ρ (5) This coefficient will have very small positive or negative value close to zero, when there is little or no correlation between the images x(m,n) and e(m,n) and this will indicate random noise. On the other hand if there is a significant correlation between x(m,n) and e(m,n) the average

EARLIER¶5-9

Manuscript received May 10, 2012; *Corresponding Author: sankar@usf.edu Tel: +1-813-974-4769, Fax: +1-813-974-5250, USF. *Dept. of Electrical Engineering, Univ. of South Florida, USA. JASC 12-1-2 An Adaptive Image Quality Assessment Algorithm 7 intensities and perceived brightness, frequency response of HVS (contrast sensitivity function) and texture masking [5-7]. The algorithm described in [5] can identify the pixels with visible distortion, but it cannot quantify the extent of easily visible distortion, because every spatial location with clearly visible distortion will have detection probability of one in the output map. Also, this algorithm does not produce one number that indicates image quality. In [6], the properties of HVS are used to design an algorithm for image coding. In this approach input image is first decomposed into 16 sub-bands using generalized quadrature mirror filters. Image quality measure in this case is defined as MSE of normalized sub-band coefficients of the original image and distorted image, where each sub-band coefficient is normalized by the perceptual threshold for that sub-band at that spatial location. Another approach presented in [7], assumes that input image is degraded by two sources of degradation: linear distortion (for example as a result of filtering) and additive noise. In the first step linear distortion and additive noise are separated and then two quality measures are computed: distortion and noise quality measures. Distortion measure is computed using discrete Fourier transform (DFT) coefficients of the original image and original image distorted by linear distortion only. Noise quality measure is computed using the original image degraded by linear distortion and noise and the original image degraded by linear distortion only. This approach requires as input not only two images but also the algorithm that causes linear distortion and produces two numbers, which assess image quality. An overview of image quality measures is presented in [8]. Performance of several proposed quality measures is tested in [9]. Distorted images are created using four different image coding algorithms, which produce different types of degradation. Then tested quality measures are computed for each distorted image and these values are compared with subjective assessment. The tested measures included MSE and the measure proposed in [6]. The quality measure from [6] showed best agreement with subjective evaluation compared with other tested measures. However, none of the tested measures was able to predict subjective image quality consistently. Problems with the types of image quality measures similar to those described in [5-7] are that there are no standards for modeling the same properties of HVS. For example, models for brightness perception in [5] and [6] are totally different. Also, there are various contrast sensitivity functions estimated by different authors. Another problem is computation of perceptual threshold, which different authors compute in a totally different way. This sometimes requires computation of local contrast (as in [5] and [7]), which is difficult to define for complex images. A different image quality measure called the universal quality index (UQI) was proposed in [10] and [11]. This measure is computed locally to model any distortion and it is defined as a product of three components: correlation, mean and variance of distortion. Most important of these components is the correlation coefficient, which measures the degree of linear relationship between the corresponding blocks of pixels. Quality measure for the whole image is the average value of locally computed values. Most other approaches try to modify MSE by modeling properties of HVS. This approach defines different criterion which is better related to the way in which human observer assesses image quality. No HVS model is used here. The UQI produced unstable results and to avoid this problem, the measure has been generalized to the structural similarity metric (SSIM) that captures the loss of image structure for quality assessment [12]. As in the case of UQI, the overall quality (Mean SSIM) is obtained by computing the average of the locally computed values in a sliding window. A visual information fidelity criterion that is similar to HVS-based measures that relates visual quality to relative image information has been described in [13]. This is a very complex measure requiring a source model, a distortion model, and a HVS model. In [14], a completely new measure based on singular value decomposition that expressed image quality graphically or numerically was presented. The approach presented here will use the basic idea from [10-12], but it will not use image quality measure as it is defined there (only correlation coefficient will be used). In the first step, the reference and distorted images are processed by a simple model of HVS. It consists of a nonlinear function modeling the brightness perception and a 2-D filter modeling the frequency response of HVS (contrast sensitivity function). The 2-D filter used in this HVS model is not fixed. It contains one user-defined parameter which can be changed depending on the content of the reference image. After this, the correlation coefficient is computed on a block-by-block basis for the processed input images. Finally, the quality measure is computed as the average correlation coefficient between the reference image and distorted image, adjusted by the average correlation coefficient between the reference image and error image. This is a simpler model to those described in [10-14]. The rest of this paper is organized as follows. Problems with MSE-based image quality measures are discussed in Section 2. HVS model used to process input images is described in Section 3. Proposed image quality measure is defined next in Section 4. Performances of the proposed measure are illustrated by examples involving images with different types of distortions in Section 5, followed by the conclusion. 2. WHY MSE IS NOT A GOOD MEASURE? MSE and MSE-based measures such as PSNR and SNR are widely used to assess performances of image enhancement and coding algorithms. MSE-based measures work well when images with the same type of degradation are compared. In a sequence of images obtained by distorting an image by various amounts of the same type of degradation, the image with the smallest MSE will be perceived by a human observer to be the closest to the original image. However, when images with different types of degradation are compared, the image with the smallest MSE will not always be perceived to be the closest to the original image. It is possible to create images with the same 8 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) MSE value but with totally different visual quality. There are three main reasons for disagreement between MSE values and subjective evaluation. First reason for this is presence of spatial filtering in HVS. This can be demonstrated by adding a white Gaussian filtered (2-D FIR filter) noise to the original Lena image for various values of filter cut-off frequencies, fmin and fmax. Each time the filtered noise is scaled so that its standard deviation is 10. The resulting images shown in Figures 2-4 all have MSE values of 100, however, their visual qualities are quite different. Noise at very low and very high frequencies are less noticeable than noise at mid frequencies. This indicates band pass characteristic nature of HVS. One way to solve this problem is by using weighted MSE, where reference and distorted images are filtered by contrast sensitivity function (CSF) and then MSE is computed using filtered images [15]. Second reason for disagreement between MSE values and subjective assessment is in the criterion itself, i.e., MSE treats image intensities as a set of uncorrelated numbers [10-12]. One pixel is never seen in isolation from the surrounding pixels and they always create some structure in an image. This information about the structure created by a group of pixels is more important for a human observer than intensity at some spatial location. However, this is completely ignored by MSE and its computation is not quite related to the way human observers perceive image quality. The images distorted by a wide variety of corruptions such as impulsive salt-pepper noise, additive Gaussian noise, multiplicative speckle noise, contrast stretching, blurring, and JPEG compression yield the same MSE value relative to the original image but very different visual quality. This is illustrated with examples using Lena image in [10-11]. This problem cannot be solved by some modification of MSE, such as weighted MSE and it is necessary to define a different image quality measure. In this paper, we propose a new quality measure, Q and it is described later in Section 4. Third reason for low correlation between MSE values and subjective visual quality is due to the fact that MSE does not differentiate between random and signal dependent distortions. Random noise such as additive white Gaussian noise is much less objectionable for a human viewer than signal dependant distortion such as blurring. The results of these experiments are presented in Tables I-III in Section 5.

EARLIER¶5-9

Manuscript received May 10, 2012; *Corresponding Author: sankar@usf.edu Tel: +1-813-974-4769, Fax: +1-813-974-5250, USF. *Dept. of Electrical Engineering, Univ. of South Florida, USA. JASC 12-1-2 An Adaptive Image Quality Assessment Algorithm 7 intensities and perceived brightness, frequency response of HVS (contrast sensitivity function) and texture masking [5-7]. The algorithm described in [5] can identify the pixels with visible distortion, but it cannot quantify the extent of easily visible distortion, because every spatial location with clearly visible distortion will have detection probability of one in the output map. Also, this algorithm does not produce one number that indicates image quality. In [6], the properties of HVS are used to design an algorithm for image coding. In this approach input image is first decomposed into 16 sub-bands using generalized quadrature mirror filters. Image quality measure in this case is defined as MSE of normalized sub-band coefficients of the original image and distorted image, where each sub-band coefficient is normalized by the perceptual threshold for that sub-band at that spatial location. Another approach presented in [7], assumes that input image is degraded by two sources of degradation: linear distortion (for example as a result of filtering) and additive noise. In the first step linear distortion and additive noise are separated and then two quality measures are computed: distortion and noise quality measures. Distortion measure is computed using discrete Fourier transform (DFT) coefficients of the original image and original image distorted by linear distortion only. Noise quality measure is computed using the original image degraded by linear distortion and noise and the original image degraded by linear distortion only. This approach requires as input not only two images but also the algorithm that causes linear distortion and produces two numbers, which assess image quality. An overview of image quality measures is presented in [8]. Performance of several proposed quality measures is tested in [9]. Distorted images are created using four different image coding algorithms, which produce different types of degradation. Then tested quality measures are computed for each distorted image and these values are compared with subjective assessment. The tested measures included MSE and the measure proposed in [6]. The quality measure from [6] showed best agreement with subjective evaluation compared with other tested measures. However, none of the tested measures was able to predict subjective image quality consistently. Problems with the types of image quality measures similar to those described in [5-7] are that there are no standards for modeling the same properties of HVS. For example, models for brightness perception in [5] and [6] are totally different. Also, there are various contrast sensitivity functions estimated by different authors. Another problem is computation of perceptual threshold, which different authors compute in a totally different way. This sometimes requires computation of local contrast (as in [5] and [7]), which is difficult to define for complex images. A different image quality measure called the universal quality index (UQI) was proposed in [10] and [11]. This measure is computed locally to model any distortion and it is defined as a product of three components: correlation, mean and variance of distortion. Most important of these components is the correlation coefficient, which measures the degree of linear relationship between the corresponding blocks of pixels. Quality measure for the whole image is the average value of locally computed values. Most other approaches try to modify MSE by modeling properties of HVS. This approach defines different criterion which is better related to the way in which human observer assesses image quality. No HVS model is used here. The UQI produced unstable results and to avoid this problem, the measure has been generalized to the structural similarity metric (SSIM) that captures the loss of image structure for quality assessment [12]. As in the case of UQI, the overall quality (Mean SSIM) is obtained by computing the average of the locally computed values in a sliding window. A visual information fidelity criterion that is similar to HVS-based measures that relates visual quality to relative image information has been described in [13]. This is a very complex measure requiring a source model, a distortion model, and a HVS model. In [14], a completely new measure based on singular value decomposition that expressed image quality graphically or numerically was presented. The approach presented here will use the basic idea from [10-12], but it will not use image quality measure as it is defined there (only correlation coefficient will be used). In the first step, the reference and distorted images are processed by a simple model of HVS. It consists of a nonlinear function modeling the brightness perception and a 2-D filter modeling the frequency response of HVS (contrast sensitivity function). The 2-D filter used in this HVS model is not fixed. It contains one user-defined parameter which can be changed depending on the content of the reference image. After this, the correlation coefficient is computed on a block-by-block basis for the processed input images. Finally, the quality measure is computed as the average correlation coefficient between the reference image and distorted image, adjusted by the average correlation coefficient between the reference image and error image. This is a simpler model to those described in [10-14]. The rest of this paper is organized as follows. Problems with MSE-based image quality measures are discussed in Section 2. HVS model used to process input images is described in Section 3. Proposed image quality measure is defined next in Section 4. Performances of the proposed measure are illustrated by examples involving images with different types of distortions in Section 5, followed by the conclusion. 2. WHY MSE IS NOT A GOOD MEASURE? MSE and MSE-based measures such as PSNR and SNR are widely used to assess performances of image enhancement and coding algorithms. MSE-based measures work well when images with the same type of degradation are compared. In a sequence of images obtained by distorting an image by various amounts of the same type of degradation, the image with the smallest MSE will be perceived by a human observer to be the closest to the original image. However, when images with different types of degradation are compared, the image with the smallest MSE will not always be perceived to be the closest to the original image. It is possible to create images with the same 8 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) MSE value but with totally different visual quality. There are three main reasons for disagreement between MSE values and subjective evaluation. First reason for this is presence of spatial filtering in HVS. This can be demonstrated by adding a white Gaussian filtered (2-D FIR filter) noise to the original Lena image for various values of filter cut-off frequencies, fmin and fmax. Each time the filtered noise is scaled so that its standard deviation is 10. The resulting images shown in Figures 2-4 all have MSE values of 100, however, their visual qualities are quite different. Noise at very low and very high frequencies are less noticeable than noise at mid frequencies. This indicates band pass characteristic nature of HVS. One way to solve this problem is by using weighted MSE, where reference and distorted images are filtered by contrast sensitivity function (CSF) and then MSE is computed using filtered images [15]. Second reason for disagreement between MSE values and subjective assessment is in the criterion itself, i.e., MSE treats image intensities as a set of uncorrelated numbers [10-12]. One pixel is never seen in isolation from the surrounding pixels and they always create some structure in an image. This information about the structure created by a group of pixels is more important for a human observer than intensity at some spatial location. However, this is completely ignored by MSE and its computation is not quite related to the way human observers perceive image quality. The images distorted by a wide variety of corruptions such as impulsive salt-pepper noise, additive Gaussian noise, multiplicative speckle noise, contrast stretching, blurring, and JPEG compression yield the same MSE value relative to the original image but very different visual quality. This is illustrated with examples using Lena image in [10-11]. This problem cannot be solved by some modification of MSE, such as weighted MSE and it is necessary to define a different image quality measure. In this paper, we propose a new quality measure, Q and it is described later in Section 4. Third reason for low correlation between MSE values and subjective visual quality is due to the fact that MSE does not differentiate between random and signal dependent distortions. Random noise such as additive white Gaussian noise is much less objectionable for a human viewer than signal dependant distortion such as blurring. The results of these experiments are presented in Tables I-III in Section 5.

EARLIER¶9-15

8 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) MSE value but with totally different visual quality. There are three main reasons for disagreement between MSE values and subjective evaluation. First reason for this is presence of spatial filtering in HVS. This can be demonstrated by adding a white Gaussian filtered (2-D FIR filter) noise to the original Lena image for various values of filter cut-off frequencies, fmin and fmax. Each time the filtered noise is scaled so that its standard deviation is 10. The resulting images shown in Figures 2-4 all have MSE values of 100, however, their visual qualities are quite different. Noise at very low and very high frequencies are less noticeable than noise at mid frequencies. This indicates band pass characteristic nature of HVS. One way to solve this problem is by using weighted MSE, where reference and distorted images are filtered by contrast sensitivity function (CSF) and then MSE is computed using filtered images [15]. Second reason for disagreement between MSE values and subjective assessment is in the criterion itself, i.e., MSE treats image intensities as a set of uncorrelated numbers [10-12]. One pixel is never seen in isolation from the surrounding pixels and they always create some structure in an image. This information about the structure created by a group of pixels is more important for a human observer than intensity at some spatial location. However, this is completely ignored by MSE and its computation is not quite related to the way human observers perceive image quality. The images distorted by a wide variety of corruptions such as impulsive salt-pepper noise, additive Gaussian noise, multiplicative speckle noise, contrast stretching, blurring, and JPEG compression yield the same MSE value relative to the original image but very different visual quality. This is illustrated with examples using Lena image in [10-11]. This problem cannot be solved by some modification of MSE, such as weighted MSE and it is necessary to define a different image quality measure. In this paper, we propose a new quality measure, Q and it is described later in Section 4. Third reason for low correlation between MSE values and subjective visual quality is due to the fact that MSE does not differentiate between random and signal dependent distortions. Random noise such as additive white Gaussian noise is much less objectionable for a human viewer than signal dependant distortion such as blurring. The results of these experiments are presented in Tables I-III in Section 5. 3. HVS MODEL Image quality measure developed here will use a simple model of HVS consisting of a nonlinear function, modeling brightness perception of HVS and a 2-D filter, modeling frequency response of HVS. This type of HVS model was used in [15] and [16]. However, nonlinear function and 2-D filter used here are not the same as those developed in [15]. Digital images are represented using a finite number of intensity levels. For example, gray scale images with 8 bits/pixel are represented by 256 intensity levels, where 0 corresponds to the darkest level and 255 corresponds to the brightest level. Intensities between these two extremes represent various shades of gray level. Perceived brightness is a nonlinear function of intensity and this is modeled by transforming input intensities by some monotonically increasing nonlinear function. Various functions have been suggested to model this. One frequently cited result states that brightness perceived by HVS is proportional to the logarithm of intensity as it is known as Weber’s law [8]. HVS model developed in [15] uses cube root function to model this effect. Some use brightness perception models developed for specific display device and an example of this can be found in [6]. Logarithmic or cube root functions are not accurate models for brightness perception by HVS especially at low intensities. According to them the sensitivity to intensity increases is greatest at low intensities and decreases at mid and high intensities. This is clearly wrong, because an increase in intensity result in a very small increase in perceived brightness, when the intensity is low. The following nonlinear function will be used here instead [17]. I I I I I I I I B I I B 2 /) ( if ) (2 2 2 /) ( if ) (2 2 0 if 0 2 2 (1) Where, ITR = 20 is threshold value for intensity, IMAX = 255 is the maximum value of intensity and BMAX = 100 is the maximum value of perceived brightness. When the intensity goes from 0 to 20 there is no increase in perceived brightness and when intensity goes from 20 to 255, perceived brightness goes from 0 to 100 as shown in Figure 1. This range is chosen for convenience. The role of this function is to emphasize intensities in the mid range and de-emphasize very high and very low intensities, which approximates the brightness perception by HVS. The second component of HVS model is a 2-D filter, which models frequency response of HVS. HVS is not equally sensitive to all spatial frequencies and it is a frequency selective system. If HVS were not a frequency selective system, then we would have seen the same amount of noise in each image. Since we see less noise at very low and very high frequencies than in the mid frequency range, HVS must be a band pass system. This is modeled by a contrast sensitivity function (CSF), which represents the frequency response of HVS. CSF models HVS as a 2-D filter. Spatial frequency is expressed in cycles per unit visual angle (degree) instead of cycles per unit length (cm). This is because images can be observed form the various distances and the same spatial frequency expressed in cycles/cm will be perceived differently for different viewing distances. Various functions have been suggested to model this effect. Most are band pass in nature but maximum at different frequencies. For example, it was noted in [9, 10] that various functions suggested by different authors have maximum at frequencies ranging from 3 to 10 cycles/degree. HVS model presented in [18] uses a high An Adaptive Image Quality Assessment Algorithm 9 pass filter. The reasoning behind this is that HVS treats near and far objects in the same way. Since there is no reliable way to tell which one of these functions represents the best model of HVS, none of them will be used here. Instead, CSF given by the following formula will be used [17]: ⎪ ⎪ ⎩ ⎪ ⎪ ⎨ ⎧ > − − < < ≤ − + = 0 1.1 0 0 for ), ) (1.0 exp( ree cycles/deg 3 for ,1 ree cycles/deg 3 for )), 3192 .0 ( exp( ) 8512 .0 0512 .0 ( ) ( f f f f

EARLIER¶12-16

I I I I I I I I B I I B 2 /) ( if ) (2 2 2 /) ( if ) (2 2 0 if 0 2 2 (1) Where, ITR = 20 is threshold value for intensity, IMAX = 255 is the maximum value of intensity and BMAX = 100 is the maximum value of perceived brightness. When the intensity goes from 0 to 20 there is no increase in perceived brightness and when intensity goes from 20 to 255, perceived brightness goes from 0 to 100 as shown in Figure 1. This range is chosen for convenience. The role of this function is to emphasize intensities in the mid range and de-emphasize very high and very low intensities, which approximates the brightness perception by HVS. The second component of HVS model is a 2-D filter, which models frequency response of HVS. HVS is not equally sensitive to all spatial frequencies and it is a frequency selective system. If HVS were not a frequency selective system, then we would have seen the same amount of noise in each image. Since we see less noise at very low and very high frequencies than in the mid frequency range, HVS must be a band pass system. This is modeled by a contrast sensitivity function (CSF), which represents the frequency response of HVS. CSF models HVS as a 2-D filter. Spatial frequency is expressed in cycles per unit visual angle (degree) instead of cycles per unit length (cm). This is because images can be observed form the various distances and the same spatial frequency expressed in cycles/cm will be perceived differently for different viewing distances. Various functions have been suggested to model this effect. Most are band pass in nature but maximum at different frequencies. For example, it was noted in [9, 10] that various functions suggested by different authors have maximum at frequencies ranging from 3 to 10 cycles/degree. HVS model presented in [18] uses a high An Adaptive Image Quality Assessment Algorithm 9 pass filter. The reasoning behind this is that HVS treats near and far objects in the same way. Since there is no reliable way to tell which one of these functions represents the best model of HVS, none of them will be used here. Instead, CSF given by the following formula will be used [17]: ⎪ ⎪ ⎩ ⎪ ⎪ ⎨ ⎧ > − − < < ≤ − + = 0 1.1 0 0 for ), ) (1.0 exp( ree cycles/deg 3 for ,1 ree cycles/deg 3 for )), 3192 .0 ( exp( ) 8512 .0 0512 .0 ( ) ( f f f f f f f f f f H (2) Where, f denotes frequency in cycles/degree and f0 represents the frequency in cycles/degree at which CSF starts to decrease exponentially. This parameter is user-defined and depends on the reference image. If the reference image contains one large object (for example “Lena” image) than f0 takes lower values than in the case when the reference image contains smaller objects or lot of fine details. Choice of values for this parameter will be discussed in Section 5. This requires some user intervention but it yields better results. This function is shown in Figure 2 for f0=5 cycles/degree.

EARLIER¶12-16

I I I I I I I I B I I B 2 /) ( if ) (2 2 2 /) ( if ) (2 2 0 if 0 2 2 (1) Where, ITR = 20 is threshold value for intensity, IMAX = 255 is the maximum value of intensity and BMAX = 100 is the maximum value of perceived brightness. When the intensity goes from 0 to 20 there is no increase in perceived brightness and when intensity goes from 20 to 255, perceived brightness goes from 0 to 100 as shown in Figure 1. This range is chosen for convenience. The role of this function is to emphasize intensities in the mid range and de-emphasize very high and very low intensities, which approximates the brightness perception by HVS. The second component of HVS model is a 2-D filter, which models frequency response of HVS. HVS is not equally sensitive to all spatial frequencies and it is a frequency selective system. If HVS were not a frequency selective system, then we would have seen the same amount of noise in each image. Since we see less noise at very low and very high frequencies than in the mid frequency range, HVS must be a band pass system. This is modeled by a contrast sensitivity function (CSF), which represents the frequency response of HVS. CSF models HVS as a 2-D filter. Spatial frequency is expressed in cycles per unit visual angle (degree) instead of cycles per unit length (cm). This is because images can be observed form the various distances and the same spatial frequency expressed in cycles/cm will be perceived differently for different viewing distances. Various functions have been suggested to model this effect. Most are band pass in nature but maximum at different frequencies. For example, it was noted in [9, 10] that various functions suggested by different authors have maximum at frequencies ranging from 3 to 10 cycles/degree. HVS model presented in [18] uses a high An Adaptive Image Quality Assessment Algorithm 9 pass filter. The reasoning behind this is that HVS treats near and far objects in the same way. Since there is no reliable way to tell which one of these functions represents the best model of HVS, none of them will be used here. Instead, CSF given by the following formula will be used [17]: ⎪ ⎪ ⎩ ⎪ ⎪ ⎨ ⎧ > − − < < ≤ − + = 0 1.1 0 0 for ), ) (1.0 exp( ree cycles/deg 3 for ,1 ree cycles/deg 3 for )), 3192 .0 ( exp( ) 8512 .0 0512 .0 ( ) ( f f f f f f f f f f H (2) Where, f denotes frequency in cycles/degree and f0 represents the frequency in cycles/degree at which CSF starts to decrease exponentially. This parameter is user-defined and depends on the reference image. If the reference image contains one large object (for example “Lena” image) than f0 takes lower values than in the case when the reference image contains smaller objects or lot of fine details. Choice of values for this parameter will be discussed in Section 5. This requires some user intervention but it yields better results. This function is shown in Figure 2 for f0=5 cycles/degree.

EARLIER¶13-19

The second component of HVS model is a 2-D filter, which models frequency response of HVS. HVS is not equally sensitive to all spatial frequencies and it is a frequency selective system. If HVS were not a frequency selective system, then we would have seen the same amount of noise in each image. Since we see less noise at very low and very high frequencies than in the mid frequency range, HVS must be a band pass system. This is modeled by a contrast sensitivity function (CSF), which represents the frequency response of HVS. CSF models HVS as a 2-D filter. Spatial frequency is expressed in cycles per unit visual angle (degree) instead of cycles per unit length (cm). This is because images can be observed form the various distances and the same spatial frequency expressed in cycles/cm will be perceived differently for different viewing distances. Various functions have been suggested to model this effect. Most are band pass in nature but maximum at different frequencies. For example, it was noted in [9, 10] that various functions suggested by different authors have maximum at frequencies ranging from 3 to 10 cycles/degree. HVS model presented in [18] uses a high An Adaptive Image Quality Assessment Algorithm 9 pass filter. The reasoning behind this is that HVS treats near and far objects in the same way. Since there is no reliable way to tell which one of these functions represents the best model of HVS, none of them will be used here. Instead, CSF given by the following formula will be used [17]: ⎪ ⎪ ⎩ ⎪ ⎪ ⎨ ⎧ > − − < < ≤ − + = 0 1.1 0 0 for ), ) (1.0 exp( ree cycles/deg 3 for ,1 ree cycles/deg 3 for )), 3192 .0 ( exp( ) 8512 .0 0512 .0 ( ) ( f f f f f f f f f f H (2) Where, f denotes frequency in cycles/degree and f0 represents the frequency in cycles/degree at which CSF starts to decrease exponentially. This parameter is user-defined and depends on the reference image. If the reference image contains one large object (for example “Lena” image) than f0 takes lower values than in the case when the reference image contains smaller objects or lot of fine details. Choice of values for this parameter will be discussed in Section 5. This requires some user intervention but it yields better results. This function is shown in Figure 2 for f0=5 cycles/degree. Fig. 1 Nonlinear Modeling of Brightness Perception This property of HVS can be taken into account by filtering input image by CSF. The input image f(m,n) is filtered by 2-D filter H(f1,f2), which results in processed image x(m,n). A simple HVS system (Figure 3) comprising of a nonlinear function followed by a 2-D filter models the brightness perception and frequency response of HVS. Nonlinear function is given by equation (1) and 2-D filter is given by ) ( ) , ( 2 2 2 1 2 1 f f H f f H + = (assumed radial symmetry), where H(f) is given by equation (2). The output signal in this case depends on the viewing distance, the width and the height and the number of pixels of the input image. This is reasonable because image perception by HVS also depends on these parameters which must be given. Fig. 6 Contrast Sensitivity Function f(m,n), p(m,n) x(m,n), y(m.n) Fig. 7 A Simple HVS System Model 4. PROPOSED IMAGE QUALITY MEASURE The original or reference image is denoted by f(m,n) and the distorted image is denoted by p(m,n). The size of both these images is M x N pixels. At first, they are transformed by the HVS system, which models the brightness perception and frequency selectivity of HVS. The processing of the input images f(m,n) and p(m,n) by this model will produce images x(m,n) and y(m,n) respectively, which will be used to evaluate the quality measure. Processed images x(m,n) and y(m,n) can be used to compute weighted MSE (WMSE) as was done in [8]. Instead of this, a different procedure, similar to what was done in [6] and [7], will be used here. The processed images x(m,n) and y(m,n) are partitioned into 8x8 pixel blocks. For each of these blocks, the correlation coefficient between the corresponding samples from images x(m,n) and y(m,n) is computed. If we denote the correlation coefficient value between x(m,n) and y(m,n)

EARLIER¶6-10

JASC 12-1-2 An Adaptive Image Quality Assessment Algorithm 7 intensities and perceived brightness, frequency response of HVS (contrast sensitivity function) and texture masking [5-7]. The algorithm described in [5] can identify the pixels with visible distortion, but it cannot quantify the extent of easily visible distortion, because every spatial location with clearly visible distortion will have detection probability of one in the output map. Also, this algorithm does not produce one number that indicates image quality. In [6], the properties of HVS are used to design an algorithm for image coding. In this approach input image is first decomposed into 16 sub-bands using generalized quadrature mirror filters. Image quality measure in this case is defined as MSE of normalized sub-band coefficients of the original image and distorted image, where each sub-band coefficient is normalized by the perceptual threshold for that sub-band at that spatial location. Another approach presented in [7], assumes that input image is degraded by two sources of degradation: linear distortion (for example as a result of filtering) and additive noise. In the first step linear distortion and additive noise are separated and then two quality measures are computed: distortion and noise quality measures. Distortion measure is computed using discrete Fourier transform (DFT) coefficients of the original image and original image distorted by linear distortion only. Noise quality measure is computed using the original image degraded by linear distortion and noise and the original image degraded by linear distortion only. This approach requires as input not only two images but also the algorithm that causes linear distortion and produces two numbers, which assess image quality. An overview of image quality measures is presented in [8]. Performance of several proposed quality measures is tested in [9]. Distorted images are created using four different image coding algorithms, which produce different types of degradation. Then tested quality measures are computed for each distorted image and these values are compared with subjective assessment. The tested measures included MSE and the measure proposed in [6]. The quality measure from [6] showed best agreement with subjective evaluation compared with other tested measures. However, none of the tested measures was able to predict subjective image quality consistently. Problems with the types of image quality measures similar to those described in [5-7] are that there are no standards for modeling the same properties of HVS. For example, models for brightness perception in [5] and [6] are totally different. Also, there are various contrast sensitivity functions estimated by different authors. Another problem is computation of perceptual threshold, which different authors compute in a totally different way. This sometimes requires computation of local contrast (as in [5] and [7]), which is difficult to define for complex images. A different image quality measure called the universal quality index (UQI) was proposed in [10] and [11]. This measure is computed locally to model any distortion and it is defined as a product of three components: correlation, mean and variance of distortion. Most important of these components is the correlation coefficient, which measures the degree of linear relationship between the corresponding blocks of pixels. Quality measure for the whole image is the average value of locally computed values. Most other approaches try to modify MSE by modeling properties of HVS. This approach defines different criterion which is better related to the way in which human observer assesses image quality. No HVS model is used here. The UQI produced unstable results and to avoid this problem, the measure has been generalized to the structural similarity metric (SSIM) that captures the loss of image structure for quality assessment [12]. As in the case of UQI, the overall quality (Mean SSIM) is obtained by computing the average of the locally computed values in a sliding window. A visual information fidelity criterion that is similar to HVS-based measures that relates visual quality to relative image information has been described in [13]. This is a very complex measure requiring a source model, a distortion model, and a HVS model. In [14], a completely new measure based on singular value decomposition that expressed image quality graphically or numerically was presented. The approach presented here will use the basic idea from [10-12], but it will not use image quality measure as it is defined there (only correlation coefficient will be used). In the first step, the reference and distorted images are processed by a simple model of HVS. It consists of a nonlinear function modeling the brightness perception and a 2-D filter modeling the frequency response of HVS (contrast sensitivity function). The 2-D filter used in this HVS model is not fixed. It contains one user-defined parameter which can be changed depending on the content of the reference image. After this, the correlation coefficient is computed on a block-by-block basis for the processed input images. Finally, the quality measure is computed as the average correlation coefficient between the reference image and distorted image, adjusted by the average correlation coefficient between the reference image and error image. This is a simpler model to those described in [10-14]. The rest of this paper is organized as follows. Problems with MSE-based image quality measures are discussed in Section 2. HVS model used to process input images is described in Section 3. Proposed image quality measure is defined next in Section 4. Performances of the proposed measure are illustrated by examples involving images with different types of distortions in Section 5, followed by the conclusion. 2. WHY MSE IS NOT A GOOD MEASURE? MSE and MSE-based measures such as PSNR and SNR are widely used to assess performances of image enhancement and coding algorithms. MSE-based measures work well when images with the same type of degradation are compared. In a sequence of images obtained by distorting an image by various amounts of the same type of degradation, the image with the smallest MSE will be perceived by a human observer to be the closest to the original image. However, when images with different types of degradation are compared, the image with the smallest MSE will not always be perceived to be the closest to the original image. It is possible to create images with the same 8 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) MSE value but with totally different visual quality. There are three main reasons for disagreement between MSE values and subjective evaluation. First reason for this is presence of spatial filtering in HVS. This can be demonstrated by adding a white Gaussian filtered (2-D FIR filter) noise to the original Lena image for various values of filter cut-off frequencies, fmin and fmax. Each time the filtered noise is scaled so that its standard deviation is 10. The resulting images shown in Figures 2-4 all have MSE values of 100, however, their visual qualities are quite different. Noise at very low and very high frequencies are less noticeable than noise at mid frequencies. This indicates band pass characteristic nature of HVS. One way to solve this problem is by using weighted MSE, where reference and distorted images are filtered by contrast sensitivity function (CSF) and then MSE is computed using filtered images [15]. Second reason for disagreement between MSE values and subjective assessment is in the criterion itself, i.e., MSE treats image intensities as a set of uncorrelated numbers [10-12]. One pixel is never seen in isolation from the surrounding pixels and they always create some structure in an image. This information about the structure created by a group of pixels is more important for a human observer than intensity at some spatial location. However, this is completely ignored by MSE and its computation is not quite related to the way human observers perceive image quality. The images distorted by a wide variety of corruptions such as impulsive salt-pepper noise, additive Gaussian noise, multiplicative speckle noise, contrast stretching, blurring, and JPEG compression yield the same MSE value relative to the original image but very different visual quality. This is illustrated with examples using Lena image in [10-11]. This problem cannot be solved by some modification of MSE, such as weighted MSE and it is necessary to define a different image quality measure. In this paper, we propose a new quality measure, Q and it is described later in Section 4. Third reason for low correlation between MSE values and subjective visual quality is due to the fact that MSE does not differentiate between random and signal dependent distortions. Random noise such as additive white Gaussian noise is much less objectionable for a human viewer than signal dependant distortion such as blurring. The results of these experiments are presented in Tables I-III in Section 5. 3. HVS MODEL Image quality measure developed here will use a simple model of HVS consisting of a nonlinear function, modeling brightness perception of HVS and a 2-D filter, modeling frequency response of HVS. This type of HVS model was used in [15] and [16]. However, nonlinear function and 2-D filter used here are not the same as those developed in [15]. Digital images are represented using a finite number of intensity levels. For example, gray scale images with 8 bits/pixel are represented by 256 intensity levels, where 0 corresponds to the darkest level and 255 corresponds to the brightest level. Intensities between these two extremes

EARLIER¶0-4

Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) DOI 10.7236/JASC2012.1.1.2 An Adaptive Image Quality Assessment Algorithm Ravi Sankar* and Goran Ivkovic iCONS Research Group, Dept. of Electrical Engineering, University of South Florida, USA Abstract An improved algorithm for image quality assessment is presented. First a simple model of human visual system, consisting of a nonlinear function and a 2-D filter, processes the input images. This filter has one user-defined parameter, whose value depends on the reference image. This way the algorithm can adapt to different scenarios. In the next step the average value of locally computed correlation coefficients between the two processed images is found. This criterion is closely related to the way in which human observer assesses image quality. Finally, image quality measure is computed as the average value of locally computed correlation coefficients, adjusted by the average correlation coefficient between the reference and error images. By this approach the proposed measure differentiates between the random and signal dependant distortions, which have different effects on human observer. Performance of the proposed quality measure is illustrated by examples involving images with different types of degradation. Key words: Brightness perception, Contrast sensitivity function, Human visual system, Image quality measure 1. INTRODUCTION Image and video quality assessment methods can be divided into: subjective and objective. The only meaningful subjective (qualitative) measure is mean opinion score (MOS), where quality of an image is rated on some scale by several individuals according to the level of distortion and the mean value of their scores is used as image quality measure. Since human observer is the ultimate receiver of the information contained in an image, this is the best way for image quality assessment. In the case of video, several subjective test methods such as Single Stimulus Continuous Quality Evaluation (SSCQE) and Quality-Oriented Adaptation Scheme (QOAS) have been described in [1-3] to obtain MOS. Objective (quantitative) measures use intensities from the distorted input image and the reference image to compute a value that indicates image quality. The goal is to find a good measure which produces results which are in good agreement with subjective evaluation. Most widely used objective measures are mean squared error (MSE) and MSE-based measures: peak signal to noise ratio (PSNR) and signal to noise ratio (SNR). These simple measures work well when images with the same type of distortion or degradation are compared. In this case distorted image with smaller MSE will be perceived closer to the original image than the one with greater MSE. However, when images with different types of distortion (additive white Gaussian noise, blurring. contrast stretching, JPEG2000 coding/compression) are compared MSE does not produce results that correlate well with subjective quality assessment. Images with different types of distortions with the same MSE values can have very different subjective visual qualities. One role of these measures is to assess performance of algorithms for image enhancement and coding. Another role is to optimize image processing algorithms. For example, in image coding the goal is to represent images with as few bits as possible with the minimal loss of image quality. Quality measure is used as a criterion in a sense that the bits that contribute more significantly to the used quality measure are coded first. The objective video quality measures have been described recently in [4]. Examples of digital video distortions including tiling, smearing, jerkiness, edge busyness, and object retention that are produced by video processing systems are fundamentally different kinds of impairments than those produced by image processing systems. So both subjective and objective quality assessments are quite different for images and videos. In this paper we focus our attention only on image quality monitoring and assessment. In order to find a criterion, which agrees with subjective assessment, several other algorithms have been developed. These algorithms try to assess image quality by taking into account the properties of human visual system (HVS). Most of these algorithms are trying to model the following three properties of HVS: nonlinear relationship between image

EARLIER¶8-13

2. WHY MSE IS NOT A GOOD MEASURE? MSE and MSE-based measures such as PSNR and SNR are widely used to assess performances of image enhancement and coding algorithms. MSE-based measures work well when images with the same type of degradation are compared. In a sequence of images obtained by distorting an image by various amounts of the same type of degradation, the image with the smallest MSE will be perceived by a human observer to be the closest to the original image. However, when images with different types of degradation are compared, the image with the smallest MSE will not always be perceived to be the closest to the original image. It is possible to create images with the same 8 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) MSE value but with totally different visual quality. There are three main reasons for disagreement between MSE values and subjective evaluation. First reason for this is presence of spatial filtering in HVS. This can be demonstrated by adding a white Gaussian filtered (2-D FIR filter) noise to the original Lena image for various values of filter cut-off frequencies, fmin and fmax. Each time the filtered noise is scaled so that its standard deviation is 10. The resulting images shown in Figures 2-4 all have MSE values of 100, however, their visual qualities are quite different. Noise at very low and very high frequencies are less noticeable than noise at mid frequencies. This indicates band pass characteristic nature of HVS. One way to solve this problem is by using weighted MSE, where reference and distorted images are filtered by contrast sensitivity function (CSF) and then MSE is computed using filtered images [15]. Second reason for disagreement between MSE values and subjective assessment is in the criterion itself, i.e., MSE treats image intensities as a set of uncorrelated numbers [10-12]. One pixel is never seen in isolation from the surrounding pixels and they always create some structure in an image. This information about the structure created by a group of pixels is more important for a human observer than intensity at some spatial location. However, this is completely ignored by MSE and its computation is not quite related to the way human observers perceive image quality. The images distorted by a wide variety of corruptions such as impulsive salt-pepper noise, additive Gaussian noise, multiplicative speckle noise, contrast stretching, blurring, and JPEG compression yield the same MSE value relative to the original image but very different visual quality. This is illustrated with examples using Lena image in [10-11]. This problem cannot be solved by some modification of MSE, such as weighted MSE and it is necessary to define a different image quality measure. In this paper, we propose a new quality measure, Q and it is described later in Section 4. Third reason for low correlation between MSE values and subjective visual quality is due to the fact that MSE does not differentiate between random and signal dependent distortions. Random noise such as additive white Gaussian noise is much less objectionable for a human viewer than signal dependant distortion such as blurring. The results of these experiments are presented in Tables I-III in Section 5. 3. HVS MODEL Image quality measure developed here will use a simple model of HVS consisting of a nonlinear function, modeling brightness perception of HVS and a 2-D filter, modeling frequency response of HVS. This type of HVS model was used in [15] and [16]. However, nonlinear function and 2-D filter used here are not the same as those developed in [15]. Digital images are represented using a finite number of intensity levels. For example, gray scale images with 8 bits/pixel are represented by 256 intensity levels, where 0 corresponds to the darkest level and 255 corresponds to the brightest level. Intensities between these two extremes represent various shades of gray level. Perceived brightness is a nonlinear function of intensity and this is modeled by transforming input intensities by some monotonically increasing nonlinear function. Various functions have been suggested to model this. One frequently cited result states that brightness perceived by HVS is proportional to the logarithm of intensity as it is known as Weber’s law [8]. HVS model developed in [15] uses cube root function to model this effect. Some use brightness perception models developed for specific display device and an example of this can be found in [6]. Logarithmic or cube root functions are not accurate models for brightness perception by HVS especially at low intensities. According to them the sensitivity to intensity increases is greatest at low intensities and decreases at mid and high intensities. This is clearly wrong, because an increase in intensity result in a very small increase in perceived brightness, when the intensity is low. The following nonlinear function will be used here instead [17]. I I I I I I I I B I I B 2 /) ( if ) (2 2 2 /) ( if ) (2 2 0 if 0 2 2 (1) Where, ITR = 20 is threshold value for intensity, IMAX = 255 is the maximum value of intensity and BMAX = 100 is the maximum value of perceived brightness. When the intensity goes from 0 to 20 there is no increase in perceived brightness and when intensity goes from 20 to 255, perceived brightness goes from 0 to 100 as shown in Figure 1. This range is chosen for convenience. The role of this function is to emphasize intensities in the mid range and de-emphasize very high and very low intensities, which approximates the brightness perception by HVS. The second component of HVS model is a 2-D filter, which models frequency response of HVS. HVS is not equally sensitive to all spatial frequencies and it is a frequency selective system. If HVS were not a frequency selective system, then we would have seen the same amount of noise in each image. Since we see less noise at very low and very high frequencies than in the mid frequency range, HVS must be a band pass system. This is modeled by a contrast sensitivity function (CSF), which represents the frequency response of HVS. CSF models HVS as a 2-D filter. Spatial frequency is expressed in cycles per unit visual angle (degree) instead of cycles per unit length (cm). This is because images can be observed form the various distances and the same spatial frequency expressed in cycles/cm will be perceived differently for different viewing distances. Various functions have been suggested to model this effect. Most are band pass in nature but maximum at different frequencies. For example, it was noted in [9, 10] that various functions suggested by different authors have maximum at frequencies ranging from 3 to 10 cycles/degree. HVS model presented in [18] uses a high

EARLIER¶17-21

Fig. 1 Nonlinear Modeling of Brightness Perception This property of HVS can be taken into account by filtering input image by CSF. The input image f(m,n) is filtered by 2-D filter H(f1,f2), which results in processed image x(m,n). A simple HVS system (Figure 3) comprising of a nonlinear function followed by a 2-D filter models the brightness perception and frequency response of HVS. Nonlinear function is given by equation (1) and 2-D filter is given by ) ( ) , ( 2 2 2 1 2 1 f f H f f H + = (assumed radial symmetry), where H(f) is given by equation (2). The output signal in this case depends on the viewing distance, the width and the height and the number of pixels of the input image. This is reasonable because image perception by HVS also depends on these parameters which must be given. Fig. 6 Contrast Sensitivity Function f(m,n), p(m,n) x(m,n), y(m.n) Fig. 7 A Simple HVS System Model 4. PROPOSED IMAGE QUALITY MEASURE The original or reference image is denoted by f(m,n) and the distorted image is denoted by p(m,n). The size of both these images is M x N pixels. At first, they are transformed by the HVS system, which models the brightness perception and frequency selectivity of HVS. The processing of the input images f(m,n) and p(m,n) by this model will produce images x(m,n) and y(m,n) respectively, which will be used to evaluate the quality measure. Processed images x(m,n) and y(m,n) can be used to compute weighted MSE (WMSE) as was done in [8]. Instead of this, a different procedure, similar to what was done in [6] and [7], will be used here. The processed images x(m,n) and y(m,n) are partitioned into 8x8 pixel blocks. For each of these blocks, the correlation coefficient between the corresponding samples from images x(m,n) and y(m,n) is computed. If we denote the correlation coefficient value between x(m,n) and y(m,n) (3) Where, L is the total number of data blocks and the correlation coefficient between two vectors X = [x1 x2 ... xN] and Y = [y1 y2 ... yN] is defined as Y X XY XY σ σ σ ρ / = where XY σ is the covariance and y x σ σ , are the standard Nonlinear Function 2-D Filter 10 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012)

EARLIER¶17-21

Fig. 1 Nonlinear Modeling of Brightness Perception This property of HVS can be taken into account by filtering input image by CSF. The input image f(m,n) is filtered by 2-D filter H(f1,f2), which results in processed image x(m,n). A simple HVS system (Figure 3) comprising of a nonlinear function followed by a 2-D filter models the brightness perception and frequency response of HVS. Nonlinear function is given by equation (1) and 2-D filter is given by ) ( ) , ( 2 2 2 1 2 1 f f H f f H + = (assumed radial symmetry), where H(f) is given by equation (2). The output signal in this case depends on the viewing distance, the width and the height and the number of pixels of the input image. This is reasonable because image perception by HVS also depends on these parameters which must be given. Fig. 6 Contrast Sensitivity Function f(m,n), p(m,n) x(m,n), y(m.n) Fig. 7 A Simple HVS System Model 4. PROPOSED IMAGE QUALITY MEASURE The original or reference image is denoted by f(m,n) and the distorted image is denoted by p(m,n). The size of both these images is M x N pixels. At first, they are transformed by the HVS system, which models the brightness perception and frequency selectivity of HVS. The processing of the input images f(m,n) and p(m,n) by this model will produce images x(m,n) and y(m,n) respectively, which will be used to evaluate the quality measure. Processed images x(m,n) and y(m,n) can be used to compute weighted MSE (WMSE) as was done in [8]. Instead of this, a different procedure, similar to what was done in [6] and [7], will be used here. The processed images x(m,n) and y(m,n) are partitioned into 8x8 pixel blocks. For each of these blocks, the correlation coefficient between the corresponding samples from images x(m,n) and y(m,n) is computed. If we denote the correlation coefficient value between x(m,n) and y(m,n) (3) Where, L is the total number of data blocks and the correlation coefficient between two vectors X = [x1 x2 ... xN] and Y = [y1 y2 ... yN] is defined as Y X XY XY σ σ σ ρ / = where XY σ is the covariance and y x σ σ , are the standard Nonlinear Function 2-D Filter 10 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012)

EARLIER¶18-22

) ( ) , ( 2 2 2 1 2 1 f f H f f H + = (assumed radial symmetry), where H(f) is given by equation (2). The output signal in this case depends on the viewing distance, the width and the height and the number of pixels of the input image. This is reasonable because image perception by HVS also depends on these parameters which must be given. Fig. 6 Contrast Sensitivity Function f(m,n), p(m,n) x(m,n), y(m.n) Fig. 7 A Simple HVS System Model 4. PROPOSED IMAGE QUALITY MEASURE The original or reference image is denoted by f(m,n) and the distorted image is denoted by p(m,n). The size of both these images is M x N pixels. At first, they are transformed by the HVS system, which models the brightness perception and frequency selectivity of HVS. The processing of the input images f(m,n) and p(m,n) by this model will produce images x(m,n) and y(m,n) respectively, which will be used to evaluate the quality measure. Processed images x(m,n) and y(m,n) can be used to compute weighted MSE (WMSE) as was done in [8]. Instead of this, a different procedure, similar to what was done in [6] and [7], will be used here. The processed images x(m,n) and y(m,n) are partitioned into 8x8 pixel blocks. For each of these blocks, the correlation coefficient between the corresponding samples from images x(m,n) and y(m,n) is computed. If we denote the correlation coefficient value between x(m,n) and y(m,n) (3) Where, L is the total number of data blocks and the correlation coefficient between two vectors X = [x1 x2 ... xN] and Y = [y1 y2 ... yN] is defined as Y X XY XY σ σ σ ρ / = where XY σ is the covariance and y x σ σ , are the standard Nonlinear Function 2-D Filter 10 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) deviations of X and Y. By computing the correlation coefficient locally we force it to concentrate more on the difference in details, which is essential in image quality assessment. This is also closely related to the way in which human observer assesses image quality by scanning an image piece by piece and determining the difference between corresponding blocks of pixels. Average correlation coefficient given by equation (3) is a good indicator of similarity between the reference image and the distorted image, but it has one downside: it does not differentiate between random and signal dependent distortions. Therefore, it will overestimate the image degradation in the case of random noise and underestimate it in the case of signal dependent noise. This problem can be eliminated by computing the average correlation coefficient between the reference image x(m,n) and the error image e(m,n), which is given by

EARLIER¶21-26

where XY σ is the covariance and y x σ σ , are the standard Nonlinear Function 2-D Filter 10 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) deviations of X and Y. By computing the correlation coefficient locally we force it to concentrate more on the difference in details, which is essential in image quality assessment. This is also closely related to the way in which human observer assesses image quality by scanning an image piece by piece and determining the difference between corresponding blocks of pixels. Average correlation coefficient given by equation (3) is a good indicator of similarity between the reference image and the distorted image, but it has one downside: it does not differentiate between random and signal dependent distortions. Therefore, it will overestimate the image degradation in the case of random noise and underestimate it in the case of signal dependent noise. This problem can be eliminated by computing the average correlation coefficient between the reference image x(m,n) and the error image e(m,n), which is given by ) , ( } { ) , ( ) , ( _ n m y sign n m x n m e avg xy ρ − = (4) Then local correlation coefficients are computed block by block between images x(m,n) and e(m,n). If the local correlation coefficient for the ith block of data is denoted by ) (i xe ρ , then the average correlation coefficient between images x(m,n) and e(m,n) is: ∑ = L i xe avg xe i L ) ( 1 _ ρ ρ (5) This coefficient will have very small positive or negative value close to zero, when there is little or no correlation between the images x(m,n) and e(m,n) and this will indicate random noise. On the other hand if there is a significant correlation between x(m,n) and e(m,n) the average correlation coefficient avg xe _ ρ will have a higher absolute value, indicating signal dependent noise. Finally, the proposed new image quality measure denoted by Q is defined as: ⎟⎟ ⎠ ⎞ ⎜⎜ ⎝ ⎛ − + = = 15 .0 3.0 | | tanh 5.0 2.1 ) ( where | |} { _ _ ) ( _ _ _ avg xe avg xe f avg xy avg xy f sign Q avg xe ρ ρ ρ ρ ρ (6) The quality measure (Q) defined by equation (6) needs some explanation. It will have the same sign as

EARLIER¶22-27

deviations of X and Y. By computing the correlation coefficient locally we force it to concentrate more on the difference in details, which is essential in image quality assessment. This is also closely related to the way in which human observer assesses image quality by scanning an image piece by piece and determining the difference between corresponding blocks of pixels. Average correlation coefficient given by equation (3) is a good indicator of similarity between the reference image and the distorted image, but it has one downside: it does not differentiate between random and signal dependent distortions. Therefore, it will overestimate the image degradation in the case of random noise and underestimate it in the case of signal dependent noise. This problem can be eliminated by computing the average correlation coefficient between the reference image x(m,n) and the error image e(m,n), which is given by ) , ( } { ) , ( ) , ( _ n m y sign n m x n m e avg xy ρ − = (4) Then local correlation coefficients are computed block by block between images x(m,n) and e(m,n). If the local correlation coefficient for the ith block of data is denoted by ) (i xe ρ , then the average correlation coefficient between images x(m,n) and e(m,n) is: ∑ = L i xe avg xe i L ) ( 1 _ ρ ρ (5) This coefficient will have very small positive or negative value close to zero, when there is little or no correlation between the images x(m,n) and e(m,n) and this will indicate random noise. On the other hand if there is a significant correlation between x(m,n) and e(m,n) the average correlation coefficient avg xe _ ρ will have a higher absolute value, indicating signal dependent noise. Finally, the proposed new image quality measure denoted by Q is defined as: ⎟⎟ ⎠ ⎞ ⎜⎜ ⎝ ⎛ − + = = 15 .0 3.0 | | tanh 5.0 2.1 ) ( where | |} { _ _ ) ( _ _ _ avg xe avg xe f avg xy avg xy f sign Q avg xe ρ ρ ρ ρ ρ (6) The quality measure (Q) defined by equation (6) needs some explanation. It will have the same sign as avg xy _ ρ given by equation (3), but its magnitude will be modified value of the magnitude of avg xe _ ρ . This modification is made according to the way the average correlation coefficient changes. This is done by using a function of avg xe _ ρ , i.e., ) ( _ avg xe f ρ given in equation (6)

EARLIER¶24-29

by ) (i xe ρ , then the average correlation coefficient between images x(m,n) and e(m,n) is: ∑ = L i xe avg xe i L ) ( 1 _ ρ ρ (5) This coefficient will have very small positive or negative value close to zero, when there is little or no correlation between the images x(m,n) and e(m,n) and this will indicate random noise. On the other hand if there is a significant correlation between x(m,n) and e(m,n) the average correlation coefficient avg xe _ ρ will have a higher absolute value, indicating signal dependent noise. Finally, the proposed new image quality measure denoted by Q is defined as: ⎟⎟ ⎠ ⎞ ⎜⎜ ⎝ ⎛ − + = = 15 .0 3.0 | | tanh 5.0 2.1 ) ( where | |} { _ _ ) ( _ _ _ avg xe avg xe f avg xy avg xy f sign Q avg xe ρ ρ ρ ρ ρ (6) The quality measure (Q) defined by equation (6) needs some explanation. It will have the same sign as avg xy _ ρ given by equation (3), but its magnitude will be modified value of the magnitude of avg xe _ ρ . This modification is made according to the way the average correlation coefficient changes. This is done by using a function of avg xe _ ρ , i.e., ) ( _ avg xe f ρ given in equation (6) and plotted in Figure 4. If the magnitude of avg xe _ ρ is small (close to zero), then it indicates random noise. If the value of this coefficient is close to one, then it indicates signal dependant noise and when the value is somewhere in between that indicates transition between random and signal dependant noise. If an image is corrupted by random noise the magnitude of the quality measure will be higher than the magnitude of avg xy _ ρ and lower if the image is corrupted by signal dependent noise, with transition period in between. This is in accordance with the previous discussion on effects of random and signal dependent noise. The values of the various constants that appear in function ) ( _ avg xe f ρ can be

EARLIER¶26-30

⎟⎟ ⎠ ⎞ ⎜⎜ ⎝ ⎛ − + = = 15 .0 3.0 | | tanh 5.0 2.1 ) ( where | |} { _ _ ) ( _ _ _ avg xe avg xe f avg xy avg xy f sign Q avg xe ρ ρ ρ ρ ρ (6) The quality measure (Q) defined by equation (6) needs some explanation. It will have the same sign as avg xy _ ρ given by equation (3), but its magnitude will be modified value of the magnitude of avg xe _ ρ . This modification is made according to the way the average correlation coefficient changes. This is done by using a function of avg xe _ ρ , i.e., ) ( _ avg xe f ρ given in equation (6) and plotted in Figure 4. If the magnitude of avg xe _ ρ is small (close to zero), then it indicates random noise. If the value of this coefficient is close to one, then it indicates signal dependant noise and when the value is somewhere in between that indicates transition between random and signal dependant noise. If an image is corrupted by random noise the magnitude of the quality measure will be higher than the magnitude of avg xy _ ρ and lower if the image is corrupted by signal dependent noise, with transition period in between. This is in accordance with the previous discussion on effects of random and signal dependent noise. The values of the various constants that appear in function ) ( _ avg xe f ρ can be used to give different weight to random and signal dependent distortion. Their values can be chosen according to the users’ personal preferences. Fig. 4 Function Modeling the Different Effects of Random and

EARLIER¶29-33

by signal dependent noise, with transition period in between. This is in accordance with the previous discussion on effects of random and signal dependent noise. The values of the various constants that appear in function ) ( _ avg xe f ρ can be used to give different weight to random and signal dependent distortion. Their values can be chosen according to the users’ personal preferences. Fig. 4 Function Modeling the Different Effects of Random and Signal Dependent Distortions Finally, the proposed algorithm can be summarized in the following four steps: 1. Given the reference image f(m,n), distorted image p(m,n), width, height and number of pixels of the input images and viewing distance, compute images x(m,n) and y(m,n) using the described model of HVS. 2. Compute avg xy _ ρ as the average value of locally computed correlation coefficients between images x(m,n) and y(m,n). 3. Compute avg xe _ ρ as the average value of the locally computed correlation coefficients between images x(m,n) and e(m,n) given by equation (4). 4. Find the image quality measure Q using equation (6). 5. EXPERIMENTAL RESULTS Performances of the algorithm will be illustrated by several examples using the following parameter values: An Adaptive Image Quality Assessment Algorithm 11

EARLIER¶29-34

by signal dependent noise, with transition period in between. This is in accordance with the previous discussion on effects of random and signal dependent noise. The values of the various constants that appear in function ) ( _ avg xe f ρ can be used to give different weight to random and signal dependent distortion. Their values can be chosen according to the users’ personal preferences. Fig. 4 Function Modeling the Different Effects of Random and Signal Dependent Distortions Finally, the proposed algorithm can be summarized in the following four steps: 1. Given the reference image f(m,n), distorted image p(m,n), width, height and number of pixels of the input images and viewing distance, compute images x(m,n) and y(m,n) using the described model of HVS. 2. Compute avg xy _ ρ as the average value of locally computed correlation coefficients between images x(m,n) and y(m,n). 3. Compute avg xe _ ρ as the average value of the locally computed correlation coefficients between images x(m,n) and e(m,n) given by equation (4). 4. Find the image quality measure Q using equation (6). 5. EXPERIMENTAL RESULTS Performances of the algorithm will be illustrated by several examples using the following parameter values: An Adaptive Image Quality Assessment Algorithm 11 equal image height and width, viewing distance equal to four times image height and image size of 512 x 512 pixels. Parameter f0 in the equation (2) will depend on the original image. Experiment #1: The quality measure performances for the sequence of “Lena” images with various types of filtered noise as discussed in Section 2 are illustrated. For this image, we set f0=5. All three images in the sequence have the same MSE value of 100, but their visual qualities are quite different. The results obtained, using the proposed adaptive image quality assessment algorithm, are shown in Table 1.

EARLIER¶30-34

used to give different weight to random and signal dependent distortion. Their values can be chosen according to the users’ personal preferences. Fig. 4 Function Modeling the Different Effects of Random and Signal Dependent Distortions Finally, the proposed algorithm can be summarized in the following four steps: 1. Given the reference image f(m,n), distorted image p(m,n), width, height and number of pixels of the input images and viewing distance, compute images x(m,n) and y(m,n) using the described model of HVS. 2. Compute avg xy _ ρ as the average value of locally computed correlation coefficients between images x(m,n) and y(m,n). 3. Compute avg xe _ ρ as the average value of the locally computed correlation coefficients between images x(m,n) and e(m,n) given by equation (4). 4. Find the image quality measure Q using equation (6). 5. EXPERIMENTAL RESULTS Performances of the algorithm will be illustrated by several examples using the following parameter values: An Adaptive Image Quality Assessment Algorithm 11 equal image height and width, viewing distance equal to four times image height and image size of 512 x 512 pixels. Parameter f0 in the equation (2) will depend on the original image. Experiment #1: The quality measure performances for the sequence of “Lena” images with various types of filtered noise as discussed in Section 2 are illustrated. For this image, we set f0=5. All three images in the sequence have the same MSE value of 100, but their visual qualities are quite different. The results obtained, using the proposed adaptive image quality assessment algorithm, are shown in Table 1.

EARLIER¶30-35

used to give different weight to random and signal dependent distortion. Their values can be chosen according to the users’ personal preferences. Fig. 4 Function Modeling the Different Effects of Random and Signal Dependent Distortions Finally, the proposed algorithm can be summarized in the following four steps: 1. Given the reference image f(m,n), distorted image p(m,n), width, height and number of pixels of the input images and viewing distance, compute images x(m,n) and y(m,n) using the described model of HVS. 2. Compute avg xy _ ρ as the average value of locally computed correlation coefficients between images x(m,n) and y(m,n). 3. Compute avg xe _ ρ as the average value of the locally computed correlation coefficients between images x(m,n) and e(m,n) given by equation (4). 4. Find the image quality measure Q using equation (6). 5. EXPERIMENTAL RESULTS Performances of the algorithm will be illustrated by several examples using the following parameter values: An Adaptive Image Quality Assessment Algorithm 11 equal image height and width, viewing distance equal to four times image height and image size of 512 x 512 pixels. Parameter f0 in the equation (2) will depend on the original image. Experiment #1: The quality measure performances for the sequence of “Lena” images with various types of filtered noise as discussed in Section 2 are illustrated. For this image, we set f0=5. All three images in the sequence have the same MSE value of 100, but their visual qualities are quite different. The results obtained, using the proposed adaptive image quality assessment algorithm, are shown in Table 1. Table 1. “Lena” Images with Different Filtered Noises Minimal and Maximal Frequency MSE Quality Measure (Q) fmin=0, fmax=0.03 100 0.9120 fmin=0.03, fmax=0.15 100 0.6561 fmin=0.5, fmax=0.707 100 0.9292

EARLIER¶32-36

computed correlation coefficients between images x(m,n) and y(m,n). 3. Compute avg xe _ ρ as the average value of the locally computed correlation coefficients between images x(m,n) and e(m,n) given by equation (4). 4. Find the image quality measure Q using equation (6). 5. EXPERIMENTAL RESULTS Performances of the algorithm will be illustrated by several examples using the following parameter values: An Adaptive Image Quality Assessment Algorithm 11 equal image height and width, viewing distance equal to four times image height and image size of 512 x 512 pixels. Parameter f0 in the equation (2) will depend on the original image. Experiment #1: The quality measure performances for the sequence of “Lena” images with various types of filtered noise as discussed in Section 2 are illustrated. For this image, we set f0=5. All three images in the sequence have the same MSE value of 100, but their visual qualities are quite different. The results obtained, using the proposed adaptive image quality assessment algorithm, are shown in Table 1. Table 1. “Lena” Images with Different Filtered Noises Minimal and Maximal Frequency MSE Quality Measure (Q) fmin=0, fmax=0.03 100 0.9120 fmin=0.03, fmax=0.15 100 0.6561 fmin=0.5, fmax=0.707 100 0.9292 Experiment #2: The quality measure performances for the sequence of “Lena” images with various types of distortions are illustrated. All images have similar MSE values, but their visual qualities are very different. The performance results are given in Table 2.

EARLIER¶32-40

computed correlation coefficients between images x(m,n) and y(m,n). 3. Compute avg xe _ ρ as the average value of the locally computed correlation coefficients between images x(m,n) and e(m,n) given by equation (4). 4. Find the image quality measure Q using equation (6). 5. EXPERIMENTAL RESULTS Performances of the algorithm will be illustrated by several examples using the following parameter values: An Adaptive Image Quality Assessment Algorithm 11 equal image height and width, viewing distance equal to four times image height and image size of 512 x 512 pixels. Parameter f0 in the equation (2) will depend on the original image. Experiment #1: The quality measure performances for the sequence of “Lena” images with various types of filtered noise as discussed in Section 2 are illustrated. For this image, we set f0=5. All three images in the sequence have the same MSE value of 100, but their visual qualities are quite different. The results obtained, using the proposed adaptive image quality assessment algorithm, are shown in Table 1. Table 1. “Lena” Images with Different Filtered Noises Minimal and Maximal Frequency MSE Quality Measure (Q) fmin=0, fmax=0.03 100 0.9120 fmin=0.03, fmax=0.15 100 0.6561 fmin=0.5, fmax=0.707 100 0.9292 Experiment #2: The quality measure performances for the sequence of “Lena” images with various types of distortions are illustrated. All images have similar MSE values, but their visual qualities are very different. The performance results are given in Table 2. Table 2. “Lena” Images with Different Distortions Type of distortion MSE Quality Measure (Q) Contrast stretching 225 0.9766 Additive white Gaussian noise 225 0.6999 Blurring 224 0.2627 JPEG2000 coding 225 0.1898 Experiment #3: The quality measure performances for the sequence of “Lena” images distorted by two sources of degradation due to signal dependent and random distortions, respectively, are illustrated. The sources of degradation considered were blurring caused by low-pass filtering and additive white Gaussian noise (zero-mean and standard deviation ofσ ). The first image contains only blurring, the second image contains some blurring and some additive Gaussian noise and the third image contains only random noise. All images in the sequence have the same MSE. The quality measure performance results are given in Table 3. Table 3. “Lena” Images with Blurring and Additive noise Image MSE Quality Measure (Q) blurring only 420 0.1395 blurring and additive noise (σ = 17.15) 420 0.2668 additive white Gaussian noise (σ = 20.55) 420 0.6273 These wide discrepancies with the subjective quality in the above three experiments illustrate why the MSE or other similar objective measures are not good image quality assessment indicators. The proposed quality measure has another interesting property. As can be seen from equation (6), this quality measure can take both positive and negative values (range

EARLIER¶34-38

equal image height and width, viewing distance equal to four times image height and image size of 512 x 512 pixels. Parameter f0 in the equation (2) will depend on the original image. Experiment #1: The quality measure performances for the sequence of “Lena” images with various types of filtered noise as discussed in Section 2 are illustrated. For this image, we set f0=5. All three images in the sequence have the same MSE value of 100, but their visual qualities are quite different. The results obtained, using the proposed adaptive image quality assessment algorithm, are shown in Table 1. Table 1. “Lena” Images with Different Filtered Noises Minimal and Maximal Frequency MSE Quality Measure (Q) fmin=0, fmax=0.03 100 0.9120 fmin=0.03, fmax=0.15 100 0.6561 fmin=0.5, fmax=0.707 100 0.9292 Experiment #2: The quality measure performances for the sequence of “Lena” images with various types of distortions are illustrated. All images have similar MSE values, but their visual qualities are very different. The performance results are given in Table 2. Table 2. “Lena” Images with Different Distortions Type of distortion MSE Quality Measure (Q) Contrast stretching 225 0.9766 Additive white Gaussian noise 225 0.6999 Blurring 224 0.2627 JPEG2000 coding 225 0.1898 Experiment #3: The quality measure performances for the sequence of “Lena” images distorted by two sources of degradation due to signal dependent and random distortions, respectively, are illustrated. The sources of degradation considered were blurring caused by low-pass filtering and additive white Gaussian noise (zero-mean and standard deviation ofσ ). The first image contains only blurring, the second image contains some blurring and some additive Gaussian noise and the third image contains only random noise. All images in the sequence have the same MSE. The quality measure performance results are given in Table 3.

EARLIER¶32-36

computed correlation coefficients between images x(m,n) and y(m,n). 3. Compute avg xe _ ρ as the average value of the locally computed correlation coefficients between images x(m,n) and e(m,n) given by equation (4). 4. Find the image quality measure Q using equation (6). 5. EXPERIMENTAL RESULTS Performances of the algorithm will be illustrated by several examples using the following parameter values: An Adaptive Image Quality Assessment Algorithm 11 equal image height and width, viewing distance equal to four times image height and image size of 512 x 512 pixels. Parameter f0 in the equation (2) will depend on the original image. Experiment #1: The quality measure performances for the sequence of “Lena” images with various types of filtered noise as discussed in Section 2 are illustrated. For this image, we set f0=5. All three images in the sequence have the same MSE value of 100, but their visual qualities are quite different. The results obtained, using the proposed adaptive image quality assessment algorithm, are shown in Table 1. Table 1. “Lena” Images with Different Filtered Noises Minimal and Maximal Frequency MSE Quality Measure (Q) fmin=0, fmax=0.03 100 0.9120 fmin=0.03, fmax=0.15 100 0.6561 fmin=0.5, fmax=0.707 100 0.9292 Experiment #2: The quality measure performances for the sequence of “Lena” images with various types of distortions are illustrated. All images have similar MSE values, but their visual qualities are very different. The performance results are given in Table 2.

EARLIER¶36-40

Experiment #2: The quality measure performances for the sequence of “Lena” images with various types of distortions are illustrated. All images have similar MSE values, but their visual qualities are very different. The performance results are given in Table 2. Table 2. “Lena” Images with Different Distortions Type of distortion MSE Quality Measure (Q) Contrast stretching 225 0.9766 Additive white Gaussian noise 225 0.6999 Blurring 224 0.2627 JPEG2000 coding 225 0.1898 Experiment #3: The quality measure performances for the sequence of “Lena” images distorted by two sources of degradation due to signal dependent and random distortions, respectively, are illustrated. The sources of degradation considered were blurring caused by low-pass filtering and additive white Gaussian noise (zero-mean and standard deviation ofσ ). The first image contains only blurring, the second image contains some blurring and some additive Gaussian noise and the third image contains only random noise. All images in the sequence have the same MSE. The quality measure performance results are given in Table 3. Table 3. “Lena” Images with Blurring and Additive noise Image MSE Quality Measure (Q) blurring only 420 0.1395 blurring and additive noise (σ = 17.15) 420 0.2668 additive white Gaussian noise (σ = 20.55) 420 0.6273 These wide discrepancies with the subjective quality in the above three experiments illustrate why the MSE or other similar objective measures are not good image quality assessment indicators. The proposed quality measure has another interesting property. As can be seen from equation (6), this quality measure can take both positive and negative values (range

EARLIER¶36-43

Experiment #2: The quality measure performances for the sequence of “Lena” images with various types of distortions are illustrated. All images have similar MSE values, but their visual qualities are very different. The performance results are given in Table 2. Table 2. “Lena” Images with Different Distortions Type of distortion MSE Quality Measure (Q) Contrast stretching 225 0.9766 Additive white Gaussian noise 225 0.6999 Blurring 224 0.2627 JPEG2000 coding 225 0.1898 Experiment #3: The quality measure performances for the sequence of “Lena” images distorted by two sources of degradation due to signal dependent and random distortions, respectively, are illustrated. The sources of degradation considered were blurring caused by low-pass filtering and additive white Gaussian noise (zero-mean and standard deviation ofσ ). The first image contains only blurring, the second image contains some blurring and some additive Gaussian noise and the third image contains only random noise. All images in the sequence have the same MSE. The quality measure performance results are given in Table 3. Table 3. “Lena” Images with Blurring and Additive noise Image MSE Quality Measure (Q) blurring only 420 0.1395 blurring and additive noise (σ = 17.15) 420 0.2668 additive white Gaussian noise (σ = 20.55) 420 0.6273 These wide discrepancies with the subjective quality in the above three experiments illustrate why the MSE or other similar objective measures are not good image quality assessment indicators. The proposed quality measure has another interesting property. As can be seen from equation (6), this quality measure can take both positive and negative values (range of -1 to 1). It takes negative values when avg xy _ ρ is negative. This happens when one of the input images is inverted, which means the bright area becomes dark and vice versa. In this example the reference image is the original “Lena” image and distorted images are inverted original “Lena” image and inverted “Lena” image with additive noise. If the original image is f(m,n), then the corresponding inverted image can be computed as 255-f(m,n). The quality measure performance results are given in Table 4. Quality measure values for inverted images have similar magnitudes as the values obtained for the corresponding non-inverted images, but they have a negative sign. This is in a sense related to what we see: images with the same structure but inverted grayscale. MSE produces only very large values, which is another illustration that MSE is not related to the way we perceive image similarity. Table 4. “Lena” Images with Inverted Grayscale Image MSE Quality measure (Q) Inverted “Lena” image 9258 -0.9955 Inverted “Lena” image with additive noise 9485 -0.6976 Experiment #4: In the last experiment, the algorithm will be applied to the sequence of distorted “Couple” images. All distorted images have same or very close MSE values but their visual qualities are quite different. However, the differences in visual qualities are smaller than in the previous example involving “Lena” images. For this image, parameter f0 is set to 12 cycles/degree because the image has a lot of small objects which attract viewer’s attention, which means that information contained in high frequency components is important. If we used f0=5 (as in the case of “Lena” image) for this image, then it would eliminate this information and distortion would be underestimated. On the other hand, if we used f0=12 for “Lena” image, then it would overestimate high frequency distortion in that case. That is why this parameter is not fixed. Instead its value is set depending on the content of each image. The rule for choice of parameter f0 is as follows: if reference image contains large object(s) then f0 takes lower values; if reference image contains smaller object(s) then f0 takes higher values. Three different images “Tiffany”, “Man” and “Couple” are used to illustrate this point. The associated results are given in Table 5.

EARLIER¶39-43

Table 3. “Lena” Images with Blurring and Additive noise Image MSE Quality Measure (Q) blurring only 420 0.1395 blurring and additive noise (σ = 17.15) 420 0.2668 additive white Gaussian noise (σ = 20.55) 420 0.6273 These wide discrepancies with the subjective quality in the above three experiments illustrate why the MSE or other similar objective measures are not good image quality assessment indicators. The proposed quality measure has another interesting property. As can be seen from equation (6), this quality measure can take both positive and negative values (range of -1 to 1). It takes negative values when avg xy _ ρ is negative. This happens when one of the input images is inverted, which means the bright area becomes dark and vice versa. In this example the reference image is the original “Lena” image and distorted images are inverted original “Lena” image and inverted “Lena” image with additive noise. If the original image is f(m,n), then the corresponding inverted image can be computed as 255-f(m,n). The quality measure performance results are given in Table 4. Quality measure values for inverted images have similar magnitudes as the values obtained for the corresponding non-inverted images, but they have a negative sign. This is in a sense related to what we see: images with the same structure but inverted grayscale. MSE produces only very large values, which is another illustration that MSE is not related to the way we perceive image similarity. Table 4. “Lena” Images with Inverted Grayscale Image MSE Quality measure (Q) Inverted “Lena” image 9258 -0.9955 Inverted “Lena” image with additive noise 9485 -0.6976 Experiment #4: In the last experiment, the algorithm will be applied to the sequence of distorted “Couple” images. All distorted images have same or very close MSE values but their visual qualities are quite different. However, the differences in visual qualities are smaller than in the previous example involving “Lena” images. For this image, parameter f0 is set to 12 cycles/degree because the image has a lot of small objects which attract viewer’s attention, which means that information contained in high frequency components is important. If we used f0=5 (as in the case of “Lena” image) for this image, then it would eliminate this information and distortion would be underestimated. On the other hand, if we used f0=12 for “Lena” image, then it would overestimate high frequency distortion in that case. That is why this parameter is not fixed. Instead its value is set depending on the content of each image. The rule for choice of parameter f0 is as follows: if reference image contains large object(s) then f0 takes lower values; if reference image contains smaller object(s) then f0 takes higher values. Three different images “Tiffany”, “Man” and “Couple” are used to illustrate this point. The associated results are given in Table 5.

EARLIER¶40-45

420 0.2668 additive white Gaussian noise (σ = 20.55) 420 0.6273 These wide discrepancies with the subjective quality in the above three experiments illustrate why the MSE or other similar objective measures are not good image quality assessment indicators. The proposed quality measure has another interesting property. As can be seen from equation (6), this quality measure can take both positive and negative values (range of -1 to 1). It takes negative values when avg xy _ ρ is negative. This happens when one of the input images is inverted, which means the bright area becomes dark and vice versa. In this example the reference image is the original “Lena” image and distorted images are inverted original “Lena” image and inverted “Lena” image with additive noise. If the original image is f(m,n), then the corresponding inverted image can be computed as 255-f(m,n). The quality measure performance results are given in Table 4. Quality measure values for inverted images have similar magnitudes as the values obtained for the corresponding non-inverted images, but they have a negative sign. This is in a sense related to what we see: images with the same structure but inverted grayscale. MSE produces only very large values, which is another illustration that MSE is not related to the way we perceive image similarity. Table 4. “Lena” Images with Inverted Grayscale Image MSE Quality measure (Q) Inverted “Lena” image 9258 -0.9955 Inverted “Lena” image with additive noise 9485 -0.6976 Experiment #4: In the last experiment, the algorithm will be applied to the sequence of distorted “Couple” images. All distorted images have same or very close MSE values but their visual qualities are quite different. However, the differences in visual qualities are smaller than in the previous example involving “Lena” images. For this image, parameter f0 is set to 12 cycles/degree because the image has a lot of small objects which attract viewer’s attention, which means that information contained in high frequency components is important. If we used f0=5 (as in the case of “Lena” image) for this image, then it would eliminate this information and distortion would be underestimated. On the other hand, if we used f0=12 for “Lena” image, then it would overestimate high frequency distortion in that case. That is why this parameter is not fixed. Instead its value is set depending on the content of each image. The rule for choice of parameter f0 is as follows: if reference image contains large object(s) then f0 takes lower values; if reference image contains smaller object(s) then f0 takes higher values. Three different images “Tiffany”, “Man” and “Couple” are used to illustrate this point. The associated results are given in Table 5. 12 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) Table 5. “Couple” Images with Different Distortions Type of Distortion MSE Quality Measure (Q) Contrast stretching 81 0.9893 Additive white Gaussian noise 81 0.8424 Blurring 81 0.7750 JPEG2000 coding 82 0.5419 In these experiments, the proposed image quality measure produces results which are in good agreement with subjective visual quality of corresponding images. In contrast MSE values are the same for images with very different visual quality.

EARLIER¶41-45

of -1 to 1). It takes negative values when avg xy _ ρ is negative. This happens when one of the input images is inverted, which means the bright area becomes dark and vice versa. In this example the reference image is the original “Lena” image and distorted images are inverted original “Lena” image and inverted “Lena” image with additive noise. If the original image is f(m,n), then the corresponding inverted image can be computed as 255-f(m,n). The quality measure performance results are given in Table 4. Quality measure values for inverted images have similar magnitudes as the values obtained for the corresponding non-inverted images, but they have a negative sign. This is in a sense related to what we see: images with the same structure but inverted grayscale. MSE produces only very large values, which is another illustration that MSE is not related to the way we perceive image similarity. Table 4. “Lena” Images with Inverted Grayscale Image MSE Quality measure (Q) Inverted “Lena” image 9258 -0.9955 Inverted “Lena” image with additive noise 9485 -0.6976 Experiment #4: In the last experiment, the algorithm will be applied to the sequence of distorted “Couple” images. All distorted images have same or very close MSE values but their visual qualities are quite different. However, the differences in visual qualities are smaller than in the previous example involving “Lena” images. For this image, parameter f0 is set to 12 cycles/degree because the image has a lot of small objects which attract viewer’s attention, which means that information contained in high frequency components is important. If we used f0=5 (as in the case of “Lena” image) for this image, then it would eliminate this information and distortion would be underestimated. On the other hand, if we used f0=12 for “Lena” image, then it would overestimate high frequency distortion in that case. That is why this parameter is not fixed. Instead its value is set depending on the content of each image. The rule for choice of parameter f0 is as follows: if reference image contains large object(s) then f0 takes lower values; if reference image contains smaller object(s) then f0 takes higher values. Three different images “Tiffany”, “Man” and “Couple” are used to illustrate this point. The associated results are given in Table 5. 12 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) Table 5. “Couple” Images with Different Distortions Type of Distortion MSE Quality Measure (Q) Contrast stretching 81 0.9893 Additive white Gaussian noise 81 0.8424 Blurring 81 0.7750 JPEG2000 coding 82 0.5419 In these experiments, the proposed image quality measure produces results which are in good agreement with subjective visual quality of corresponding images. In contrast MSE values are the same for images with very different visual quality.

EARLIER¶43-46

9485 -0.6976 Experiment #4: In the last experiment, the algorithm will be applied to the sequence of distorted “Couple” images. All distorted images have same or very close MSE values but their visual qualities are quite different. However, the differences in visual qualities are smaller than in the previous example involving “Lena” images. For this image, parameter f0 is set to 12 cycles/degree because the image has a lot of small objects which attract viewer’s attention, which means that information contained in high frequency components is important. If we used f0=5 (as in the case of “Lena” image) for this image, then it would eliminate this information and distortion would be underestimated. On the other hand, if we used f0=12 for “Lena” image, then it would overestimate high frequency distortion in that case. That is why this parameter is not fixed. Instead its value is set depending on the content of each image. The rule for choice of parameter f0 is as follows: if reference image contains large object(s) then f0 takes lower values; if reference image contains smaller object(s) then f0 takes higher values. Three different images “Tiffany”, “Man” and “Couple” are used to illustrate this point. The associated results are given in Table 5. 12 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) Table 5. “Couple” Images with Different Distortions Type of Distortion MSE Quality Measure (Q) Contrast stretching 81 0.9893 Additive white Gaussian noise 81 0.8424 Blurring 81 0.7750 JPEG2000 coding 82 0.5419 In these experiments, the proposed image quality measure produces results which are in good agreement with subjective visual quality of corresponding images. In contrast MSE values are the same for images with very different visual quality. 5. CONCLUSION In this paper, a simple image quality assessment algorithm that adapts to various image scenarios by a user-controlled parameter has been presented. First, three main reasons for disagreement between MSE values and subjective visual quality of images have been identified. Then a new quality measure has been introduced. The proposed measure takes into account two HVS properties: nonlinear relationship between intensity and perceived brightness and the presence of spatial filtering in HVS. This measure is based on average value of locally computed correlation coefficients, which is more closely related to the way in which human observer determines quality of an image than a traditional objective quality measure such as MSE. Finally, this value is modified by the average value of locally computed correlation coefficients between the original and error images. This way the proposed measure differentiates between random and signal-dependant distortions, which have different effects on human observer. The proposed image quality measure performs reasonably well. The experimental performance results presented here demonstrate that this measure ranks images according to their visual quality in cases when MSE-based measures fail to do that. However, subjective evaluation is still the best way for image quality assessment. HVS is more sophisticated than any mathematically defined image quality measure that we try hard to mimic.

EARLIER¶44-46

12 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) Table 5. “Couple” Images with Different Distortions Type of Distortion MSE Quality Measure (Q) Contrast stretching 81 0.9893 Additive white Gaussian noise 81 0.8424 Blurring 81 0.7750 JPEG2000 coding 82 0.5419 In these experiments, the proposed image quality measure produces results which are in good agreement with subjective visual quality of corresponding images. In contrast MSE values are the same for images with very different visual quality. 5. CONCLUSION In this paper, a simple image quality assessment algorithm that adapts to various image scenarios by a user-controlled parameter has been presented. First, three main reasons for disagreement between MSE values and subjective visual quality of images have been identified. Then a new quality measure has been introduced. The proposed measure takes into account two HVS properties: nonlinear relationship between intensity and perceived brightness and the presence of spatial filtering in HVS. This measure is based on average value of locally computed correlation coefficients, which is more closely related to the way in which human observer determines quality of an image than a traditional objective quality measure such as MSE. Finally, this value is modified by the average value of locally computed correlation coefficients between the original and error images. This way the proposed measure differentiates between random and signal-dependant distortions, which have different effects on human observer. The proposed image quality measure performs reasonably well. The experimental performance results presented here demonstrate that this measure ranks images according to their visual quality in cases when MSE-based measures fail to do that. However, subjective evaluation is still the best way for image quality assessment. HVS is more sophisticated than any mathematically defined image quality measure that we try hard to mimic.

EARLIER¶7-11

value of locally computed values. Most other approaches try to modify MSE by modeling properties of HVS. This approach defines different criterion which is better related to the way in which human observer assesses image quality. No HVS model is used here. The UQI produced unstable results and to avoid this problem, the measure has been generalized to the structural similarity metric (SSIM) that captures the loss of image structure for quality assessment [12]. As in the case of UQI, the overall quality (Mean SSIM) is obtained by computing the average of the locally computed values in a sliding window. A visual information fidelity criterion that is similar to HVS-based measures that relates visual quality to relative image information has been described in [13]. This is a very complex measure requiring a source model, a distortion model, and a HVS model. In [14], a completely new measure based on singular value decomposition that expressed image quality graphically or numerically was presented. The approach presented here will use the basic idea from [10-12], but it will not use image quality measure as it is defined there (only correlation coefficient will be used). In the first step, the reference and distorted images are processed by a simple model of HVS. It consists of a nonlinear function modeling the brightness perception and a 2-D filter modeling the frequency response of HVS (contrast sensitivity function). The 2-D filter used in this HVS model is not fixed. It contains one user-defined parameter which can be changed depending on the content of the reference image. After this, the correlation coefficient is computed on a block-by-block basis for the processed input images. Finally, the quality measure is computed as the average correlation coefficient between the reference image and distorted image, adjusted by the average correlation coefficient between the reference image and error image. This is a simpler model to those described in [10-14]. The rest of this paper is organized as follows. Problems with MSE-based image quality measures are discussed in Section 2. HVS model used to process input images is described in Section 3. Proposed image quality measure is defined next in Section 4. Performances of the proposed measure are illustrated by examples involving images with different types of distortions in Section 5, followed by the conclusion. 2. WHY MSE IS NOT A GOOD MEASURE? MSE and MSE-based measures such as PSNR and SNR are widely used to assess performances of image enhancement and coding algorithms. MSE-based measures work well when images with the same type of degradation are compared. In a sequence of images obtained by distorting an image by various amounts of the same type of degradation, the image with the smallest MSE will be perceived by a human observer to be the closest to the original image. However, when images with different types of degradation are compared, the image with the smallest MSE will not always be perceived to be the closest to the original image. It is possible to create images with the same 8 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) MSE value but with totally different visual quality. There are three main reasons for disagreement between MSE values and subjective evaluation. First reason for this is presence of spatial filtering in HVS. This can be demonstrated by adding a white Gaussian filtered (2-D FIR filter) noise to the original Lena image for various values of filter cut-off frequencies, fmin and fmax. Each time the filtered noise is scaled so that its standard deviation is 10. The resulting images shown in Figures 2-4 all have MSE values of 100, however, their visual qualities are quite different. Noise at very low and very high frequencies are less noticeable than noise at mid frequencies. This indicates band pass characteristic nature of HVS. One way to solve this problem is by using weighted MSE, where reference and distorted images are filtered by contrast sensitivity function (CSF) and then MSE is computed using filtered images [15]. Second reason for disagreement between MSE values and subjective assessment is in the criterion itself, i.e., MSE treats image intensities as a set of uncorrelated numbers [10-12]. One pixel is never seen in isolation from the surrounding pixels and they always create some structure in an image. This information about the structure created by a group of pixels is more important for a human observer than intensity at some spatial location. However, this is completely ignored by MSE and its computation is not quite related to the way human observers perceive image quality. The images distorted by a wide variety of corruptions such as impulsive salt-pepper noise, additive Gaussian noise, multiplicative speckle noise, contrast stretching, blurring, and JPEG compression yield the same MSE value relative to the original image but very different visual quality. This is illustrated with examples using Lena image in [10-11]. This problem cannot be solved by some modification of MSE, such as weighted MSE and it is necessary to define a different image quality measure. In this paper, we propose a new quality measure, Q and it is described later in Section 4. Third reason for low correlation between MSE values and subjective visual quality is due to the fact that MSE does not differentiate between random and signal dependent distortions. Random noise such as additive white Gaussian noise is much less objectionable for a human viewer than signal dependant distortion such as blurring. The results of these experiments are presented in Tables I-III in Section 5. 3. HVS MODEL Image quality measure developed here will use a simple model of HVS consisting of a nonlinear function, modeling brightness perception of HVS and a 2-D filter, modeling frequency response of HVS. This type of HVS model was used in [15] and [16]. However, nonlinear function and 2-D filter used here are not the same as those developed in [15]. Digital images are represented using a finite number of intensity levels. For example, gray scale images with 8 bits/pixel are represented by 256 intensity levels, where 0 corresponds to the darkest level and 255 corresponds to the brightest level. Intensities between these two extremes represent various shades of gray level. Perceived brightness is a nonlinear function of intensity and this is modeled by transforming input intensities by some monotonically increasing nonlinear function. Various functions have been suggested to model this. One frequently cited result states that brightness perceived by HVS is proportional to the logarithm of intensity as it is known as Weber’s law [8]. HVS model developed in [15] uses cube root function to model this effect. Some use brightness perception models developed for specific display device and an example of this can be found in [6]. Logarithmic or cube root functions are not accurate models for brightness perception by HVS especially at low intensities. According to them the sensitivity to intensity increases is greatest at low intensities and decreases at mid and high intensities. This is clearly wrong, because an increase in intensity result in a very small increase in perceived brightness, when the intensity is low. The following nonlinear function will be used here instead [17].

EARLIER¶44-46

12 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) Table 5. “Couple” Images with Different Distortions Type of Distortion MSE Quality Measure (Q) Contrast stretching 81 0.9893 Additive white Gaussian noise 81 0.8424 Blurring 81 0.7750 JPEG2000 coding 82 0.5419 In these experiments, the proposed image quality measure produces results which are in good agreement with subjective visual quality of corresponding images. In contrast MSE values are the same for images with very different visual quality. 5. CONCLUSION In this paper, a simple image quality assessment algorithm that adapts to various image scenarios by a user-controlled parameter has been presented. First, three main reasons for disagreement between MSE values and subjective visual quality of images have been identified. Then a new quality measure has been introduced. The proposed measure takes into account two HVS properties: nonlinear relationship between intensity and perceived brightness and the presence of spatial filtering in HVS. This measure is based on average value of locally computed correlation coefficients, which is more closely related to the way in which human observer determines quality of an image than a traditional objective quality measure such as MSE. Finally, this value is modified by the average value of locally computed correlation coefficients between the original and error images. This way the proposed measure differentiates between random and signal-dependant distortions, which have different effects on human observer. The proposed image quality measure performs reasonably well. The experimental performance results presented here demonstrate that this measure ranks images according to their visual quality in cases when MSE-based measures fail to do that. However, subjective evaluation is still the best way for image quality assessment. HVS is more sophisticated than any mathematically defined image quality measure that we try hard to mimic.

EARLIER¶0-3

Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) DOI 10.7236/JASC2012.1.1.2 An Adaptive Image Quality Assessment Algorithm Ravi Sankar* and Goran Ivkovic iCONS Research Group, Dept. of Electrical Engineering, University of South Florida, USA Abstract An improved algorithm for image quality assessment is presented. First a simple model of human visual system, consisting of a nonlinear function and a 2-D filter, processes the input images. This filter has one user-defined parameter, whose value depends on the reference image. This way the algorithm can adapt to different scenarios. In the next step the average value of locally computed correlation coefficients between the two processed images is found. This criterion is closely related to the way in which human observer assesses image quality. Finally, image quality measure is computed as the average value of locally computed correlation coefficients, adjusted by the average correlation coefficient between the reference and error images. By this approach the proposed measure differentiates between the random and signal dependant distortions, which have different effects on human observer. Performance of the proposed quality measure is illustrated by examples involving images with different types of degradation. Key words: Brightness perception, Contrast sensitivity function, Human visual system, Image quality measure 1. INTRODUCTION Image and video quality assessment methods can be divided into: subjective and objective. The only meaningful subjective (qualitative) measure is mean opinion score (MOS), where quality of an image is rated on some scale by several individuals according to the level of distortion and the mean value of their scores is used as image quality measure. Since human observer is the ultimate receiver of the information contained in an image, this is the best way for image quality assessment. In the case of video, several subjective test methods such as Single Stimulus Continuous Quality Evaluation (SSCQE) and Quality-Oriented Adaptation Scheme (QOAS) have been described in [1-3] to obtain MOS. Objective (quantitative) measures use intensities from the distorted input image and the reference image to compute a value that indicates image quality. The goal is to find a good measure which produces results which are in good agreement with subjective evaluation. Most widely used objective measures are mean squared error (MSE) and MSE-based measures: peak signal to noise ratio (PSNR) and signal to noise ratio (SNR). These simple measures work well when images with the same type of distortion or degradation are compared. In this case distorted image with smaller MSE will be perceived closer to the original image than the one with greater MSE. However, when images with different types of distortion (additive white Gaussian noise, blurring. contrast stretching, JPEG2000 coding/compression) are compared MSE does not produce results that correlate well with subjective quality assessment. Images with different types of distortions with the same MSE values can have very different subjective visual qualities. One role of these measures is to assess performance of algorithms for image enhancement and coding. Another role is to optimize image processing algorithms. For example, in image coding the goal is to represent images with as few bits as possible with the minimal loss of image quality. Quality measure is used as a criterion in a sense that the bits that contribute more significantly to the used quality measure are coded first. The objective video quality measures have been described recently in [4]. Examples of digital video distortions including tiling, smearing, jerkiness, edge busyness, and object retention that are produced by video processing systems are fundamentally different kinds of impairments than those produced by image processing systems. So both subjective and objective quality assessments are quite different for images and videos. In this paper we focus our attention only on image quality monitoring and assessment.

EARLIER¶44-46

12 Journal of Advanced Smart Convergence Vol.1 No.1 6-13 (2012) Table 5. “Couple” Images with Different Distortions Type of Distortion MSE Quality Measure (Q) Contrast stretching 81 0.9893 Additive white Gaussian noise 81 0.8424 Blurring 81 0.7750 JPEG2000 coding 82 0.5419 In these experiments, the proposed image quality measure produces results which are in good agreement with subjective visual quality of corresponding images. In contrast MSE values are the same for images with very different visual quality. 5. CONCLUSION In this paper, a simple image quality assessment algorithm that adapts to various image scenarios by a user-controlled parameter has been presented. First, three main reasons for disagreement between MSE values and subjective visual quality of images have been identified. Then a new quality measure has been introduced. The proposed measure takes into account two HVS properties: nonlinear relationship between intensity and perceived brightness and the presence of spatial filtering in HVS. This measure is based on average value of locally computed correlation coefficients, which is more closely related to the way in which human observer determines quality of an image than a traditional objective quality measure such as MSE. Finally, this value is modified by the average value of locally computed correlation coefficients between the original and error images. This way the proposed measure differentiates between random and signal-dependant distortions, which have different effects on human observer. The proposed image quality measure performs reasonably well. The experimental performance results presented here demonstrate that this measure ranks images according to their visual quality in cases when MSE-based measures fail to do that. However, subjective evaluation is still the best way for image quality assessment. HVS is more sophisticated than any mathematically defined image quality measure that we try hard to mimic.