A reduced-reference perceptual image and video quality metric based on edge preservation
© Martini et al; licensee Springer. 2012
Received: 16 May 2011
Accepted: 16 March 2012
Published: 16 March 2012
Skip to main content
© Martini et al; licensee Springer. 2012
Received: 16 May 2011
Accepted: 16 March 2012
Published: 16 March 2012
In image and video compression and transmission, it is important to rely on an objective image/video quality metric which accurately represents the subjective quality of processed images and video sequences. In some scenarios, it is also important to evaluate the quality of the received video sequence with minimal reference to the transmitted one. For instance, for quality improvement of video transmission through closed-loop optimisation, the video quality measure can be evaluated at the receiver and provided as feedback information to the system controller. The original image/video sequence--prior to compression and transmission--is not usually available at the receiver side, and it is important to rely at the receiver side on an objective video quality metric that does not need reference or needs minimal reference to the original video sequence. The observation that the human eye is very sensitive to edge and contour information of an image underpins the proposal of our reduced reference (RR) quality metric, which compares edge information between the distorted and the original image. Results highlight that the metric correlates well with subjective observations, also in comparison with commonly used full-reference metrics and with a state-of-the-art RR metric.
For recent and emerging multimedia systems and applications, such as modern video broadcasting systems (including DVB/DVB-H, IPTV, webTV, HDTV,...) and telemedical applications, user requirements are going beyond requirements on connectivity, and users now expect the services to meet their requirements on quality. In recent years, the concept of quality of service (QoS) has been augmented towards the new concept of quality of experience (QoE), as the first only focuses on the network performance (e.g., packet loss, delay, and jitter) without a direct link to the perceived quality, whereas the QoE reflects the overall experience of the consumer accessing and using the provided service. The main target in the design of modern multimedia systems is thus the improvement of the (video) quality perceived by the user. For the provision of such quality improvement the availability of an objective quality metric well representing the human perception is crucial. Objective quality assessment methods based on subjective measurements are based either on a perceptual model of the human visual system (HVS) , or on a combination of relevant parameters tuned with subjective tests [2, 3].
We propose here a RR video quality metric well correlated with the perceived quality, based on the comparison of the edge information between the distorted image and the original one. The human eye is in fact very sensitive to the edge and contour information of an image, i.e., the edge and contour information gives a good indication of the structure of an image and it is critical for a human to capture the scene .
Some works in the literature proposed considering edge structure information. For instance in  the structural information error between the reference and the distorted image is computed based on the statistics of the spatial position error of the local modulus maxima in the wavelet domain. In  a parameter is considered to detect a decrease or loss of spatial information (e.g., blurring). This parameter uses a 13 pixel spatial information filter (SI13) to measure edge impairments rather than Sobel filtering. Differently from  we consider here the Sobel operator  for edge detection, since this is one of the most used methodologies to obtain edge information due to its simplicity and efficiency. Further details on this choice are reported in the following section.
A few RR metrics have been proposed, with different characteristics in terms of complexity, of correlation with subjective quality and of overhead associated to the transmission of side information.
The ITS/NTIA (Institute for Telecommunication Sciences/National Telecommunications and Information Administration) has developed a general video quality model (VQM)  that was selected by both ANSI and ITU as a video quality assessment standard based on its performance. This general model requires however a bit-rate of several Mbps (more than 4 Mbps for 30 fps, CIF size video) of quality features for the calculation of the VQM value, which prevents its use as a RR metric in practical systems. The possibility to use spatial-temporal features/regions was considered in  in order to provide a trade-off between the correlation with subjective values and the overhead for side-information. Later on a low-rate RR metric based on the full reference metric  ("10 kbits/s VQM") was developed by the same authors. A subjective data set was used to determine the optimal linear combination of the eight video quality parameters in the metric. The performance of the metric was presented in terms of a scatter plot with respect to subjective data, although numerical performance results are not provided in .
The quality index in  is based on features which describe the histograms of wavelet coefficients. Two parameters describe the distribution of the wavelet coefficients of the reference image using a generalized Gaussian density (GGD) model, hence only a relatively small number of RR features are needed for the evaluation of image quality.
The RR objective picture quality measurement tool of compressed video in  is based on a discriminative analysis of harmonic strength computed from edge-detected pictures to create harmonics gain and loss information that could be associated with the picture. The results achieved are compared by the authors with a VQEG RR metric [9, 12] and the performance of the proposed metric is shown to be comparable to the latter, with a reduction in overhead with respect to it and a global reduction of overhead with respect to full reference metrics of 1024:1. The focus is on the detection of blocking and blurring artifacts. This metric considers edge detection as our proposed metric, but in  edge detection is performed over the whole image and edge information is not used as side information, but just as a step for further processing of the image for the extraction of different side information.
The quality criterion presented in  presents relies on the extraction, from an image represented in a perceptual space, of visual features that can be compared to those used by the HVS (perceptual color space, CSF, psychophysical subband decomposition, masking effect modeling). Then a sim-ilarity metric computes the objective quality score of a distorted image by comparing the features extracted from this image to features extracted from its reference image. The performance is evaluated with the aid of three different databases with respect to three full reference metrics. The size of the side information is flexible. The main drawback of this metric is its complexity, since the HVS model (which is an essential part of the proposed image quality criterion) requires a high computation complexity.
In  an RR objective perceptual image quality metric for use in wireless imaging is proposed. Specifically, the normalized hybrid image quality metric (NHIQM) and a perceptual relevance weighted Lp-norm are designed, based on the observation that the HVS is trained to extract structural information from the viewing area. Image features are identified and measured based on the extent by which individual artifacts are present in a given image. The overall quality measure is then computed as a weighted sum of the features. The authors did not rely on public databases for performance evaluation, but performed their own subjective tests. The performance of this metric is evaluated with respect to full reference metrics and the metric in .
The metric in  is based on a divisive normalization image representation. No assumptions are made about the type of impairment. This metric requires training: before applying the proposed algorithm for image quality assessment, five parameters need to be learned from the data. These parameters are cross-validated with different selections of the training and testing data. Results are compared with the RR metric in  and with peak signal-to-noise ratio (PSNR).
In this article we propose a low complexity RR metric based on edge preservation which can be calculated in real time in practical image/video processing and transmission systems, performs comparably with the mostly used full reference metrics and requires a limited overhead for the transmission of side information.
The remainder of this article is organized as follows. Edge detection methodologies are introduced in Section 2. Section 3 presents the proposed RR image and video quality metric. Simulation set-up and results are reported in Section 4. Conclusions about the novelty and performance of the metric are then reported in Section 5.
There are many methods to perform edge detection. The majority of these may be grouped into two categories: gradient and Laplacian. The gradient method detects the edges by finding the maximum and minimum in the first derivative of the image. This method is characteristic of the gradient filter family of edge detection and includes the Sobel method. A pixel location is declared an edge location if the value of the gradient exceeds a threshold. Edges will have higher pixel intensity values than those surrounding it. Once a threshold is set, the gradient value can be compared to the threshold value and an edge is detected when the threshold is exceeded. When the first derivative is at a maximum, the second derivative is zero. As a result, an alternative to finding the location of an edge is to locate the zeros in the second derivative. This method is known as the Laplacian.
The aforementioned methods can be extended to the 2D case. The Sobel operator performs a 2D spatial gradient measurement on an image. Typically it is used to find the approximate absolute gradient magnitude at each point in an input grayscale image. The Sobel edge detector uses a pair of 3 × 3 convolution masks, one estimating the gradient in the x-direction (columns) and the other estimating the gradient in the y-direction (rows). The mask is then slid over the image, manipulating a square block of pixels at a time.
The Sobel operator can detect edges by calculating partial derivatives in 3 × 3 neighborhood. The main reason for using the Sobel operator is that it is relatively insensitive to noise and it has relatively smaller masks than other operators such as the Roberts operator and the two-order Laplacian operator.
The gradient of each pixel is calculated according to and a threshold value t is selected. If g(x, y) > t, this point is regarded as an edge point.
Since structural distortion is tightly linked with edge degradation, we propose a RR quality metric which compares edge information between the distorted image and the original one. We propose to apply Sobel filtering locally, only for some blocks of the entire image, after subsampling the images.
In order to reduce the overhead associated with the transmission of side information, only 12 blocks are selected to represent the different areas of the images. The block pattern utilized for our tests is chosen after several investigations based on visual attention (VA). Various experiments have been proposed in the literature for VA modeling and salient region identification, aiming at the detection of salient regions in an image. Models on VA are often developed and validated by visual fixation patterns through eye tracking experiments [16, 17]. In  a framework is proposed in order to extend existing image quality metrics with a simple VA model. A subjective region of interest (ROI) experiment was performed, with seven images, in which the viewers' task was to select within each image the region that drew most of their attention. For simplicity, in this experiment only rectangular-shaped ROIs were allowed. Considering the obtained ROI as a random value, it is possible to calculate the mean value and the standard deviation. It was observed that the ROI's center coordinates are around the image center for most of the images, and the mean of the ROI dimensions are very similar in both x and y directions. This confirms that the salient region, which include the most important informative content of the image, is often placed in the center of the picture.
Following these guidelines we have chosen the block pattern as a subset of the ROI with a central symmetry, minimizing the number of blocks to reduce the overhead associated to the transmission of side information. Figure 3 shows an example of block pattern.
For the assessment of the quality of the corrupted image, the edge structure of the blocks of the corrupted image should be compared to the structure of the correspondent blocks in the original image. For the identification of edges we use Sobel filtering, which is applied locally in these selected blocks.
For each pixel in each block we obtain a bit value, where one represents an edge and zero means that there are no edges. If m and n are the block dimensions, we denote the corresponding blocks l in the original and the possibly corrupted image as the m × n matrices O l and C l respectively, and the Sobel-filtered version of blocks l as the m × n binary matrices , with elements so i, j , with i = 1,..., m, j = 1, ..., n, and , with elements sc i, j , with i = 1, ..., m, j = 1, ..., n. We denoted above with the Sobel operator. The similarity of two images can be assessed based on the similarity of the edge structures, i.e., by comparing the matrices SO l , associated to the filtered version of the block in the original image, and SC l , associated to the filtered version of the block in the possibly corrupted image.
is the number of zeros and ones unchanged in the l-th block and p l = m × n is the total number of pixels in the l-th block.
The threshold value is an important parameter that depends on a number of factors, such as image brightness, contrast, level of noise, and even edge direction. The selection of the threshold in Sobel filtering is associated to the sensitivity of the filter to edges. In particular, the lower the value of the threshold, the higher the sensitivity to edges. Too high values of the threshold do not detect edges which are important for quality assessment. On the other side, if the value of the threshold is too small, large parts of the image are considered as edges, whereas these are irrelevant for quality assessment. The threshold can be selected following an analysis of the gradient image histogram. Based on this consideration and on the analysis of Sobel filtering performance for the images of the considered databases, the selected threshold value is t = 0.001.
The selection of Sobel filtering results in a low complexity metric. The Sobel algorithm is characterized, in fact, by a low computational complexity and consequently high calculation speed. In  some edge detection techniques are compared for an application which uses a DSP implementation: the Sobel filter exhibits the best performance in terms of edge detection time in comparison with the other wavelet-based edge detectors. Sobel filtering has been implemented in hardware and used in different areas, often when realtime performance is required, such as for real-time volume rendering systems, and video assisted transportation systems [21, 22]. This makes the proposed metric suitable for real-time implementation, an important aspect when an image/video metric is used for the purpose of "on the fly" system adaptation as in the scenario considered here.
In order to perform the proposed edge comparison, we should transmit the matrices composed of one's and zeros's in the reference blocks. By considering the pattern in Figure 3, this would result for images of resolution 512 × 768 in the transmission of 19 × 32 × 12 = 7.29 kbits per image. Note that the size of the original image (not compressed) is 3 × 512 × 768 × 8 = 9.4 Mbits.
In the worst case (side information not compressed) our metric reduces thus the needed reference with respect to FR metrics of a factor 1290:1. As a comparison, the RR metric in  reduces it of a factor 1024:1 and the metric in  of 64:1.
Since side information is in our case composed of a large number of zeros appearing in long runs, it is possible to further reduce the overhead by compressing the relevant data, e.g., through run-length encoding, or to transmit only the positions of ones in the matrix.
Furthermore, in the case of video, quality assessment can be performed only on a fraction of the transmitted frames (e.g., five frames per second) in order to reduce the side information overhead needed for the calculation of the quality metric.
In order to test the performance of our quality assessment algorithm, we considered publicly available databases.
With the aid of the databases above, we compare the performance versus subjective tests of our metric with respect to the most popular full reference metrics and to the RR metrics with the best performance and whose results are directly comparable or reproducible.
To apply the MSSIM metric, the images have been modified according to .
The figures report, besides scatter plots, the linear approximation best fitting the data using the least-squares method, the residuals and the norm of residuals L for the linear model, i.e., , where the residual d i is the difference between the predicted quality value and the experimental subjective quality value for image i, and N is the number of the considered images. The values of the norms of residuals enable a simple numerical comparison among the different metrics. Note that in the case of the MSSIM metric we have provided a non-linear approximation, better fitting the data.
Norm of residuals versus DMOS, LIVE image database 
Correlation coefficient versus DMOS, LIVE image database 
Spearman rank versus DMOS, LIVE image database 
We can observe that our metric well correlates with subjective tests, with results comparable to those achieved by full reference metrics. For the images in the LIVE database our metric outperforms the considered state-of-the-art RR metrics in all the considered scenarios, except for the case of WN, where the metric  performs better at the expense of a higher complexity, and the case of JPEG2000 where the benchmark RR metric , based on the wavelet transform, provides a better performance in terms of norm of residuals.
Norm of residuals versus MOS, IRCCyN/IVC image database 
Table 3 reports results in terms of Spearman rank, an indicator of monotonicity, for the LIVE image database. With this criterion, our metric outperforms the full reference PSNR metric for all impairments except Gaussian noise, and the RR metric in  for all the reported cases except the case of fast fading. The more complex RR metric in  is outperformed in the case of GB.
Proposed RR metric
Note that for video sequences, in order to reduce the overhead, it is possible to apply the metric only for selected frames, for instance by every 5, 10, 25, and 50 frames. The necessity of a more or less frequent calculation of the metric depends on the motion characteristics of the video sequence.
We can observe that the performance of our metric is comparable with the considered full reference metrics, and our metric outperforms PSNR in the case of both MPEG2 and H.264 compression and also in the case "IP distortion", i.e., the case of H.264 video transmitted over a network. Our metric outperforms also the MSSIM metric in terms of correlation coefficient with subjective data for the case of MPEG2 compressed video.
where the notation used is defined in Section 3, and Ntot is the total number of blocks in the image.
Norm of residuals versus DMOS, full reference versus RR edge-based metric, LIVE image database
We proposed in this article a perceptual RR image and video quality metric which compares edge information between portions of the distorted image and the original one by using Sobel filtering. The algorithms is simple and has a low computational complexity. Results highlight that the proposed metric well correlates with subjective observations, also in comparison with commonly used full-reference metrics and with state-of-the-art RR metrics.
This work was partially supported by the European Commission (FP7 projects OPTIMIX and CONCERTO).
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.