A novel method in adaptive image enlargement

Abstract

This article introduces a new adaptive method for image interpolation. In order to obtain a high resolution (HR) image from its low resolution (LR) counterpart (original image), an interpolator function (array) is used, and the main focus of this manuscript is to formulate and define this function. By applying this interpolator function to each row and column of a LR image, it is possible to construct its HR counterpart. One of the main challenges of image interpolation algorithms is to maintain the edge structures while developing an HR image from the LR replica. The proposed approach overcomes this challenge and exhibits remarkable results at the image edges. The peak signal to noise ratio and structural similarity criteria by using this innovative technique are notably better than those achieved by alternative schemes. Also, in terms of implementation speed, this method displays a clear advantage and outperforms the high performance algorithms in the ability to decrease the artifact results of image enlargement such as blurring and zigzagging.

Introduction

Image interpolation in this article refers to algorithms that transform a low resolution (LR) image to high resolution (HR) one and has many applications. In medical applications, it is highly preferred for images to have high resolution while medical equipments cannot produce images with resolutions higher than a specific standard. In another application, with the development of digital monitors, we need to transform Standard Definition TeleVision (SDTV) video frames to High Definition TeleVision (HDTV) equivalents. Another application of this technology is to find missing pixels or blocks in an image.

Traditional methods such as bilinear, bicubic, and cubic convolution [1, 2] are the common methods in image enlargement. Although the merits of these techniques are simplicity and fast implementation, they suffer from visual degradations such as jagged edges, blurring, and ringing around the image edges.

Human visual system is sensitive to the edges of the objects in an image. As a result, there are several mechanisms which improve the edges of an image. Undoubtedly, as the quality of image enhances, the computational complexity increases. Hence, in all of these methods, there is a trade off between performance and affordable complexity. In the last few decades, several authors have carried out researches on this topic. Along the same line, we briefly introduce [3–10] as follows.

In NEDI [3], the aim is to maintain the edge structure in HR images. This is a nonlinear algorithm whose basic idea is to estimate the covariance of HR image from the covariance of LR counterpart. The estimation is based on the geometric duality between low and high resolution. The algorithm presented in [4] which is based on directional filtering and data fusion to find the missing pixel, considers two orthogonal sets which produce an estimation of the pixel value. In SAI [5], using a moving window in the input LR image, one can find the model parameters. The pixel structure obtained in this way can be led to a block of both available and estimated pixels using a soft-decision estimation process. In WZP-CS [6], wavelet transformation is applied to LR image. Using sub-bands of the resultant image, one can estimate sub-bands of HR image. Afterwards, the HR image can be obtained by applying the inverse wavelet transform. In Optimized spline [7], by taking advantages of calculus of variations, the optimization problem is simplified from a nonlinear infinite dimensional case to a linear finite dimensional one in order to design compact-support interpolation kernels for a class of signals. The method presented in [8] tries to create up-scaled images that are suitable for real time applications and appear natural to human observer. In this algorithm a two step grid filling is used. Then, an iterative correction is obtained by minimizing an objective function depending on the second order directional derivatives of the image intensity.

In [9], in order to enlarge the image, it is down-sampled by using the bilinear method and then, the missing pixels are estimated by using a combination of the other pixels, while the desired coefficients are optimized by using a least mean square technique. In [10], the authors have extended the algorithm presented in [9] to display products such as video restoration.

In our approach, we work out the parameters that are used to enlarge an image from its LR counterpart. These parameters depend on the original image and the factor which we want to enlarge the image by. In order to enlarge an image or find the missing pixels in it, an interpolation array which is constructed by the mentioned parameters is convolved by the matrix that contains the values of the image pixels.

The rest of the article is organized as follows: The following section describes the proposed interpolation method for one-dimensional vectors. The two-dimensional application is proposed in section “Two-dimensional algorithm”. Simulation results and comparisons are given in section “Simulation results” , and section “Conclusion” concludes the paper.

Algorithm description

In order to describe the proposed algorithm for image interpolation, we first apply it to one-dimensional vectors. Let _{X LR} be a 1×N vector. Suppose we wish to enlarge _{X LR} to a kN sized vector _{X HR} using the following steps: We put k−1 zeros between any two successive entries of vector _{X LR} . In order to achieve these kN−N new entries of _{X HR} , we choose 2M of N entries of _{X LR} . The new value will be a linear combination of these 2M entries.

In order to find the 2M indeterminate multipliers of this linear combination, we should follow the procedure stated below, bearing in mind that for each fixed vector of any size the procedure and the multipliers applied to enlarge the vector are fixed. This means that, the multipliers used for obtaining a 1×N vector from its $1\times \left[\frac{N}{k}\right]$ counterpart are the same as the ones used to calculate a $1\times \left[\frac{N}{k}\right]$ vector from its $1\times \left[\frac{N}{k^2}\right]$ counterpart.

Step(1-1): As presented in (1) and (2), we down-sample the original vector _{X LR} by a factor of k to find $X_{1\times \left[\frac{N}{k}\right]}^{\prime }$.

Step(1-2): Again we down-sample $X_{1\times \left[\frac{N}{k}\right]}^{″}$ by a factor of k to obtain $X_{1\times \left[\frac{N}{k^2}\right]}^{\prime }$ vector.

Step(1-3): Now, by constructing $1\times \left[\frac{N}{k}\right]$ vector $X^{\prime }$ from vector $X^{″}$ of size $1\times \left[\frac{N}{k^2}\right]$, we can find the multipliers used to obtain the enlarged image. To approach this, we must zero-pad $X^{″}$ by a factor of k to build vector $X_{\mathrm{ZP}}^{″}$. This means that, k−1 zeros are inserted between every two successive entries of $X^{″}$.

As it was stated earlier, by using these multipliers, we can find a new vector _{X HR} from $X_{1\times \left[\frac{N}{k}\right]}^{\prime }$.

In order to evaluate the enlarged image, we take advantages of the criteria peak signal to noise ratio (PSNR) and Structural SIMilarity (SSIM) [11] by down-sampling the original vector by a factor of k and interpolating it using the proposed algorithm.

$$X=\left(\begin{array}{llll}{x}_1& x_2& \dots & x_N\end{array}\right)$$

(1)

down-sample by a factor of k ↓

$$X^{″}=\left(\begin{array}{lllll}{x}_1& x_{k+1}& x_{2k+1}& \dots & x_{\left(\right[\frac{N}{k}]-1)k+1}\end{array}\right)$$

(2)

down-sample by a factor of k ↓

$$X^{″}=\left(\begin{array}{lllll}{x}_1^{\prime }& x_{k+1}^{\prime }& x_{2k+1}^{\prime }& \dots & x_{\left(\right[\frac{N}{k^2}]-1)k+1}^{\prime }\end{array}\right)$$

(3)

Zero padding ↓

$$X_{\mathrm{ZP}}^{″}=\left(\begin{array}{llllllll}{x}_1^{″}& \underset{(k-1)\text{times}}{\underbrace{0\dots 0}}& x_2^{″}& \underset{(k-1)\text{times}}{\underbrace{0\dots 0}}& x_3^{″}& \dots & 0& x_{\left[\frac{N}{k^2}\right]}^{″}\end{array}\right)$$

(4)

Step(1-4): In order to compute vector $X^{″}$ from $X_{\mathrm{ZP}}^{\mathrm{″″}}$, we must insert a linear combination of $x_i^{\mathrm{″″}}$’s; $i=1:\left[\frac{N}{k^2}\right]$ instead of each zero in $X_{\mathrm{ZP}}^{\mathrm{″″}}$. For this purpose, we can convolve $X_{\mathrm{ZP}}^{\mathrm{″″}}$ with an interpolator vector, A (which is determined by (5)).

In this equation, A is a 1×(1 + 2(M−1) + 2M(k−1)) vector. The 1 + (M−1) + M(k−1)^st entry, which is located at the middle of the vector, is 1. Except for the mentioned entry, the entries with the indices that are multiples of k are 0, and there are (k−1)aⁱ _″ s and (k−1) _{a i} s between any two successive zeros at the left and right side of the entry 1, respectively which can be calculated by using the proposed algorithm.

In (5), the parameter M can take an optional value which is set to 2 in our experiments. However, by setting M higher than this value, implementation speed decreases while image quality increases.

$$\begin{array}{l}A=[a_{M(k-1)}^{\prime }{a}_{M(k-1)-1}^{\prime }\\ \dots a_{(M-1)(k-1)+1}^{\prime }0a_{(M-1)(k-1)}^{\prime }{a}_{(M-1)(k-1)-1}^{\prime }\\ \dots a_{(M-2)(k-1)+1}^{\prime }0\dots a_{k-1}^{\prime }\dots a_2^{\prime }{a}_1^{\prime }1a_1{a}_2\\ \dots a_{k-1}0\dots a_{(M-2)(k-1)+1}\\ \dots a_{(M-1)(k-1)}0a_{(M-1)(k-1)+1}\dots a_{M(k-1)}]\end{array}$$

(5)

$$X^{\prime }=X_{\mathrm{ZP}}^{″}\ast A$$

(6)

In (6), $X^{\prime }$ and $X_{\mathrm{ZP}}^{″}$ are known and A is unknown, and there are $\left[\frac{N}{k}\right]-\left[\frac{N}{k^2}\right]$ equations and 2M(k−1) unknown variables. This statement means that, if (6) is to have a solution, $\left[\frac{N}{k}\right]-\left[\frac{N}{k^2}\right]$ must be greater than or equal to 2M(k−1).

Step(1-5): To solve (6), we have the following matrix equation:

$$\begin{array}{l}{B}_{\left[\frac{N}{k}\right]\times (1+2(M-1)+2M(k-1\left)\right)}\\ \times A_{(1+2(M-1)+2M(k-1\left)\right)\times 1}=C_{\left[\frac{N}{k}\right]\times 1}\end{array}$$

(7)

where $C=X^{\prime }{}^T$, and A denotes the interpolator matrix.

Step(1-6): In order to find matrix B, We add (Mk−1) zeros to the beginning and the end of vector $X_{\mathrm{ZP}}^{″}$ to obtain vector $X_{\mathrm{ZP}0}^{″}$.

$$X_{\mathrm{ZP}0}^{″}=\left(\begin{array}{llllllllll}0& \dots & 0& x_{\mathrm{ZP},1}^{″}& x_{\mathrm{ZP},2}^{″}& \dots & x_{\mathrm{ZP},\left[\frac{N}{k}\right]}^{″}& 0& \dots & 0\end{array}\right)$$

(8)

where $x_{\mathrm{ZP},i}^{″}$ is the ^{i th} entry of vector $X_{\mathrm{ZP}}^{″}$.

Step(1-7): By using $X_{\mathrm{ZP}0}^{″}$, we can find matrix B as follows:

$$\begin{array}{ll}{B}_{i\ast }& =X_{\mathrm{ZP}0}^{″}(i:i+\text{size}(A)-1)\\ =X_{\mathrm{ZP}0}^{″}(i:i+2M(k-1)-1)\end{array}$$

(9)

$i=1:\text{size}\left(X^{\prime }\right)=1:\left[\frac{N}{k}\right]$

where _{B i∗}is the ^{i th}row of matrix B.

Now, following Step(1-5) and then Step(1-4), we can obtain the multipliers used for image enlargement.

To understand the above algorithm, we can illustrate the above relation with an example in which N=36,K=3,M=2.

Step(1-1):

$$\begin{array}{ll}{X}^{\prime }& =\text{downsample}(X,3)\\ =\left(\begin{array}{lllllllll}{x}_1& x_4& x_7& x_{10}& \dots & x_{25}& x_{28}& x_{31}& x_{34}\end{array}\right)\\ =\left(\begin{array}{lllllllll}{x}_1^{\prime }& x_2^{\prime }& x_3^{\prime }& x_4^{\prime }& \dots & x_9^{\prime }& x_{10}^{\prime }& x_{11}^{\prime }& x_{12}^{\prime }\end{array}\right)\end{array}$$

(10)

Step(1-2):

$$\begin{array}{ll}{X}^{″}& =\text{downsample}(X^{\prime },3)\\ =\left(\begin{array}{llll}{x}_1^{\prime }& x_4^{\prime }& x_7^{\prime }& x_{10}^{\prime }\end{array}\right)\end{array}$$

(11)

Step(1-3):

$$X_{\mathrm{ZP}}^{″}=\left(\begin{array}{llllllllll}{x}_1^{\prime }& 0& 0& x_4^{\prime }& 0& 0& x_7^{\prime }& 0& 0& x_{10}^{\prime }\end{array}\right)$$

(12)

$$B=\left(\begin{array}{lllllllllll}0& 0& 0& 0& 0& x_1^{\prime }& 0& 0& x_4^{\prime }& 0& 0\\ 0& 0& 0& 0& x_1^{\prime }& 0& 0& x_4^{\prime }& 0& 0& x_7^{\prime }\\ 0& 0& 0& x_1^{\prime }& 0& 0& x_4^{\prime }& 0& 0& x_7^{\prime }& 0\\ 0& 0& x_1^{\prime }& 0& 0& x_4^{\prime }& 0& 0& x_7^{\prime }& 0& 0\\ 0& x_1^{\prime }& 0& 0& x_4^{\prime }& 0& 0& x_7^{\prime }& 0& 0& x_{10}^{\prime }\\ x_1^{\prime }& 0& 0& x_4^{\prime }& 0& 0& x_7^{\prime }& 0& 0& x_{10}^{\prime }& 0\\ 0& 0& x_4^{\prime }& 0& 0& x_7^{\prime }& 0& 0& x_{10}^{\prime }& 0& 0\\ 0& x_4^{\prime }& 0& 0& x_7^{\prime }& 0& 0& x_{10}^{\prime }& 0& 0& 0\\ x_4^{\prime }& 0& 0& x_7^{\prime }& 0& 0& x_{10}^{\prime }& 0& 0& 0& 0\\ 0& 0& x_7^{\prime }& 0& 0& x_{10}^{\prime }& 0& 0& 0& 0& 0\\ 0& x_7^{\prime }& 0& 0& x_{10}^{\prime }& 0& 0& 0& 0& 0& 0\\ x_7^{\prime }& 0& 0& x_{10}^{\prime }& 0& 0& 0& 0& 0& 0& 0\end{array}\right)$$

(13)

Two-dimensional algorithm

A typical image is a matrix of N rows and N columns which can be enlarged by a factor of k to kN×kN by applying the one-dimensional algorithm discussed in the previous section to each row and column separately. Enlargement requires that we first down-sample the given matrix by factors of _{k 1} and _{k 2}, then up-sample the modified $\left[\frac{N}{k_1}\right]\times \left[\frac{N}{k_2}\right]$ matrix by factors of _{k 1}and _{k 2} to obtain a new N×N matrix. In order to illustrate the mentioned statement, we assume a 16×16 matrix F. We also define a matrix $F^{\prime }$ with entries that are the odd entries of F. Down-sampling $F^{\prime }$ by a factor of k=2, we can obtain $F^{″}$, and up-sampling $F^{″}$ by a factor of k=2 leads us to $F_{\mathrm{up}}^{″}$.

In this case, we know all the entries of F. Equation (7) can be applied to each row and column of $F^{\prime }$. As an illustration, let us consider the following statement for the first row: C is the first row of matrix $F^{\prime }$, B is obtained using (9), and matrix A is unknown. As a result, we should use (7) eight times, four times for the rows, and four times for the columns of matrix F. In as much as (7), itself, contains n equations, where n is the size of vector C, we have 4×8=32 equations (where four is the number of equations:$\left[\frac{N}{k}\right]-\left[\frac{N}{k^2}\right]=\left[\frac{16}{2}\right]-\left[\frac{16}{4}\right]$). We use matrices B and C that are obtained from (7) as the blocks of new matrices named $B^{\prime }$ and $C^{\prime }$, respectively. Then, these 32 equations are solved using linear least square method; moreover, A can be calculated as follows:

$$B^{\prime }\times A=C^{\prime }$$

(14)

Hence:

$$A={\left({B{}^{\prime }}^T{B}^{\prime }\right)}^{-1}{B{}^{\prime }}^T{C}^{\prime }$$

(15)

Now, using A, we zero pad matrix $F^{\prime }$ by factors of _{k 1}and _{k 1}, through the following steps:

$$F_{8\times 8}^{\prime }=\left(\begin{array}{llll}{f}_{1,1}& f_{1,3}& \dots & f_{1,15}\\ f_{3,1}& f_{3,3}& \dots & f_{3,3}\\ ⋮& ⋮& \ddots & ⋮\\ f_{15,1}& f_{15,3}& \dots & f_{15,15}\end{array}\right)$$

(16)

$$F_{4\times 4}^{″}=\left(\begin{array}{llll}{f}_{1,1}& f_{1,5}& f_{1,9}& f_{1,13}\\ f_{5,1}& f_{5,5}& f_{5,9}& f_{5,13}\\ f_{9,1}& f_{9,5}& f_{9,9}& f_{9,13}\\ f_{13,1}& f_{13,5}& f_{13,9}& f_{13,13}\end{array}\right)$$

(17)

$$F_{\mathrm{up}}^{″}=\left(\begin{array}{llllll}{f}_{1,1}& 0& f_{1,5}& 0& \dots & f_{1,13}\\ 0& 0& 0& 0& \dots & 0\\ f_{5,1}& 0& f_{5,5}& 0& \dots & f_{5,13}\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮\\ s{f}_{13,1}& 0& f_{13,5}& 0& \dots & f_{13,13}\end{array}\right)$$

(18)

Figure 1 shows an $\left[\frac{N}{k}\right]\times \left[\frac{N}{k}\right]$ image that is zero-padded by factors of _{k 1}=_{k 2}=2.

Figure 1

A zero-padded image by factors of k₁ = k₂ = 2.

The gray blocks in Figure 1 contain the pixels of the main N×N image and the remaining blocks contain zero. Now, in order to apply the proposed method to this figure, we need to take the following steps:

Step (2-1): We first inspect the odd rows and then apply the rule mentioned before to each one to determine the amounts which must be replaced with zeros in the white blocks in Figure 1, using the amounts of adjacent blocks.

Step (2-2): Next, we apply step (1) to the odd columns.

As can be seen from Figure 1, even rows and even columns contain zeros; therefore by using steps (1) and (2), the values that some of these blocks contain can be calculated. In Figure 2, blocks that contain circles are obtained by applying step (1), and those containing diamonds are obtained by applying step (2) to Figure 1.

Figure 2

Blocks containing circles are obtained by applying step (1), and those with diamonds are obtained by applying step (2) to Figure1.

Consequently, only the blocks in even rows and columns hold zeros (white blocks in Figure 2), and the goal is to determine the values of these blocks. To overcome this challenge, we can use one of the following approaches:

Method (1) After applying step (1), the blocks that contain circles in Figure 2 are identified. Now, by applying step (2) to the same columns, we can obtain the remaining unknown blocks.

Method (3) After applying step (2), the blocks that contain diamonds in Figure 2 are spotted. Then, by applying step (1) to the same rows, the remaining unknown blocks are extracted.

Method (3) We can average the amounts obtained through methods (1) and (2) to each block to work out a new value.

In this way, the proposed method can provide us a high resolution image from a low resolution one. We can find the interpolator multipliers by employing the downsampled version of the original image. Then, the HR image can be estimated by using the original LR image and the computed multipliers.

Simulation results

The proposed image interpolation algorithm was implemented and its performance was compared with some existing methods. The results of the proposed method simulation are benchmarked against some classic methods such as bilinear, bicubic, and some recent methods such as SAI [5] and WZP-CS [6] that are based on wavelet transform, and the results are tabulated in Tables 1 and 2. Table 1 shows the PSNR of implementing the mentioned methods on nine 256×256 images, in order to enlarge them by a factor of 2, and the SSIM of them is indicated in Table 2. The simulations are implemented using Matlab 7.10.0.499(R2010a) software on a VAIO SR 490 DCB laptop with an Intel Core2 Duo P8800 2.66 GHz CPU and a 4 GB RAM.

Table 1 PSNR

Table 2 SSIM

As can be seen from these two tables, the results achieved by our proposed approach outperform those from other solutions. The bold values given in each row of these two tables indicate that the proposed method outperforms the other compared methods when applied to the images of the mentioned rows. The average PSNR of implementing the proposed method, is 29.0030 dB, which is higher than the other values of the compared solutions. On the other hand, the average SSIM of implementing the proposed method on the mentioned images is 0.8229 dB which is higher than the average SSIM for the other compared algorithms. In order to compare the methods objectively, Figure 3 shows that, applying the bilinear method yields blurring and also has less implementation speed than the proposed method. In addition, the scheme based on WZP-CS causes damages at the edges of the image. In the Cameraman image (Figure 4), applying the proposed method produces a better visual quality compared to other known models.

Figure 3

Comparison of different methods for the Lena image: (a) the original image, (b) bilinear interpolation, (c) bicubic interpolation, (d) WZP cycle-spinning[6], (e) SAI[5], (f) NEDI[3], (g) optimized spline[7], and (h) the proposed method.

Figure 4

Comparison of different methods for the Cameraman image: (a) the original image, (b) bilinear interpolation, (c) bicubic interpolation, (d) WZP cycle-spinning[6], (e) SAI[5], (f) NEDI[3], (g) optimized spline[7], and (h) the proposed method.

Also, since we do not use blocks in this presentation, unlike the SAI method, there is no blocking result in the obtained image.

Moreover, one of the most important criteria that should be taken into account for comparison is computational complexity of the algorithms, which can be computed by considering the number of operations in the algorithms. In the proposed method, the number of multiplications and summations in order to calculate the matrix A from expression (15) for an N×N image to be enlarged by a factor of K are $\left[\right(\frac{N}{K}\left)\right(2\mathrm{MK}-1\left)\right(2\mathrm{MK})+(2\mathrm{MK}-1)!+(2\mathrm{MK}-1\left)\right(6\mathrm{MK}-1\left)\right]$ and $\left[\right(\frac{N}{K}\left)\right(2\mathrm{MK}-1\left)\right(2\mathrm{MK})+0.5(2\mathrm{MK}-1)!-3(2\mathrm{MK}-1\left)\right]$, respectively; where 2M is the number of entries of the LR image that are used for interpolation of each pixel. By assuming M=K=2 in our experiments, there will be (28N + 5201) multiplications and (28N + 2499) summations for calculating the matrix A. On the other hand, after calculating the matrix A, the number of multiplications and summations to estimate the value of each pixel are $2M(N\times N-\frac{N}{K}\times \frac{N}{K})$ and $(2M-1)(N\times N-\frac{N}{K}\times \frac{N}{K})$, respectively. By assuming M=K=2 in our experiments, there will be 3^{N 2}multiplications and $\frac{9}{4}{N}^2$ summations for estimating the value of each pixel. The number of operations to estimate the value of each pixel is approximately on the same order in the compared methods, and the difference is in the number of operations in order to calculate the multipliers of the linear combination, hence, in order to compare the proposed method with the other mentioned ones, we consider the number of operations in order to calculate the matrix A, where, this number of operations is acceptable for our purpose.

In addition to this, since all algorithms are simulated in the same situation, running time may have a better sense of complexity. As it is indicated in Table 3, the proposed method outperforms the high performance algorithms such as SAI and optimized spline in terms of the average time that is needed to simulate the algorithm, and features a higher implementation speed.

Table 3 Average calculation time for enlarging a 256 * 256 image by a factor of 2

The linear least square study unveiled in this article also offers an advantage when transmitting an image where we have memory limitation; i.e. instead of transmitting the enlarged version of the original image by a factor of k, we can transmit the original image and the interpolator matrix A, and then use the interpolator matrix to enlarge the original image by a factor of k at the receiving end. Accordingly, less memory is used in transmitting the image. In the images that are simulated, the interpolator matrix entries are decimal numbers with three digits to the right of the decimal point; hence, ten bits would be sufficient in order to send the interpolator matrix as side information.

Conclusion

In this article, we presented a novel image interpolation method. For each image to be interpolated, we found an interpolator function which was applied to each row and column of the image separately by convolving this interpolator function with the vector considered for interpolation. PSNR and SSIM criteria were then used to compare our results versus those obtained by other algorithms. Experimental results demonstrated that the solution revealed in this article achieves better results both objectively and subjectively.

Another key feature of the proposed method is that it allows enlargement of the image by any factor of k=2,3,4,5,… by simply setting this parameter to the desired value.