A novel method in adaptive image enlargement
- Mozhgan Bayat^{1},
- Ghazaleh Kafaie^{1}Email author,
- Ali Ayremlou^{2} and
- Farrokh Marvasti^{1}
DOI: 10.1186/1687-6180-2012-197
© Bayat et al.; licensee Springer. 2012
Received: 2 March 2012
Accepted: 30 July 2012
Published: 7 September 2012
Abstract
This article introduces a new adaptive method for image interpolation. In order to obtain a high resolution (HR) image from its low resolution (LR) counterpart (original image), an interpolator function (array) is used, and the main focus of this manuscript is to formulate and define this function. By applying this interpolator function to each row and column of a LR image, it is possible to construct its HR counterpart. One of the main challenges of image interpolation algorithms is to maintain the edge structures while developing an HR image from the LR replica. The proposed approach overcomes this challenge and exhibits remarkable results at the image edges. The peak signal to noise ratio and structural similarity criteria by using this innovative technique are notably better than those achieved by alternative schemes. Also, in terms of implementation speed, this method displays a clear advantage and outperforms the high performance algorithms in the ability to decrease the artifact results of image enlargement such as blurring and zigzagging.
Keywords
Image interpolation Linear least square Low resolution High resolutionIntroduction
Image interpolation in this article refers to algorithms that transform a low resolution (LR) image to high resolution (HR) one and has many applications. In medical applications, it is highly preferred for images to have high resolution while medical equipments cannot produce images with resolutions higher than a specific standard. In another application, with the development of digital monitors, we need to transform Standard Definition TeleVision (SDTV) video frames to High Definition TeleVision (HDTV) equivalents. Another application of this technology is to find missing pixels or blocks in an image.
Traditional methods such as bilinear, bicubic, and cubic convolution [1, 2] are the common methods in image enlargement. Although the merits of these techniques are simplicity and fast implementation, they suffer from visual degradations such as jagged edges, blurring, and ringing around the image edges.
Human visual system is sensitive to the edges of the objects in an image. As a result, there are several mechanisms which improve the edges of an image. Undoubtedly, as the quality of image enhances, the computational complexity increases. Hence, in all of these methods, there is a trade off between performance and affordable complexity. In the last few decades, several authors have carried out researches on this topic. Along the same line, we briefly introduce [3–10] as follows.
In NEDI [3], the aim is to maintain the edge structure in HR images. This is a nonlinear algorithm whose basic idea is to estimate the covariance of HR image from the covariance of LR counterpart. The estimation is based on the geometric duality between low and high resolution. The algorithm presented in [4] which is based on directional filtering and data fusion to find the missing pixel, considers two orthogonal sets which produce an estimation of the pixel value. In SAI [5], using a moving window in the input LR image, one can find the model parameters. The pixel structure obtained in this way can be led to a block of both available and estimated pixels using a soft-decision estimation process. In WZP-CS [6], wavelet transformation is applied to LR image. Using sub-bands of the resultant image, one can estimate sub-bands of HR image. Afterwards, the HR image can be obtained by applying the inverse wavelet transform. In Optimized spline [7], by taking advantages of calculus of variations, the optimization problem is simplified from a nonlinear infinite dimensional case to a linear finite dimensional one in order to design compact-support interpolation kernels for a class of signals. The method presented in [8] tries to create up-scaled images that are suitable for real time applications and appear natural to human observer. In this algorithm a two step grid filling is used. Then, an iterative correction is obtained by minimizing an objective function depending on the second order directional derivatives of the image intensity.
In [9], in order to enlarge the image, it is down-sampled by using the bilinear method and then, the missing pixels are estimated by using a combination of the other pixels, while the desired coefficients are optimized by using a least mean square technique. In [10], the authors have extended the algorithm presented in [9] to display products such as video restoration.
In our approach, we work out the parameters that are used to enlarge an image from its LR counterpart. These parameters depend on the original image and the factor which we want to enlarge the image by. In order to enlarge an image or find the missing pixels in it, an interpolation array which is constructed by the mentioned parameters is convolved by the matrix that contains the values of the image pixels.
The rest of the article is organized as follows: The following section describes the proposed interpolation method for one-dimensional vectors. The two-dimensional application is proposed in section “Two-dimensional algorithm”. Simulation results and comparisons are given in section “Simulation results” , and section “Conclusion” concludes the paper.
Algorithm description
In order to describe the proposed algorithm for image interpolation, we first apply it to one-dimensional vectors. Let _{ X LR } be a 1×N vector. Suppose we wish to enlarge _{ X LR } to a kN sized vector _{ X HR }using the following steps: We put k−1 zeros between any two successive entries of vector _{ X LR }. In order to achieve these kN−N new entries of _{ X HR }, we choose 2M of N entries of _{ X LR }. The new value will be a linear combination of these 2M entries.
In order to find the 2M indeterminate multipliers of this linear combination, we should follow the procedure stated below, bearing in mind that for each fixed vector of any size the procedure and the multipliers applied to enlarge the vector are fixed. This means that, the multipliers used for obtaining a 1×N vector from its $1\times \left[\frac{N}{k}\right]$ counterpart are the same as the ones used to calculate a $1\times \left[\frac{N}{k}\right]$ vector from its $1\times \left[\frac{N}{{k}^{2}}\right]$ counterpart.
Step(1-1): As presented in (1) and (2), we down-sample the original vector _{ X LR } by a factor of k to find ${X}_{1\times \left[\frac{N}{k}\right]}^{\prime}$.
Step(1-2): Again we down-sample ${X}_{1\times \left[\frac{N}{k}\right]}^{\u2033}$ by a factor of k to obtain ${X}_{1\times \left[\frac{N}{{k}^{2}}\right]}^{\prime}$ vector.
Step(1-3): Now, by constructing $1\times \left[\frac{N}{k}\right]$ vector ${X}^{\prime}$ from vector ${X}^{\u2033}$ of size $1\times \left[\frac{N}{{k}^{2}}\right]$, we can find the multipliers used to obtain the enlarged image. To approach this, we must zero-pad ${X}^{\u2033}$ by a factor of k to build vector ${X}_{\mathrm{ZP}}^{\u2033}$. This means that, k−1 zeros are inserted between every two successive entries of ${X}^{\u2033}$.
As it was stated earlier, by using these multipliers, we can find a new vector _{ X HR } from ${X}_{1\times \left[\frac{N}{k}\right]}^{\prime}$.
Step(1-4): In order to compute vector ${X}^{\u2033}$ from ${X}_{\mathrm{ZP}}^{\mathrm{\u2033\u2033}}$, we must insert a linear combination of ${x}_{i}^{\mathrm{\u2033\u2033}}$’s; $i=1:\phantom{\rule{2.0pt}{0ex}}\left[\frac{N}{{k}^{2}}\right]$ instead of each zero in ${X}_{\mathrm{ZP}}^{\mathrm{\u2033\u2033}}$. For this purpose, we can convolve ${X}_{\mathrm{ZP}}^{\mathrm{\u2033\u2033}}$ with an interpolator vector, A (which is determined by (5)).
In this equation, A is a 1×(1 + 2(M−1) + 2M(k−1)) vector. The 1 + (M−1) + M(k−1)^{st} entry, which is located at the middle of the vector, is 1. Except for the mentioned entry, the entries with the indices that are multiples of k are 0, and there are (k−1)a^{ i }_{ ″ }s and (k−1)_{ a i }s between any two successive zeros at the left and right side of the entry 1, respectively which can be calculated by using the proposed algorithm.
In (6), ${X}^{\prime}$ and ${X}_{\mathrm{ZP}}^{\u2033}$ are known and A is unknown, and there are $\left[\frac{N}{k}\right]-\left[\frac{N}{{k}^{2}}\right]$ equations and 2M(k−1) unknown variables. This statement means that, if (6) is to have a solution, $\left[\frac{N}{k}\right]-\left[\frac{N}{{k}^{2}}\right]$ must be greater than or equal to 2M(k−1).
where $C={X}^{\prime}{}^{T}$, and A denotes the interpolator matrix.
where ${x}_{\mathrm{ZP},i}^{\u2033}$ is the ^{i th} entry of vector ${X}_{\mathrm{ZP}}^{\u2033}$.
$i=1:\text{size}\left({X}^{\prime}\right)=1:\phantom{\rule{2.0pt}{0ex}}\left[\frac{N}{k}\right]$
where _{B i∗}is the ^{i th}row of matrix B.
Now, following Step(1-5) and then Step(1-4), we can obtain the multipliers used for image enlargement.
To understand the above algorithm, we can illustrate the above relation with an example in which N=36,K=3,M=2.
Two-dimensional algorithm
A typical image is a matrix of N rows and N columns which can be enlarged by a factor of k to kN×kN by applying the one-dimensional algorithm discussed in the previous section to each row and column separately. Enlargement requires that we first down-sample the given matrix by factors of _{k 1} and _{k 2}, then up-sample the modified $\left[\frac{N}{{k}_{1}}\right]\times \left[\frac{N}{{k}_{2}}\right]$ matrix by factors of _{k 1}and _{k 2} to obtain a new N×N matrix. In order to illustrate the mentioned statement, we assume a 16×16 matrix F. We also define a matrix ${F}^{\prime}$ with entries that are the odd entries of F. Down-sampling ${F}^{\prime}$ by a factor of k=2, we can obtain ${F}^{\u2033}$, and up-sampling ${F}^{\u2033}$ by a factor of k=2 leads us to ${F}_{\mathrm{up}}^{\u2033}$.
The gray blocks in Figure 1 contain the pixels of the main N×N image and the remaining blocks contain zero. Now, in order to apply the proposed method to this figure, we need to take the following steps:
Step (2-1): We first inspect the odd rows and then apply the rule mentioned before to each one to determine the amounts which must be replaced with zeros in the white blocks in Figure 1, using the amounts of adjacent blocks.
Step (2-2): Next, we apply step (1) to the odd columns.
Consequently, only the blocks in even rows and columns hold zeros (white blocks in Figure 2), and the goal is to determine the values of these blocks. To overcome this challenge, we can use one of the following approaches:
Method (1) After applying step (1), the blocks that contain circles in Figure 2 are identified. Now, by applying step (2) to the same columns, we can obtain the remaining unknown blocks.
Method (3) After applying step (2), the blocks that contain diamonds in Figure 2 are spotted. Then, by applying step (1) to the same rows, the remaining unknown blocks are extracted.
Method (3) We can average the amounts obtained through methods (1) and (2) to each block to work out a new value.
In this way, the proposed method can provide us a high resolution image from a low resolution one. We can find the interpolator multipliers by employing the downsampled version of the original image. Then, the HR image can be estimated by using the original LR image and the computed multipliers.
Simulation results
PSNR
Images | Bilinear | Bicubic | WZP-CS | SAI | NEDI | Optimized spline | Proposed method |
---|---|---|---|---|---|---|---|
Lena | 29.8768 | 29.7936 | 29.7871 | 32.5795 | 32.999 | 32.2976 | 33.4913 |
Couple | 26.7166 | 26.5342 | 26.6294 | 27.4276 | 27.8451 | 27.912 | 28.6865 |
Barbara | 23.6373 | 23.0442 | 23.5365 | 23.0267 | 21.1754 | 25.1 | 25.3936 |
Fishing boat | 26.8857 | 26.7076 | 26.8139 | 28.1566 | 28.1784 | 28.5054 | 28.8645 |
Cameraman | 30.0606 | 30.2018 | 29.972 | 32.2363 | 32.995 | 33.2886 | 38.1691 |
Crowd | 19.8674 | 19.6677 | 19.7829 | 20.7484 | 21.2964 | 21.8277 | 21.906 |
Lake | 27.1699 | 27.0709 | 27.0786 | 28.6805 | 28.5925 | 28.4984 | 29.8961 |
Baboon | 22.0974 | 21.7332 | 22.0273 | 22.6396 | 23.1457 | 22.509 | 23.0153 |
Girl | 30.2527 | 29.8225 | 30.1298 | 29.2056 | 31.7538 | 30.9017 | 31.605 |
Average | 26.2849 | 26.0639 | 26.1952 | 27.1889 | 27.5534 | 27.8711 | 29.0030 |
SSIM
Images | Bilinear | Bicubic | WZP-CS | SAI | NEDI | Optimized spline | Proposed method |
---|---|---|---|---|---|---|---|
Lena | 0.8735 | 0.8737 | 0.8728 | 0.8843 | 0.9076 | 0.9046 | 0.9075 |
Couple | 0.7809 | 0.7863 | 0.7825 | 0.8062 | 0.8304 | 0.8306 | 0.8356 |
Barbara | 0.7346 | 0.7414 | 0.7397 | 0.7646 | 0.7473 | 0.7885 | 0.7592 |
Fishing boat | 0.7868 | 0.7882 | 0.7871 | 0.8115 | 0.8252 | 0.8339 | 0.8348 |
Cameraman | 0.937 | 0.9448 | 0.9374 | 0.9368 | 0.9568 | 0.9621 | 0.9648 |
Crowd | 0.7091 | 0.7229 | 0.7129 | 0.7718 | 0.783 | 0.801 | 0.7984 |
Lake | 0.8032 | 0.802 | 0.8028 | 0.8232 | 0.8374 | 0.8474 | 0.8392 |
Baboon | 0.6182 | 0.6367 | 0.6262 | 0.6899 | 0.7195 | 0.6766 | 0.7026 |
Girl | 0.7486 | 0.7337 | 0.7462 | 0.7491 | 0.7802 | 0.7616 | 0.7648 |
Average | 0.7768 | 0.7810 | 0.7786 | 0.8041 | 0.8208 | 0.8228 | 0.8229 |
Also, since we do not use blocks in this presentation, unlike the SAI method, there is no blocking result in the obtained image.
Moreover, one of the most important criteria that should be taken into account for comparison is computational complexity of the algorithms, which can be computed by considering the number of operations in the algorithms. In the proposed method, the number of multiplications and summations in order to calculate the matrix A from expression (15) for an N×N image to be enlarged by a factor of K are $\left[\right(\frac{N}{K}\left)\right(2\mathrm{MK}-1\left)\right(2\mathrm{MK})+(2\mathrm{MK}-1)!+(2\mathrm{MK}-1\left)\right(6\mathrm{MK}-1\left)\right]$ and $\left[\right(\frac{N}{K}\left)\right(2\mathrm{MK}-1\left)\right(2\mathrm{MK})+0.5(2\mathrm{MK}-1)!-3(2\mathrm{MK}-1\left)\right]$, respectively; where 2M is the number of entries of the LR image that are used for interpolation of each pixel. By assuming M=K=2 in our experiments, there will be (28N + 5201) multiplications and (28N + 2499) summations for calculating the matrix A. On the other hand, after calculating the matrix A, the number of multiplications and summations to estimate the value of each pixel are $2M(N\times N-\frac{N}{K}\times \frac{N}{K})$ and $(2M-1)(N\times N-\frac{N}{K}\times \frac{N}{K})$, respectively. By assuming M=K=2 in our experiments, there will be 3^{N 2}multiplications and $\frac{9}{4}{N}^{2}$ summations for estimating the value of each pixel. The number of operations to estimate the value of each pixel is approximately on the same order in the compared methods, and the difference is in the number of operations in order to calculate the multipliers of the linear combination, hence, in order to compare the proposed method with the other mentioned ones, we consider the number of operations in order to calculate the matrix A, where, this number of operations is acceptable for our purpose.
Average calculation time for enlarging a 256 * 256 image by a factor of 2
Images | Bilinear | Bicubic | WZP-CS | SAI | NEDI | Optimized spline | Proposed method |
---|---|---|---|---|---|---|---|
Average calculation time (in seconds) | 0.0112 | 0.0137 | 2.85 | 8.6846 | 1.4103 | 3.7137 | 1.7579 |
The linear least square study unveiled in this article also offers an advantage when transmitting an image where we have memory limitation; i.e. instead of transmitting the enlarged version of the original image by a factor of k, we can transmit the original image and the interpolator matrix A, and then use the interpolator matrix to enlarge the original image by a factor of k at the receiving end. Accordingly, less memory is used in transmitting the image. In the images that are simulated, the interpolator matrix entries are decimal numbers with three digits to the right of the decimal point; hence, ten bits would be sufficient in order to send the interpolator matrix as side information.
Conclusion
In this article, we presented a novel image interpolation method. For each image to be interpolated, we found an interpolator function which was applied to each row and column of the image separately by convolving this interpolator function with the vector considered for interpolation. PSNR and SSIM criteria were then used to compare our results versus those obtained by other algorithms. Experimental results demonstrated that the solution revealed in this article achieves better results both objectively and subjectively.
Another key feature of the proposed method is that it allows enlargement of the image by any factor of k=2,3,4,5,… by simply setting this parameter to the desired value.
Declarations
Acknowledgements
The authors would like to thank Seyyed Amirhossein Hosseini for editing this article and his helpful comments.
Authors’ Affiliations
References
- Keys R: Cubic convolution interpolation for digital image processing. IEEE Trans. Acoust. Speech Signal Process 1981, 29(6):1153-1160. 10.1109/TASSP.1981.1163711MathSciNetView ArticleMATHGoogle Scholar
- Hou HS, Andrews HC: Cubic splines for image interpolation and digital filtering. IEEE Trans. Acoust. Speech Signal Process 1978, 26(6):508-517. 10.1109/TASSP.1978.1163154View ArticleMATHGoogle Scholar
- Li X, Orchard MT: New edge-directed interpolation. IEEE Trans. Image Process 2001, 10(10):1521-1527. 10.1109/83.951537View ArticleGoogle Scholar
- Zhang L, Wu X: An edge guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process 2006, 15(8):2226-2238.View ArticleGoogle Scholar
- Zhang X, Wu X: Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation. IEEE Trans. Image Process 2008, 17(6):887-896.MathSciNetView ArticleGoogle Scholar
- Temizel A, Vlachos T, Visioprime W: Wavelet domain image resolution enhancement using cycle-spinning. Electron. Lett 2005, 41(3):119-121. 10.1049/el:20057150View ArticleGoogle Scholar
- Madani R, Ayremlou A, Amini A, Marvasti F: Optimized compact-support interpolation kernels. IEEE Trans. Signal Process 2012, 60(2):626-633.MathSciNetView ArticleGoogle Scholar
- Giachetti A, Asuni N: Real time artifact-free image upscaling. IEEE Trans. Image Process 2011, 20(10):2760-2768.MathSciNetView ArticleGoogle Scholar
- Shao L, Hu H, de Haan G: Coding artifact robust resolution up-conversion. IEEE International Conference on Image Processing, vol. 5 (San Antonio, Texas, USA, Sept 2007), pp. 409–412Google Scholar
- Shao L, Zhang H, de Haan G: An overview and performance evaluation of classification based least squares trained filters. IEEE Trans. Image Process 2008, 17(10):1772-1782.MathSciNetView ArticleGoogle Scholar
- Wang Z, Bovik AC, Sheikh HR, Simoncelli EP: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process 2004, 13(4):600-612. 10.1109/TIP.2003.819861View ArticleGoogle Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.