Adaptive examplebased superresolution using kernel PCA with a novel classification approach
 Takahiro Ogawa^{1}Email author and
 Miki Haseyama^{1}
DOI: 10.1186/168761802011138
© Ogawa and Haseyama; licensee Springer. 2011
Received: 8 June 2011
Accepted: 22 December 2011
Published: 22 December 2011
Abstract
An adaptive examplebased superresolution (SR) using kernel principal component analysis (PCA) with a novel classification approach is presented in this paper. In order to enable estimation of missing highfrequency components for each kind of texture in target lowresolution (LR) images, the proposed method performs clustering of highresolution (HR) patches clipped from training HR images in advance. Based on two nonlinear eigenspaces, respectively, generated from HR patches and their corresponding lowfrequency components in each cluster, an inverse map, which can estimate missing highfrequency components from only the known lowfrequency components, is derived. Furthermore, by monitoring errors caused in the above estimation process, the proposed method enables adaptive selection of the optimal cluster for each target local patch, and this corresponds to the novel classification approach in our method. Then, by combining the above two approaches, the proposed method can adaptively estimate the missing highfrequency components, and successful reconstruction of the HR image is realized.
Keywords
Superresolution resolution enhancement image enlargement Kernel PCA classification1 Introduction
In the field of image processing, highresolution images are needed for various fundamental applications such as surveillance, highdefinition TV and medical image processing [1]. However, it is often difficult to capture images with sufficient high resolution (HR) from current image sensors. Thus, methodologies for increasing resolution levels are used to bridge the gap between demands of applications and the limitations of hardware; and such methodologies include image scaling, interpolation, zooming and enlargement.
Traditionally, nearest neighbor, bilinear, bicubic [2], and sinc [3] (Lanczos) approaches have been utilized for enhancing spatial resolutions of lowresolution (LR) images. However, since they do not estimate highfrequency components missed from the original HR images, their results suffer from some blurring. In order to overcome this difficulty, many researchers have proposed superresolution (SR) methods for estimating the missing highfrequency components, and this enhancement technique has recently been one of the most active research areas [1, 4–7]. Superresolution refers to the task which generates an HR image from one or more LR images by estimating the highfrequency components while minimizing the effects of aliasing, blurring, and noise. Generally, SR methods are divided into two categories: reconstructionbased and learningbased (examplebased) approaches [7, 8]. The reconstructionbased approach tries to recover the HR image from observed multiple LR images. Numerous SR reconstruction methods have been proposed in the literature, and Park et al. provided a good review of them [1]. Most reconstructionbased methods perform registration between LR images based on their motions, followed by restoration for blur and noise removal. On the other hand, in the learningbased approach, the HR image is recovered by utilizing several other images as training data. These motionfree techniques have been adopted by many researchers, and a number of learningbased SR methods have been proposed [9–18]. For example, Freeman et al. proposed examplebased SR methods that estimate missing highfrequency components from midfrequency components of a target image based on Markov networks and provide an HR image [10, 11]. In this paper, we focus on the learningbased SR approach. Conventionally, learningbased SR methods using principal component analysis (PCA) have been proposed for face hallucination [19]. Furthermore, by applying kernel methods to the PCA, Chakrabarti et al. improved the performance of the face hallucination [20] based on the Kernel PCA (KPCA; [21, 22]). Most of these techniques are based on global approaches in the sense that processing is done on the whole of LR images simultaneously. This imposes the constraint that all of the training images should be globally similar, i.e., they should represent a similar class of objects [7, 23, 24]. Therefore, the global approach is suitable for images of a particular class such as face images and fingerprint images. However, since the global approach requires the assumption that all of the training images are in the same class, it is difficult to apply it to arbitrary images.
As a solution to the above problem, several methods based on local approaches in which processing is done for each local patch within target images have recently been proposed [13, 25, 26]. Kim et al. developed a globalbased face hallucination method and a localbased SR method of general images by using the KPCA [27]. It should be noted that even if the PCA or KPCA is used in the local approaches, all of the training local patches are not necessarily in the same class, and their eigenspace tends not to be obtained accurately. In addition, Kanemura et al. proposed a framework for expanding a given image based on an interpolator which is trained in advance with training data by using sparse Bayesian estimation [12]. This method is not based on PCA and KPCA, but calculates the Bayesbased interpolator to obtain HR images. In this method, one interpolator is estimated for expanding a target image, and thus, the image should also contain only the same kind of class. Then it is desirable that training local patches are first clustered and the SR is performed for each target local patch using the optimal cluster. Hu et al. adopted the above scheme to realize the reconstruction of HR local patches based on nonlinear eigenspaces obtained from clusters of training local patches by the KPCA [8]. Furthermore, we have also proposed a method for reconstructing missing intensities based on a new classification scheme [28]. This method performs the superresolution by treating this problem as a missing intensity interpolation problem. Specifically, our previous method introduces two constraints, eigenspaces of HR patches and known intensities, and the iterative projection onto these constraints is performed to estimate HR images based on the interpolation of the missing intensities removed by the subsampling process. Thus, in our previous work, intensities of a target LR image are directly utilized as those of the enlarged result. Thus, if the target LR image is obtained by blurring and subsampling its HR image, the intensities in the estimated HR image contain errors.
In conventional SR methods using the PCA or KPCA, but not including our previous work [28], there have been two issues. First, it is assumed in these methods that the LR patches and their corresponding HR patches that are, respectively, projected onto linear or nonlinear eigenspaces are the same, these eigenspaces being obtained from training HR patches [8, 27]. However, these two are generally different, and there is a tendency for this assumption not to be satisfied. Second, to select optimal training HR patches for target LR patches, distances between their corresponding LR patches are only utilized.
Unfortunately, it is well known that the selected HR patches are not necessarily optimal for the target LR patches, and this problem is known as the outlier problem. This problem has also been reported by Datsenko and Elad [29, 30].
In this paper, we present an adaptive examplebased SR method using KPCA with a novel texture classification approach. The proposed method first performs the clustering of training HR patches and generates two nonlinear eigenspaces of HR patches and their corresponding lowfrequency components belonging to each cluster by the KPCA.
Furthermore, to avoid the problems of previously reported methods, we introduce two novel approaches into the estimation of missing highfrequency components for the corresponding patches containing lowfrequency components obtained from a target LR image: (i) an inverse map, which estimates the missing highfrequency components, is derived from a degradation model of the LR image and the two nonlinear eigenspaces of each cluster and (ii) classification of the target patches is performed by monitoring errors caused in the estimation process of the missing highfrequency components. The first approach is introduced to solve the problem of the assumptions utilized in the previously reported methods. Then, since the proposed method directly derives the inverse map of the missing process of the highfrequency components, we do not rely on their assumptions. The second approach is introduced to solve the outlier problem. Obviously, it is difficult to perfectly perform classification that can avoid this problem as long as the highfrequency components of the target patches are completely unknown. Thus, the proposed method modifies the conventional classification schemes utilizing distances between LR patches directly. Specifically, the error caused in the estimation process of the missing highfrequency components by each cluster is monitored and utilized as a new criterion for performing the classification. This error corresponds to the minimum distance of the estimation result and the known parts of the target patch, and thus we adopt it as the new criterion. Consequently, by the inverse map determined from the nonlinear eigenspaces of the optimal cluster, the missing highfrequency components of the target patches are adaptively estimated. Therefore, successful performance of the SR can be expected. This paper is organized as follows: first, in Section 2, we briefly explain KPCA used in the proposed method. In Section 3, we discuss the formulation model of LR images. In Section 4, the adaptive KPCAbased SR algorithm is presented. In Section 5, the effectiveness of our method is verified by some results of experiments. Concluding remarks are presented in Section 6.
2 Kernel principal component analysis
In this paper, vector/matrix transpose in both input and feature spaces is denoted by the superscript '.
is obtained. By solving the above equation, α can be obtained, and the eigenvectors u can be obtained from Equation 4.
Note that (i, j)th element of K is obtained by ϕ(x_{ i })'ϕ(x_{ j }). In kernel methods, it can be obtained by using kernel trick [21]. Specifically, it can be obtained by some kernel functions κ(x_{ i }, x_{ j }) using only x_{ i }and x_{ j }in the input space.
3 Formulation model of LR images
where f and F are, respectively, vectors whose elements are the rasterscanned intensities in the LR image f and its corresponding HR image F. Therefore, the dimension of these vectors are, respectively, the number of pixels in f and F. D and B are the decimation and blur matrices, respectively. The vector n is the noise vector, whose dimension is the same as that of f. In this paper, we assume that n is the zero vector in order to make the problem easier. Note that if decimation is performed without any blur, the observed LR image is severely aliased.
Generally, actual LR images captured from commercially available cameras tend to be taken without suffering from aliasing. Thus, we assume that such captured LR images do not contain any aliasing effects. However, it should be noted that for realizing the SR, we can consider several assumptions, and thus, we focus on the following three cases:
Case 1 : LR images are captured based on the lowpass filter followed by the decimation procedure, and any aliasing effects do not occur, where this case corresponds to our assumption. Therefore, we should estimate the missing highfrequency components removed by the lowpass filter.
Case 2 : LR images are captured by only the decimation procedure without using any lowpass filters. In this case, some aliasing effects occur, and interpolationbased methods work better than our method.
Case 3 : LR images are captured based on the lowpass filter followed by the decimation procedure, but some aliasing effects occur. In this case, the problem becomes much more difficult than those of Cases 1 and 2. Furthermore, in our method, it becomes difficult to model this degradation process.
We focus only on Case 1 to realize the SR, but some comparisons between our method and the methods focusing on Case 2 are added in the experiments.
For the following explanation, we clarify the definitions of the following four images:
HR image F whose vector is F in Equation 9 is the original image that we try to estimate.
Blurred HR image $\widehat{F}$ whose vector is BF is obtained by applying the lowpass filter to the HR image F. Its size is the same as that of the HR image.
LR image f whose vector is f (= DBF) is obtained by applying the subsampling to the blurred HR image $\widehat{F}$.
Highfrequency components whose vector is F  BF are obtained by subtracting BF from F.
Note that the HR image, the blurred HR image, and the highfrequency components have the same size. In order to define the blurred HR image, the LR image, and the highfrequency components, we have to provide which kind of the lowpass filter is utilized for defining the matrix B. Generally, it is difficult to know the details of the lowpass filter and provide the knowledge of the blur matrix B. Therefore, we simply assume that the lowpass filter is fixed to the sinc filter with the hamming window in this paper. In the proposed method, highfrequency components of target images must be estimated from only their lowfrequency components and other HR training images. This means when the highfrequency components are perfectly removed, the problem becomes the most difficult and useful for the performance verification. Since it is well known that the sinc filter is suitable one to effectively remove the highfrequency components, we adopted this filter. Furthermore, the sinc filter has infinite length coefficients, and thus we also adopted the hamming window to truncate the filter coefficients. The details of the lowpass filter is shown in Section 5. Since the matrix B is fixed, we discuss the sensitivity of our method to the errors in the matrix B in Section 5.
In the proposed method, we assume that LR images are captured based on the lowpass filter followed by the decimation, and aliasing effects do not occur. Furthermore, the decimation matrix is only an operator which subsamples pixel values. Therefore, when the magnification factor is determined for target LR images, the matrices B and D can be also obtained in our method. Specifically, the decimation matrix D can be easily defined when the magnification factor is determined. In addition, the blurring matrix B is also defined by the sinc function with the hamming window in such a way that target LR images do not suffer from aliasing effects. In this way, the matrices B and D can be defined, but in our method, these matrices are not directly utilized for the reconstruction. The details are shown in the following section.
4 KPCAbased adaptive SR algorithm
In order to realize the adaptive SR algorithm, the training HR patches must first be assigned to several clusters before generating each cluster's nonlinear eigenspaces. Therefore, the clustering method is described in detail in 4.1, and the method for estimating the missing highfrequency components of the target local patches is presented in 4.2.
4.1 Clustering of training HR patches
In this subsection, clustering of training HR patches into K clusters is described. In the proposed method, we calculate a nonlinear eigenspace for each cluster and enable the modeling of the elements belonging to each cluster by its nonlinear eigenspace. Then, based on these nonlinear eigenspaces, the proposed method can perform the clustering of training HR patches in this subsection and the highfrequency component estimation, which simultaneously realizes the classification of target patches for realizing the adaptive reconstruction, in the following subsection. This subsection focuses on the clustering of training HR patches based on the nonlinear eigenspaces.
Then a new vector ϕ_{ i } = [ϕ(l_{ i })', h_{ i }']' is defined. Note that an exact preimage, which is the inverse mapping from the feature space back to the input space, typically does not exist [31]. Therefore, the estimated preimage includes some errors. Since the final results estimated in the proposed method are the missing highfrequency components, we do not utilize the nonlinear map for h_{ i }(i = 1, 2, . . . , N).
where N^{ k } is the number of elements belonging to cluster k. Generally, superscript is used to indicate the power of a number. However, in this paper, only k does not represent the power of a number. The vectors ${\mathbf{l}}_{j}^{k}$ and ${\mathbf{h}}_{j}^{k}$ (j = 1, 2, . . . , N^{ k } ), respectively, represent l_{ i }and h_{ i }of g_{ i } (i = 1, 2, . . . , N) assigned to cluster k. In Equation 12, the proposed method minimizes C with respect to the belonging cluster number of each local patch g_{ i } . Each known local patch belongs to the cluster whose nonlinear eigenspace can perform the most accurate approximation of its low and highfrequency components. Therefore, using Equation 12, we try to determine the clustering results, i.e., which cluster is the optimal for each known local patch g_{ i } .
In the above equation, e^{ k }= [1, 1, . . . , 1]' is an N^{ k } × 1 vector. As described above, ${\stackrel{\u0303}{\varphi}}_{j}^{k}$ is the projection result of ${\varphi}_{j}^{k}$ onto the nonlinear eigenspace of cluster k, i.e., the approximation result of ${\varphi}_{j}^{k}$ in the subspace of cluster k. Therefore, Equation 13 represents the projection of jth element of cluster k onto the nonlinear eigenspace of cluster k. Note that from Equation 13, ${\stackrel{\u0303}{\varphi}}_{j}^{k}$ can be defined as ${\stackrel{\u0303}{\varphi}}_{j}^{k}={\left[{\zeta}_{j}^{k\prime},\phantom{\rule{2.77695pt}{0ex}}{\stackrel{\u0303}{\mathbf{h}}}_{j}^{k\prime}\right]}^{\prime}$. In detail, ${\zeta}_{j}^{k}$ corresponds to the projection result of the lowfrequency components in the feature space. Furthermore, ${\stackrel{\u0303}{\mathbf{h}}}_{j}^{k}$ corresponds to the result of the highfrequency components, and it can be obtained directly. However, ${\stackrel{\u0303}{\mathbf{l}}}_{j}^{k}$ in Equation 12 cannot be directly obtained since the projection result ${\zeta}_{j}^{k}$ is in the feature space. Generally, we have to solve the preimage estimation problem of ${\stackrel{\u0303}{\mathbf{l}}}_{j}^{k}$ from ${\zeta}_{j}^{k}$, i.e., ${\stackrel{\u0303}{\mathbf{l}}}_{j}^{k}$, which satisfies ${\zeta}_{j}^{k}\cong \varphi \left({\stackrel{\u0303}{\mathbf{l}}}_{j}^{k}\right)$, has to be estimated. In this paper, we call this preimage approximation as [Approximation 1] for the following explanation. Generally, if we perform the preimage estimation of ${\stackrel{\u0303}{\mathbf{l}}}_{j}^{k}$ from ${\zeta}_{j}^{k}$, estimation errors occur. In the proposed method, we adopt some useful derivations in the following explanation and enable the calculation of ${\parallel {\mathbf{l}}_{j}^{k}{\stackrel{\u0303}{\mathbf{l}}}_{j}^{k}\parallel}^{2}$ in Equation 12 without directly solving the preimage problem of ${\zeta}_{j}^{k}$.
where E^{ k }is the N^{ k } × N^{ k } identity matrix. The matrix H plays the centralizing role, and it is commonly used in general PCA and KPCAbased methods.
As described above, the approximation of the matrix U^{ k }is performed. This is a common scheme in KPCAbased methods, where we call this approximation [Approximation 2], hereafter. Since the columns of the matrix U^{ k }are infinitedimensional, we cannot directly use this matrix for the projection onto the nonlinear eigenspace. Therefore, to solve this problem, the matrix U^{ k }is approximated by Equation 18 for realizing the kernel trick. Note that if D^{ k } becomes the same as the rank of Ξ^{ k }, the approximation in Equation 18 becomes equivalent relationship.
Then, from Equations 23 and 25, the criterion C in Equation 12 can be calculated. It should be noted that for calculating the criterion C, we, respectively, use Approximations 1 and 2 once through Equations 2125.
In Equation 13, U^{ k }is utilized for the projection onto the eigenspace spanned by their eigenvectors ${\mathbf{u}}_{d}^{k}\phantom{\rule{2.77695pt}{0ex}}\left(d=1,2,\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}{D}^{k}\right)$. Therefore, the criterion C represents the sum of the approximation errors of ${\varphi}_{j}^{k}\phantom{\rule{2.77695pt}{0ex}}\left(j=1,2,\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}{N}^{k}\right)$ in their eigenspaces. This means that the squared error in Equation 12 corresponds to the distance from the nonlinear eigenspace of each cluster in the input space. Then, the new criterion C is useful for the clustering of training HR local patches. From the clustering results, we can obtain the eigenvector matrix U^{ k }for ${\varphi}_{j}^{k}\phantom{\rule{2.77695pt}{0ex}}\left(j=1,2,\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}{N}^{k}\right)$ belonging to cluster k. Furthermore, we define ${\widehat{\varphi}}_{j}^{k}={\left[\varphi {\left({\mathbf{l}}_{j}^{k}\right)}^{\prime},\phantom{\rule{2.77695pt}{0ex}}{0}^{\prime}\right]}^{\prime}\phantom{\rule{2.77695pt}{0ex}}\left(j=1,2,\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}{N}^{k}\right)$ and also calculate the eigenvector matrix ${\widehat{\mathbf{U}}}^{k}$ for ${\widehat{\varphi}}_{j}^{k}\phantom{\rule{2.77695pt}{0ex}}\left(j=1,2,\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}{N}^{k}\right)$ belonging to cluster k. Finally, we can, respectively, obtain the two nonlinear eigenspaces of HR training patches and their corresponding lowfrequency components for each cluster k.
4.2 Adaptive estimation of missing highfrequency components
where E_{ p × q }and O_{ p × q }are, respectively, the identity matrix and the zero matrix whose sizes are p × q. Furthermore, D_{ ϕ } represents the dimension of the feature space, i.e., infinite dimension in our method. The matrix ${\mathbf{E}}_{{D}_{\varphi}\times {D}_{\varphi}}$ is the identity matrix whose dimension is the same as that of ϕ(l) and O_{ wh × wh }represents the zero matrix which removes the highfrequency components. As shown in the previous section, our method assumes that LR images are obtained by removing their highfrequency components, and aliasing effects do not occur. This means our problem is to estimate the perfectly removed highfrequency components from the known lowfrequency components. Therefore, the problem shown in this section is equivalent to Equation 9, and the solution that is consistent with Equation 9 can be obtained.
is derived, where ${\widehat{\mathbf{\Xi}}}^{k}=\left[{\widehat{\varphi}}_{1}^{k},{\widehat{\varphi}}_{2}^{k},\phantom{\rule{2.77695pt}{0ex}}\dots ,{\widehat{\varphi}}_{{N}^{k}}^{k}\right]$.
Then, by utilizing the nonlinear eigenspace of cluster k, the proposed method can estimate the missing highfrequency components. In this scheme, we, respectively, use Approximations 2 and 3 once through Equations 3141.
The proposed method enables the calculation of the inverse map which can directly reconstruct the highfrequency components. In the previously reported methods [8, 27], they simply project the known frequency components to the eigenspaces of the HR patches, and their schemes do not correspond to the estimation of the missing highfrequency components. Thus, these methods do not always provide the optimal solutions. On the other hand, the proposed method can provide the optimal estimation results if the target local patches can be represented in the obtained eigenspaces, correctly. This is the biggest difference between our method and the conventional methods.
Furthermore, we analyze our method in detail as follows.
In this derivation, Approximation 1 is used once. The criterion ${\stackrel{\u0303}{C}}^{k}$ represents the squared error calculated between the lowfrequency components l^{ k }reconstructed with the highfrequency components h^{ k }by cluster k's nonlinear eigenspace and the known original lowfrequency components l.
from the estimation results. However, we cannot calculate the above equation since the true highfrequency components h are unknown. There will always be a finite value for the last term h  h^{ k }^{2}. However, since h is unknown, we cannot know this term, and thus some assumptions become necessary. Thus, we assume that this term is constant, i.e., if we set h  h^{ k }^{2} = 0, the result will not change. Therefore, we set h  h^{ k }^{2} = 0 and calculate the minimum errors ${\stackrel{\u0303}{C}}^{k}$ of ${\stackrel{\u0303}{C}}_{\mathsf{\text{true}}}^{k}$. This means the proposed method utilizes the minimum errors caused in the HR result estimated by the inverse projection which can optimally provide the original image for the elements of each cluster. Then the proposed method utilizes the error ${\stackrel{\u0303}{C}}^{k}$ in Equation 45 as the criterion for the classification. In the previously reported method based on KPCA [8], they only applied the simple kmeans method to the known lowfrequency components for the clustering and the classification. Thus, this approach is quite independent of the KPCAbased reconstruction scheme, and there is no guarantee of providing the optimal clustering and classification results. On the other hand, the proposed method derives all of the criteria for the clustering and the classification from the KPCAbased reconstruction scheme. Therefore, it can be expected that this difference between the previously reported method and our method provides a solution to the outlier problem.
From the above explanation, we can see ${\stackrel{\u0303}{C}}^{k}$ in Equation 45 is a suitable criterion for classifying the target local patch into the optimal cluster k^{opt}. Then, the proposed method regards ${{\mathbf{h}}^{k}}^{{\mathsf{\text{opt}}}^{}}$ estimated by the selected cluster k^{opt} as the output, and $\mathbf{l}+{\mathbf{h}}^{{k}^{\mathsf{\text{opt}}}}$ becomes the estimated vector of the target HR patch g.
As described above, it becomes feasible to reconstruct the HR patches from the optimal cluster in the proposed method. Finally, we clip local patches (w × h pixels) at the same interval $(\stackrel{\u0303}{w}\times \stackrel{\u0303}{h}\phantom{\rule{2.77695pt}{0ex}}\mathsf{\text{pixels)}}$ from the blurred HR image $\widehat{F}$ and reconstruct their corresponding HR patches. Note that each pixel has multiple reconstruction results if the clipping interval is smaller than the size of the local patches. In such a case, the proposed method outputs the result minimizing Equation 45 as the final result. Then, the adaptive SR can be realized by the proposed method.
5 Experimental results
where s corresponds to the magnification factor, and we set L = 12. In these figures, we simply enlarged the LR images to the size of the original images. When we estimate an HR result from its LR image, the other two HR images and Boat, Girl, Mandrill are utilized as the training data. In the proposed method, we simply set its parameters as follows: w = 8, h = 8, $\stackrel{\u0303}{w}=8$, $\stackrel{\u0303}{h}=8$, Th = 0.9, ${\sigma}_{\mathbf{l}}^{2}$ is 0.075 times the variance of l_{ i } l_{ j }^{2} (i, j = 1, 2, . . . , N), and K = 7. Note that the parameters ${\sigma}_{\mathbf{l}}^{2}$ and K seem to affect the performance of the proposed method. Thus, we discuss the determinations of these two parameters and their sensitivities in Appendix. In this experiment, we applied the previously reported methods and the proposed method to Lena, Peppers, and Goldhill and obtained their HR results, where the magnification factor was set to four. For comparison, we adopt the method utilizing the sinc interpolation, which is the same filter used in the downsampling process and the most traditional approach, and three previously reported methods [8, 11, 27]. Since the method in [11] is a representative method of the examplebased superresolution, we utilized this method in the experiment. Furthermore, the method [27] is also a representative method which utilizes KPCA for performing the superresolution, and its improvement is achieved by utilizing the classification scheme in [8]. Therefore, these two methods are suitable for the comparison to verify the proposed KPCAbased method including the novel classification approach. In addition, the methods in [12, 28] have been proposed for realizing accurate SR. Therefore, since these methods can be regarded as stateoftheart ones, we also adopted them for comparison of the proposed method.
In the previously reported methods, the obtained HR images tend to be blurred in edge and texture regions. In detail, the proposed method keeps the sharpness in edge regions of test image Lena as shown in Figure 7. Furthermore, in the texture regions which are shown in Figure 8, the difference between the proposed method and the other methods becomes significant. Furthermore, in Figures 9 and 10, the center regions contain more highfrequency components compared with the other regions. Thus, the proposed method successfully reconstructs sharp edges and textures. As described above, test image Goldhill contains more highfrequency components than the other two test images, the difference of our method and the other ones is quite significant. Particularly, in Figure 11, roofs and windows can be successfully reconstructed with keeping sharpness by the proposed method. In addition, in Figure 12, the whole areas can be also accurately enhanced.
Some previously reported methods such as [12, 27] estimate one model for performing the SR. Then, if various kinds of training images are provided, it becomes difficult to successfully estimate the highfrequency components, and the obtained results tend to be blurred. Thus, we have to perform clustering of training patches in advance and reconstruct the highfrequency components by the optimal cluster. However, if the selection of the optimal cluster is not accurate, the estimation of the highfrequency components becomes also difficult. We guess that the limitation of the method in [8] occurs from this reason. The detailed analysis is shown later.
Note that our previously reported method [28] also includes the classification procedures, but its SR approach is different from our approach. This means the method in [28] performs the SR by interpolating new intensities between the intensities of LR images. Thus, the degradation model is different from that of this paper. Thus, it suffers from some degradation. On the other hand, the proposed method realizes the superresolution by estimating missing highfrequency components removed by the blurring in the downsampling process. In detail, the proposed method derives the inverse projection of the blurring process by using the nonlinear eigenspaces. Since the estimation of the inverse projection for the blurring process is an illposed problem, the proposed method performs the approximation of the blurring process in the lowdimensional subspaces, i.e., the nonlinear eigenspaces, and enables the derivation of its inverse projection.
Image reconstruction performance comparison (SSIM) of the proposed method and the previously reported methods
As described above, the MSE cannot reflect perceptual distortions, and its value becomes higher for images altered with some distortions such as mean luminance shift, contrast stretch, spatial shift, spatial scaling, and rotation, etc., yet negligible loss of subjective image quality. Furthermore, blurring severely deteriorates the image quality, but its MSE becomes lower than those of the above alternation. On the other hand, the SSIM index is defined by separately calculating the three similarities in terms of the luminance, variance, and structure, which are derived based on the human visual system (HVS) not accounted for by the MSE. Therefore, it becomes a better quality measure providing a solution to the above problem, and this is also confirmed in several researchers.
6 Conclusions
In this paper, we have presented an adaptive SR method based on KPCA with a novel texture classification approach. In order to obtain accurate HR images, the proposed method first performs clustering of the training HR patches and derives an inverse map for estimating the missing highfrequency components from the two nonlinear eigenspaces of training HR patches and their corresponding lowfrequency components in each cluster. Furthermore, the adaptive selection approach of the optimal cluster based on the errors caused in the estimation process of the missing highfrequency components enables each HR patch to be reconstructed successfully. Then, by combining the above two approaches, the proposed method realizes adaptive examplebased SR. Finally, the improvement of the proposed method over previously reported methods was confirmed.
In the experiments, the parameters of our method were set to simple values from some experiments. These parameters should be adaptively determined from the observed images. Thus, we need to complement this determination algorithm.
Appendix: Determination of parameters
Parameter of the Gaussian kernel ${\sigma}_{1}^{2}\left(=0.075\times \mathsf{\text{the}}\phantom{\rule{2.77695pt}{0ex}}\mathsf{\text{variance}}\phantom{\rule{2.77695pt}{0ex}}\mathsf{\text{of}}\phantom{\rule{2.77695pt}{0ex}}{\parallel {\mathbf{l}}_{i}{\mathbf{l}}_{j}\parallel}^{2}\right)$
From Figure 23, we can see the SSIM index almost monotonically increases with decreasing ${\sigma}_{\mathbf{l}}^{2}$. When the parameter of the Gaussian kernel is set to a larger value, the expression ability of local patches tends to become worse. On the other hand, if it is set to a smaller value, the overfitting tends to occur. Therefore, from this figure, we set the parameter of the Gaussian kernel as ${\sigma}_{\mathbf{l}}^{2}$ = 0.075 × the variance of l_{ i } l_{ j }^{2} since the performance of the proposed method for the three test images tends to become the highest. Note that this parameter is not so sensitive as shown in the results of Figure 23, i.e., the results are not sensitive to the parameter even if we set it to the larger or smaller values.
Number of clusters: K(= 7) From Figure 23, we can see the SSIM index of the proposed method becomes the highest value when K = 7 in several images, and the performance is not severely sensitive to the value of K. The parameter K is the number of clusters, and it should be set to the number of textures contained in the target image. However, since it is difficult to automatically find the number of textures in the target image, we simply set K = 7 in the experiments. The adaptive determination of the number of clusters will be the subject of the subsequent reports.
Abbreviations
 HR:

highresolution
 KPCA:

kernel principal component analysis
 LR:

lowresolution
 PCA:

principal component analysis
 SR:

superresolution.
Declarations
Acknowledgements
This work was partly supported by GrantinAid for Scientific Research (B) 21300030, Japan Society for the Promotion of Science (JSPS).
Authors’ Affiliations
References
 Park SC, Park MK, Kang MG: Superresolution image reconstruction: A technical overview. IEEE Signal Proces Mag 2003,20(3):2136. 10.1109/MSP.2003.1203207View ArticleGoogle Scholar
 Keys R: Cubic convolution interpolation for digital image processing. IEEE Trans Acoust Speech Signal Proces 1981,29(6):11531160. 10.1109/TASSP.1981.1163711MathSciNetView ArticleMATHGoogle Scholar
 Oppenheim AV, Schafer RW: DiscreteTime Signal Processing. 2nd edition. Prentice Hall, New Jersey; 1999.MATHGoogle Scholar
 Baker S, Kanade S, Kanade T: Limits on superresolution and how to break them. IEEE Trans Pattern Anal Mach Intell 2002,24(9):11671183. 10.1109/TPAMI.2002.1033210View ArticleGoogle Scholar
 Farsiu S, Robinson D, Elad M, Milanfar P: Advances and challenges in superresolution. Int J Imaging Syst Technol 2004,14(2):4757. 10.1002/ima.20007View ArticleGoogle Scholar
 van Ouwerkerk JD: Image superresolution survey. Image Vis Comput 2006,24(10):10391052. 10.1016/j.imavis.2006.02.026View ArticleGoogle Scholar
 Jiji CV, Chaudhuri S, Chatterjee P: Single frame image superresolution: should we process locally or globally? Multidimens Syst Signal Process 2007,18(23):123125. 10.1007/s1104500700241MathSciNetView ArticleMATHGoogle Scholar
 Hu Y, Shen T, Lam KM, Zhao S: A novel examplebased superresolution approach based on patch classification and the KPCA prior model. Comput Intell Secur 2008, 1: 611.Google Scholar
 Hertzmann A, Jacobs CE, Oliver BC N, Salesin DH: Image analogies. Comput Graph (Proc Siggraph) 2001, 2001: 327340.Google Scholar
 Freeman WT, Pasztor EC, Carmichael OT: Learning lowlevel vision. Int J Comput Vis 2000, 40: 2547. 10.1023/A:1026501619075View ArticleMATHGoogle Scholar
 Freeman WT, Jones TR, Pasztor EC: Examplebased superresolution. IEEE Comput Graph Appl 2002,22(2):5665. 10.1109/38.988747View ArticleGoogle Scholar
 Kanemura A, Maeda S, Ishii S: Sparse Bayesian learning of filters for efficient image expansion. IEEE Trans Image Process 2010,19(6):14801490.MathSciNetView ArticleGoogle Scholar
 Stephenson TA, Chen T: Adaptive Markov random fields for examplebased superresolution of faces. EURASIP J Appl Signal Process 2006, 2006: 225225.View ArticleGoogle Scholar
 Wang Q, Tang X, Shum H: Patch based blind image super resolution. Proceedings of ICCV 2005 2005, 1: 709716.Google Scholar
 Li X, Lam KM, Qiu G, Shen L, Wang S: An efficient examplebased approach for image superresolution. Proceedings of ICNNSP 2008, 2008: 575580.Google Scholar
 Sun J, Zheng N, Tao H, Shum H: Image hallucination with primal sketch priors. Proceedings of IEEE CVPR '03 2003, 2: 729736.Google Scholar
 Jiji CV, Joshi MV, Chaudhuri S: Singleframe image superresolution using learned wavelet coefficients. Int J Imaging Syst Technol 2004,14(3):105112. 10.1002/ima.20013View ArticleGoogle Scholar
 Jiji CV, Chaudhuri S: Singleframe image superresolution through contourlet learning. EURASIP J Appl Signal Process 2006,2006(10):111.View ArticleGoogle Scholar
 Wang X, Trang X: Hallucinating face by eigentransformation. IEEE Trans Syst Man Cybern 2005,35(3):425434. 10.1109/TSMCC.2005.848171View ArticleGoogle Scholar
 Chakrabarti A, Rajagopalan AN, Chellappa R: Superresolution of face images using kernel PCAbased prior. IEEE Trans Multimedia 2007,9(4):888892.View ArticleGoogle Scholar
 Schölkopf B, Smola A, Müller KR: Nonlinear principal component analysis as a kernel eigen value problem. Neural Comput 1998, 10: 12991319. 10.1162/089976698300017467View ArticleGoogle Scholar
 Schölkoph B, Mika S, Burges C, Knirsch P, Müller KR, Rätsch G, Smola A: Input space versus feature space in kernelbased methods. IEEE Trans Neural Netw 1999,10(5):10001017. 10.1109/72.788641View ArticleGoogle Scholar
 Chaudhuri S, Joshi MV: MotionFree SuperResolution. Springer, New York; 2005.MATHGoogle Scholar
 Turk M, Pentland A: Eigenfaces for recognition. J Cogn Neurosci 1991, 3: 7186. 10.1162/jocn.1991.3.1.71View ArticleGoogle Scholar
 Bishop C, Blake A, Marthi B: Superresolution enhancement of video. In Proceedings of 9th International Workshop on Artificial Intelligence and Statistics (AISTATS '03). Key West; 2003.Google Scholar
 Kim KI, Kwon Y: Examplebased learning for singleimage superresolution. Proceedings of the 30th DAGM Symposium on Pattern Recognition. Lecture Notes in Computer Science 2008, 456465.Google Scholar
 Kim KI, Schölkopf B: Iterative kernel principal component analysis for image modeling. IEEE Trans Pattern Anal Mach Intell 2005,27(9):13511366.View ArticleGoogle Scholar
 Ogawa T, Haseyama M: Missing intensity interpolation using a kernel PCAbased POCS algorithm and its applications. IEEE Trans Image Process 2011,20(2):417432.MathSciNetView ArticleGoogle Scholar
 Datsenko D, Elad M: Examplebased single document image superresolution: a global MAP approach with outlier rejection. Multidimens Syst Signal Process 2007,18(23):103121. 10.1007/s110450070018zMathSciNetView ArticleMATHGoogle Scholar
 Elad M, Datsenko D: Examplebased regularization deployed to superresolution reconstruction of a single image. Comput J 2009, 52: 1530.View ArticleGoogle Scholar
 Kwok JTY, Tsang IWH: The preimage problem in kernel methods. IEEE Trans Neural Netw 2004,15(6):15171525. 10.1109/TNN.2004.837781View ArticleGoogle Scholar
 Wang Z, Bovik AC, Sheikh HR, Simoncelli EP: Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 2004,13(4):600612. 10.1109/TIP.2003.819861View ArticleGoogle Scholar
 Avcbas I, Sankur B, Sayood K: Statistical evaluation of image quality measures. J Elec Imaging 2003,11(2):206223.Google Scholar
 Staelin C, Greig D, Fischer M, Maurer R: Neural network image scaling using spatial errors. Technical Report (HP Laboratories, Israel) 2003.Google Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.