Kclustervalued compressive sensing for imaging
 Mai Xu^{1}Email author and
 Jianhua Lu^{1}
DOI: 10.1186/16876180201175
© Xu and Lu; licensee Springer. 2011
Received: 18 February 2011
Accepted: 26 September 2011
Published: 26 September 2011
Abstract
The success of compressive sensing (CS) implies that an image can be compressed directly into acquisition with the measurement number over the whole image less than pixel number of the image. In this paper, we extend the existing CS by including the prior knowledge of Kcluster values available for the pixels or wavelet coefficients of an image. In order to model such prior knowledge, we propose in this paper Kclustervalued CS approach for imaging, by incorporating the Kmeans algorithm in CoSaMP recovery algorithm. One significant advantage of the proposed approach, rather than the conventional CS, is the capability of reducing measurement numbers required for the accurate image reconstruction. Finally, the performance of conventional CS and Kclustervalued CS is evaluated using some natural images and background subtraction images.
Keywords
compressive sensing Kmeans algorithm modelbased method1 Introduction
Image compression is currently an active research area, as it offers the promise of making the storage or transmission of images more efficient. The aim of image compression [1] is to reduce the data size of image and then make the image stored or transmitted in an efficient form. In image compression, we may transform the image into an appropriate basis and only store or transmit the important expansion coefficients [2]. Since such coefficients are normally sparse (only few coefficients are nonzero) or compressible (decaying rapidly according to power law), the compression (e.g., image compression in JEPG2000 [3]) can be achieved via storing and transmitting the nonzero coefficients.
For example, assume that we have acquired an image signal x∈ ℝ^{ N } with N pixels. Through DCT or wavelet transform, image x may be represented in terms of sets of coefficients via a basis expansion: x= Ψ α, where Ψ is an N × N basis matrix. Therefore, x may be represented by sparse coefficients $\mathit{\alpha}={\left\{{\alpha}_{i}\right\}}_{1}^{N}$, where S (≪ N ) coefficients are nonzero and then only these S coefficients with their locations need to be stored such that the compression can be achieved. Note that such α is defined as Ssparse. In practice, it is clear [4] that the natural images normally have compressible coefficients, decaying rapidly enough to zero when sorted, and thus can be approximated well as Ssparse.
A Compressive sensing
Equation 2 can be solved by a linear program within polynomial time [9]. For reducing the computational time, some other approaches have been proposed in the spirit of either greedy algorithms or combinatorial algorithms.
These include orthogonal matching pursuit (OMP) [10], StOMP [11], subspace pursuit [12] and CoSaMP [13].
It is attractive that CS is also applicable to images with sparse or compressible coefficients in the transform domain since y can be written as y= ΦΨ x, in which ΦΨ can be seen as M × N measurement matrix. In the sequel, without generality loss we shall focus on the images, sparse or compressible in the pixel domain. However, in our experiments of Section 4, we shall also consider the images, compressible in wavelet domain.
B Basic idea
Beyond CS, most recently, various extensions of CS have been proposed. CS, at heart, utilizes the prior knowledge of the sparsity of signal to compress the signal. Actually, some signals, such as digital images, have some prior knowledge other than sparsity. For example, we know that the nonzero coefficients of images usually cluster together, and a modelbased CS was thus proposed in [14–16] to integrate the prior knowledge of signal structure in CS for reducing the amount of measurements required for the recovery of images. However, to our best knowledge, all the stateoftheart modelbased CS approaches only concentrate on the prior knowledge of the locations of nonzero value pixels in digital images and assume that all the N pixel values of a digital image ∈ ℝ^{ N }. For example, [17] proposed (S, C)modelbased CS for reconstructing the Ssparse signal with the prior knowledge of block sparsity model in which there are at most C clusters with respect to the locations of nonzero coefficients of the signal. This approach is applicable to some practical problems such as MIMO channel equalization. However, in some other applications, the values of the sparse signal rather than nonzerovalued locations cluster together. Therefore, in this paper, we consider sparse signals with the prior knowledge of Kclustervalued coefficients, either in the canonical (pixel) domain or in the wavelet domain.
As a matter of fact, it has been shown [18] that for the most digital images, the intensities of each pixel are usually the subspace^{a} of [0, 255]. The motivation of this paper is thereby to extend the modelbased CS theory to include such prior knowledge. Then, we propose a reconstruction approach based on Kclustervalued intensities for CS (called Kclustervalued CS) to incorporate Kmeans algorithm in CS for recovering the images using only K clusters of nonzero intensity values, {μ_{1}, μ_{2},..., μ_{ K }} ⊆ [1, 255]. Once the measurement number M is less than required 4S for image compression with CS, there may exist several unreasonable solutions to the estimation of target image. However, during the reconstruction procedure, the proposed Kclustervalued CS avoids the possibility of intensity values being assigned beyond K clusters: {μ_{1}, μ_{2},..., μ_{ K }}, and it thus may be capable of discarding those unreasonable solutions. So, Kclustervalued CS is possible to reduce the number of measurements M required for robust image recovery. Note that in a gray image even we set cluster number K to be 255 at limit, Kclustervalued CS can still avoid the recovered intensity values of each pixel to be greater than 255 or less than 0 in conventional CS. Since our proposed Kclustervalued CS is an extension of modelbased CS, we shall briefly review the modelbased CS in the following section.
2 Overview of modelbased CS
Modelbased CS [14] incorporates some other prior knowledge rather than the sparsity or compressibility of signal in CS. It is intuitive that the restriction of such an additive prior knowledge may decrease the redundancy of measurements in CS, and the reduction in measurement number M of CS may therefore be possible.
In order to introduce the modelbased CS, let us first consider the modelbased restricted isometry property (RIP). Here, we define structured S sparsity model ${\mathcal{M}}_{S}$ as the union of m_{ S } subspaces subjective to x_{0} ≤ S. Thus, the prior knowledge of the Ssparse signals can be encoded in ${\mathcal{M}}_{S}$. Then, RIP of [19] can be rewritten as

An M × N matrix Φ has the ${\mathcal{M}}_{S}$restricted isometry property with constant ${\delta}_{{\mathcal{M}}_{S}}$ for all $\mathit{x}\in {\mathcal{M}}_{S}$, we have$\left(1{\delta}_{{\mathcal{M}}_{S}}\right)\parallel \mathit{x}{\parallel}_{2}^{2}\phantom{\rule{0.3em}{0ex}}\le \phantom{\rule{0.3em}{0ex}}\parallel \mathbf{\Phi}\mathit{x}{\parallel}_{2}^{2}\phantom{\rule{0.3em}{0ex}}\le \left(1+{\delta}_{{\mathcal{M}}_{S}}\right)\parallel \mathit{x}{\parallel}_{2}^{2}$(3)
where c is a positive constant, then Φ has ${\mathcal{M}}_{S}$restricted isometry property with the probability at least 1  e^{t}given constant ${\delta}_{{\mathcal{M}}_{S}}$. It can be seen from Equation 4 that as number of m_{ S } increases, more measurement numbers will be required for recovering the target signal, and modelbased CS increasingly resembles conventional CS, becoming equivalent to conventional CS at limit m_{ S } of being $\left(\begin{array}{c}\hfill N\hfill \\ \hfill S\hfill \end{array}\right)$. It satisfies the intuition that the more prior knowledge we have (such that m_{ S } decreases), the less measurement number is required for target signal recovery.
Then, the prior knowledge can be encoded in algorithm $M$ in advance. Given such an algorithm, the recovery method CoSaMP [13] may be extended for modelbased CS (See Algorithm 1 of [14] for the summary of modelbased CoSaMP). Also, note that there is no difference between conventional CS and modelbased CS in the measuring/sampling step summarized in Equation 1.
for ${\mathcal{M}}_{S}$RIP constant ${\delta}_{{\mathcal{M}}_{4S}}\le 0.1$. In Equation 6, ε is the noises additive to the measurements, i is the iteration number and ${\widehat{\mathit{x}}}^{i}$ is the estimated $\widehat{\mathit{x}}$ at the i th iteration. This equation guarantees the error of modelbased CS to be the same as conventional CS.
In a word, on the basis of prior knowledge encoded in advance, the modelbased CS is capable of reducing the measurement numbers without increasing any error bound.
3 Kclustervalued CS
The Kmeans algorithm [20] ensures that the clusters of data with the same or similar values can be identified in the same data set. So, Kmeans algorithm can be applied to the algorithm of Equation 7 using at most K = 255 clusters of nonzero intensities to reconstruct the target image at each iteration of recovery algorithm in CS. However, in practice, since most digital images have less than 255 clusters of nonzero intensities (the statistical analysis will be presented in the last part of this section), K is normally set to be less than 255 for image reconstruction. Even though K is the rough estimation of real cluster numbers, Kmeans algorithm still works in modelbased CS due to the fact that the goal of Kmeans algorithm is to minimize ${\sum}_{n}\parallel {x}_{n}{\widehat{x}}_{n}\parallel $, where ${\widehat{x}}_{n}\in \left\{{\mu}_{1},\phantom{\rule{2.77695pt}{0ex}}{\mu}_{2},\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}{\mu}_{K}\right\}$ with μ_{ k } being the center of k th cluster.
As assumed above, ${\widehat{\mathit{x}}}^{i}=\left\{{\widehat{x}}_{1}^{i},{\widehat{x}}_{2}^{i},\phantom{\rule{2.77695pt}{0ex}}\dots ,{\widehat{x}}_{N}^{i}\right\}$ are the estimated gray values of all pixels in the image at the i th iteration of recovery algorithm for CS. Then, we have the prior knowledge that all the nonzero values of ${\widehat{\mathit{x}}}^{i}$ can be replaced at iteration i by K clusters of intensity values $\left\{{\mu}_{1}^{i},\phantom{\rule{2.77695pt}{0ex}}{\mu}_{2}^{i},\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}{\mu}_{K}^{i}\right\}$, in which each ${\mu}_{k}^{i}\in \left[1,255\right]$. Our aim then is to partition the nonzero values of $\left\{{\widehat{x}}_{1}^{i},{\widehat{x}}_{2}^{i},\phantom{\rule{2.77695pt}{0ex}}\dots ,{\widehat{x}}_{N}^{i}\right\}$ into K (≤ 255) clusters^{b}, at each iteration of CS reconstruction step. To this end, we may apply Kmeans algorithm [20] in modelbased CS for target image recovery.
 1.Maximization: Since Equation 8 is a linear function of r_{ nk }, this optimization can be easily solved by setting r_{ nk } to be 1 once k makes $\parallel {\widehat{x}}_{n}^{i}{\mu}_{k}^{i}{\parallel}_{2}$ minimum. In another word, each nonzero ${\widehat{x}}_{n}^{i}$ is assigned to the closest ${\mu}_{k}^{i}$. So, this may be represented as${r}_{nk}^{i}=\left\{\begin{array}{cc}\hfill 1\hfill & \hfill \phantom{\rule{1em}{0ex}}if\phantom{\rule{1em}{0ex}}k=arg{min}_{j}\parallel {\widehat{x}}_{n}^{i}{\mu}_{k}^{i}{\parallel}_{2}\phantom{\rule{2.77695pt}{0ex}}and\phantom{\rule{2.77695pt}{0ex}}{\widehat{x}}_{n}^{i}\ne 0\hfill \\ \hfill 0\hfill & \hfill \phantom{\rule{1em}{0ex}}otherwise\hfill \end{array}\right.$(9)
 2.Expectation: In this stage, r_{ nk } has been fixed such that Equation 8 can be minimized with respect to ${\mu}_{k}^{i}$ by setting its derivative to be 0:$2\sum _{n=1}^{N}{r}_{nk}^{i}\left({\widehat{x}}_{n}^{i}{\mu}_{k}^{i}\right)=0$(10)
The above two stages are then repeated until reaching at convergence. However, it may converge to a local minimization rather than global minimization. Therefore, a good initialization procedure can reduce the oscillations and improve the performance of the proposed approach. Fortunately, we have the prior knowledge that for an image, $\left\{{\mu}_{1}^{i},\phantom{\rule{2.77695pt}{0ex}}{\mu}_{2}^{i},\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}{\mu}_{K}^{i}\right\}\subseteq \left\{1,2,\phantom{\rule{2.77695pt}{0ex}}\dots ,255\right\}$. Hence, in the proposed Kclustervalued CS, the initial values of ${\mu}_{k}^{i}$ may be chosen randomly from [1, 255]. Then, the iterations of the two stages of Kmeans are run until there is trivial change in objective function J in Equation 8 or until some maximum number of iterations (100 as set in Section 4) is exceeded.
Summary of the Kclustervalued CoSaMP algorithm
 Input: Measurement matrix Φ, measurement vector y, sparsity level S, and intensity cluster number K. 

 Output: Ssparse approximation $\widehat{\mathit{x}}$ of target image x. 
 Initialization: ${\widehat{\mathit{x}}}^{0}=0$, r= y and i = 1. 
While halting criterion = true 
1 z ← Φ*r{Compute the proxy of residual} 
2 Ω ← supp(z_{2K}) {Identify the largest 2K components of the proxy} 
3 $\mathit{T}\leftarrow \mathbf{\Omega}\cup supp\left({\widehat{\mathit{x}}}^{\left(i1\right)}\right)$ {Merge supports} 
4 $\mathit{b}{}_{\mathit{T}}\leftarrow {\mathbf{\Phi}}_{\mathit{T}}^{\u2020}\mathit{y}$ and $\mathit{b}{}_{{\mathit{T}}^{C}}\leftarrow 0${Estimate the image by leastsquares solution} 
5 ${\widehat{\mathit{x}}}^{i}\leftarrow {\mathit{b}}_{K}$ {Prune to obtain the image approximation for the next iteration or output} 
While ${\sum}_{n=1}^{N}{\sum}_{k=1}^{K}{r}_{nk}\parallel {\widehat{x}}_{n}^{i}{\mu}_{k}^{i}{\parallel}_{2}<threshold$ 
6 For each n = 1, 2,..., N, {Assign intensity values of each pixel to the closest intensity cluster} 
${r}_{nk}^{i}=\left\{\begin{array}{cc}\hfill 1\hfill & \hfill if\phantom{\rule{2.77695pt}{0ex}}k=arg{min}_{j}\parallel {\widehat{x}}_{n}^{i}{\mu}_{k}^{i}{\parallel}_{2}\phantom{\rule{1em}{0ex}}and\phantom{\rule{2.77695pt}{0ex}}\phantom{\rule{2.77695pt}{0ex}}{\widehat{x}}_{n}^{i}\ne 0\hfill \\ \hfill 0\hfill & \hfill otherwise\hfill \end{array}\right.$ 
7 For each $n=1,2,\phantom{\rule{2.77695pt}{0ex}}\dots ,\phantom{\rule{2.77695pt}{0ex}}N,\phantom{\rule{2.77695pt}{0ex}}{\mu}_{k}^{i}=\frac{{\Sigma}_{n=1}^{N}{r}_{nk}^{i}{\widehat{x}}_{n}^{i}}{{\Sigma}_{n=1}^{N}{r}_{nk}^{i}}$. {Obtain the K cluster centres} 
End 
8 For n = 1, 2,..., N, if r_{ nk } = 1, then ${\widehat{x}}_{n}^{i}={\mu}_{k}^{i}$ {Optimize the estimated image with Kcluster intensities} 
9 $\mathit{r}=\mathit{y}\mathbf{\Phi}{\widehat{\mathit{x}}}^{i}$ {Update the measurement residual for the next iteration} 
10 i = i + 1 {Update the iteration number} 
End 
return $\widehat{\mathit{x}}$${\widehat{\mathit{x}}}^{i}$ 
At each iteration, not only the measurement residual but also EM is applied for estimating the target signal. Since there are only a few clusters of intensity values for the target image, it is able to reduce the error caused by assigning unreasonable estimated values (e.g., more than 255 clusters) to each pixel at each iteration. Although the estimation error of the image may influence the clustering accuracy at each iteration, the minimization of Equation 8 also makes such influence minimal and the proposed approach converges after a few iterations. The computational time reported in section 4 also reveals that the robust of clustering.
4 Experimental results
In this section, experiments were performed for validating the proposed Kclustervalued CS. For comparison, we also applied the conventional CS to exactly the same images. In all the experiments, we utilized random Gaussian matrix as the measurement matrix Φ on the images. For conventional CS and Kclustervalued CS, the maximum iteration numbers of CoSaMP were both set to be 30. Besides, the iterations can also be halted once $\parallel {\widehat{\mathit{x}}}^{i}{\widehat{\mathit{x}}}^{i1}{\parallel}_{2}\phantom{\rule{0.3em}{0ex}}<1{0}^{2}\parallel {\widehat{\mathit{x}}}^{i}{\parallel}_{2}$. For Kclustervalued CS, the iterative two stages of Kmeans algorithm are repeated 100 times. The experiments have been performed under the following system environments: Matlab R2008b on a computer with Pentium(R) D 2.8GHz CPU and 3GB RAM. Section A focuses on utilizing the Kclustervalued and conventional CSs to compress one lunar image relying on a canonical (pixel) sparsity basis. This subsection shows the results in detail. In Section B, we demonstrate the experiments on other extensive images in brief. This subsection mainly concentrates on the 2D images, using either a canonical sparsity basis or wavelet sparsity basis, as the input to our experiments. In addition, the experiments on some background subtracted images in color are demonstrated as well.
A One experiment in detail
From the viewpoint of computational time, as can been seen from Figure 4, Kclustervalued CS runs faster than conventional CS when cluster number K is small (e.g., K = 2, 5 and 10). It may be due to the fast convergence of Kclustervalued CoSaMP caused by more accurate recovery result at each iteration.
where N is the number of pixels at each image and 255 is the dynamic range of intensities of the image. x and $\widehat{\mathit{x}}$ are the intensities of the original image and compressed image, respectively.
B More experiments in general
In this subsection, we evaluated our proposed approach on three different images sets: (1) the image set chosen from Caltech 101 database contains five images, sparse in pixel domain; (2) the natural image set contains four images, sparse or compressible in wavelet domain; (3) background subtracted color image set.
The PSNRs of reconstructed images of Figure 7
Methods  (a)  (b)  (c)  (d) 

Conventional CoSaMP  22.61  22.10  20.85  21.67 
Kclustervalued CoSaMP  31.03  27.39  28.20  26.95 
5 Conclusions
In this paper, in order to compress the image, we have aimed to propose an advanced modelbased CS, named Kclustervalued CS, which utilizes Kmeans algorithm as the model for CS. In contrast to conventional CS, the proposed Kclustervalued CS incorporates the prior knowledge that only K clusters values of intensities are available for all the pixels of an image. In this paper, we also investigated cluster number K as prior knowledge. Such prior knowledge goes beyond the simple sparsity/compressibility of CS and therefore has the advantage in using fewer measurements than conventional CS for accurate image reconstruction. This way, Kclustervalued CS is applicable to other Kclustervalued signals (e.g., binary digital signals) besides the images. Also, it is applicable to other modelbased CS by considering all the prior knowledge together. Moreover, the experiments were performed and presented to validate the proposed approach.
Endnotes
^{a}This is the usual case since an image is normally comprised of a few categories of objects with limited color intensities as exploited in computer vision community. ^{b}The K clusters can also be applied in color image by extending each gray value ${\widehat{x}}_{n}^{i}$ to be 3 space comprising the intensities of the red, blue and green channels. ^{c}It is due to the fact that conventional CS does not have any prior knowledge of the range of intensity values of the pixels.
Declarations
Acknowledgements
This work was partially supported by China National Basic Research Program (973) under Grant number 2007CB310600 and partially supported by NSFC 30971689.
Authors’ Affiliations
References
 Petrou M, Petrou C: Image Processing: The Fundamentals. 2nd edition. Wiley, Amsterdam; 2010.View Article
 Mallat S: A Wavelet Tour of Signal Processing. Academic Press, New York; 1999.
 Taubman D, Marcellin M: JPEG 2000: Image Compression Fundamentals, Standards and Practice. 1st edition. Kluwer Academic, Dordrecht; 2001.
 Candès E, Wakin M: An introduction to compressive sampling. IEEE Signal Process Mag 2008,25(2):2130.View Article
 Candès E, Romberg J: T Tao, Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans Inf Theory 2006,52(02):489509.View Article
 Candès E, Tao T: Nearoptimal signal recovery from random projections: Universal encoding strategies? IEEE Trans Inf Theory 2006,52(12):54065425.View Article
 Donoho D: Compressed sensing. IEEE Trans Inf Theory 2006,52(4):12891306.MathSciNetView Article
 Candès E, Romberg J, Tao T: Stable signal recovery from incomplete and inaccurate measurements. Commun Pure Appl Math 2006,59(8):12071223. 10.1002/cpa.20124View Article
 Boyd S, Vandenberghe L: Convex Optimization. Cambridge University Press, Cambridge; 2004.View Article
 Tropp J, Gilbert A: Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans Inf Theory 2007,53(12):267288.MathSciNetView Article
 Donoho DL, Tsaig Y, Drori I, luc Starck J: Sparse solution of underdetermined linear equations by stagewise orthogonal matching pursuit. Tech Rep 2006, (March):139.
 Dai W, Milenkovic O: Subspace pursuit for compressive sensing signal reconstruction. IEEE Trans Inf Theory 2009,55(5):267288.
 Needell D, Tropp J: Cosamp: iterative signal recovery from incomplete and inaccurate samples. Appl Comput Harmon Anal 2009,26(3):301321. 10.1016/j.acha.2008.07.002MathSciNetView Article
 Baraniuk R, Cevher V, Duarte M, Hegde C: Modelbased compressive sensing. IEEE Trans Inf Theory 2010,56(4):19822001.MathSciNetView Article
 Eldar Y, Kuppinger P, Bolcskei H: Compressed sensing of blocksparse signals: uncertainty relations and efficient recovery. IEEE Trans Signal Process 2009, in press.
 Cevher V, Duarte M, Hegde C, Baraniuk R: Sparse signal recovery using markov random fields. Proceedings of NIPS 2008.
 Cevher V, Indyk P, Chinmay H, Baraniuk R: Recovery of clustered sparse signals from compressive measurements. Proceedings of Sampta09 2009.
 Bishop CM: Pattern Recognition and Machine Learning. 1st edition. Springer, New York; 2006.
 Blumensath T, Davies M: Sampling theorems for signals from the union of finitedimensional linear subspaces. IEEE Trans Inf Theory 2008,55(4):18721882.MathSciNetView Article
 Lloyd SP: Least squares quantization in pcm. IEEE Trans Inf Theory 1982,28(2):129136. 10.1109/TIT.1982.1056489MathSciNetView Article
 Thomos N, Boulgouris N, Strintzis M: Optimized transmission of jpeg2000 streams over wireless channels. IEEE Trans Image Process 2006,15(1):5467.View Article
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.