Dependent Gaussian mixture models for source separation

Abstract

Source separation is a common task in signal processing and is often analogous to factor analysis. In this study, we look at a factor analysis model for source separation of multi-spectral image data where prior information about the sources and their dependencies is quantified as a multivariate Gaussian mixture model with an unknown number of factors. Variational Bayes techniques for model parameter estimation are used. The development of this methodology is motivated by the need to bring an efficient solution to the separation of components in the microwave radiation maps that are being obtained by the satellite mission Planck which has the objective of uncovering cosmic microwave background radiation. The proposed algorithm successfully incorporates a rich variety of prior information available to us in this problem in contrast to many previous solutions that assume completely blind separation of the sources. Results on realistic simulations of Planck maps and on Wilkinson microwave anisotropy probe fifth year images are shown. The technique suggested is easily applicable to other source separation applications by modifying some of the priors.

Introduction

The discovery of the cosmic microwave background (CMB) is a strong evidence for the Big Bang theory of the formation and development of the universe. According to the theory, the early universe was smaller and hotter but cooled as it expanded. Once the temperature cooled to about 3000 K, photons were free to propagate without being scattered off ionized matter; the CMB is an image of this event and is visible across the entire sky. Three satellites have been launched to measure the CMB: the cosmic background explorer, Wilkinson microwave anisotropy probe (WMAP) and most recently the Planck surveyor. Planck is the highest resolution data to date, of the order of 10⁷ pixels across the sky measured at nine channels.

Unfortunately, the signals measured by these satellites as shown in Figure 1 contain radiation not only from CMB, but also contributions from a number of other sources, namely foreground radiations and extragalactic sources in addition to antenna receiver noise. Foreground sources from our galaxy include synchrotron, dust, and free–free emission. Therefore, the separation of the CMB signal from other sources is an important stage in the production of CMB maps [1].

Figure 1

Observed WMAP 7 year-data. The data were taken from the NASA WMAP website [2].

To date, there have been several attempts to achieve it in a Bayesian framework using both (a) Gaussian mixture model (GMM) prior [3], and (b) Markov Random Field (MRF) prior [4, 5]. Full sky maps at low resolution through MCMC, using masks to reduce the effect of the signal in the galactic plane, were described in [6]. Some of these are fully Bayesian source separation methods which are developed to separate the underlying CMB from the mixed observed signals of extraterrestrial microwaves made at several frequencies.

A common assumption among works in the literature is the independence of the cosmological sources. Although it is well known that CMB is independent from the rest of the sources, the galactic sources demonstrate significant statistical dependence among themselves, as stated in [1]. Recently, a small number of researchers have started addressing this problem [7, 8]. Various dependent component analysis approaches are compared in [9], demonstrating their superior performance with respect to classical ICA.

In this study, we present a dependent components model for source separation of multi-spectral image data, where prior information about the sources and between-source dependencies is quantified as a multivariate GMM, using variational Bayes techniques for model parameter estimation. This article can thus be considered as an extension of [3], modeling dependencies between-sources through generalizing the prior to multivariate GMM.

The rest of the article is structured as follows. The next section gives the model for the mixing problem and describes the hierarchical Bayesian model that we use, including the prior we assume for the sources. Section “Implementing the source separation” describes the variational Bayes approach we use for the implementation of the separation. Section “Examples” provides results on both synthetic Planck and real WMAP images. Finally, we provide a discussion of the results in the last section.

Model

The model description is defined in terms of the microwave source separation problem, where there are n _f maps of the sky at frequencies $({\nu }_1,\dots ,{\nu }_{n_f})$, each map consisting of J pixels. The data are denoted ${\mathbf{d}}_j\in {\mathbb{R}}^{n_f},j=1,\dots ,J$. The source model consists of n _s sources and is represented by the vectors ${\mathbf{s}}_j\in {\mathbb{R}}^{n_s}$, with each component representing the amplitude of a physical source of microwaves. We assume that the d _j can be represented as a linear combination of the s _j :

$${\mathbf{d}}_j=\mathbf{A}{\mathbf{s}}_j+{\mathbf{e}}_j,$$

(1)

where A is an n _f  × n _s “mixing” matrix and e _j is a vector of n _f independent Gaussian error terms with precisions (inverse variances) $\mathbf{\tau }=({\tau }_1,\dots ,{\tau }_{n_f})$. For convenience, define

$$\begin{array}{lcr}\mathbf{D}& =& \left\{d_{\mathrm{ij}}\right|i=1,\dots ,n_f,j=1,\dots ,J\};\\ \mathbf{S}& =& \left\{s_{\mathrm{kj}}\right|k=1,\dots ,n_s,j=1,\dots ,J\}\end{array}$$

to represent all data and sources.

We assume dependence between the sources, defined by a prior distribution p(S|ψ) with parameters ψ. The goal is to estimate the S and the parameters ψ associated with the model for S, given observation of D. The noise variances τ and the mixing matrix A are assumed known. GMM are used to represent the non-Gaussian sources, in which case it is an example of a model known as a mixture of factor analyzers [10]. As in [10], we adopt a Bayesian approach to the data fitting, implemented by a variational Bayes approach.

Bayesian inference will be based on the posterior distribution, which following the above description can be factorized as

$$p(\mathbf{S},\mathit{\psi }|\mathbf{A},\mathbf{D},\mathbf{\tau })\propto p(\mathbf{D}|\mathbf{S},\mathbf{A},\mathbf{\tau })p\left(\mathbf{S}\right|\mathit{\psi }\left)p\right(\mathit{\psi }).$$

(2)

Each element of this distribution is defined next in turn.

Noise structure

Gaussian error, e _j , is assumed independent within and between pixels j and frequency, which gives

$$p\left(\mathbf{D}\right|\mathbf{S},\mathbf{A},\mathbf{\tau })=\prod _{j=1}^J\prod _{i=1}^{n_f}\sqrt{\frac{{\tau }_i}{2\Pi }}\text{exp}\left(-\frac{{\tau }_i}{2}{(d_{\mathrm{ij}}-{\mathbf{A}}_{i·}{\mathbf{s}}_j)}^2\right)$$

(3)

where A_i· is the i th row of A.

Mixing matrix structure

In this application, A is parameterized and denoted A(θ). Each column of A(θ) is the contribution to the observation of a source at different frequencies, which is written as a function of the frequencies and θ. These parameterizations are approximations that come from the current state of knowledge about how the sources are generated. Here, we merely state the parameterization that we are going to use, and refer to [11] for a more detailed exposition on the background to them. Some restrictions are usually placed on A(θ)in order to force a unique solution; this is achieved here by setting the first row of A(θ) to be ones.

It is assumed that the CMB is the first source and therefore it corresponds to the first column of A(θ). It is modeled as a black body at a temperature, and its contribution is a known constant at each frequency. The parametrization of the mixing matrix is given as

$${\mathbf{A}}_{i1}\left(\theta \right)=\frac{g\left({\nu }_i\right)}{g\left({\nu }_1\right)},$$

(4)

where

$$g\left({\nu }_i\right)={\left(\frac{h{\nu }_i}{k_B{T}_0}\right)}^2\frac{\text{exp}(h{\nu }_i/k_B{T}_0)}{{\left(\text{exp}\right(h{\nu }_i/k_B{T}_0)-1)}^2},$$

T₀ = 2.725K is the average CMB temperature, h is the Planck constant, and k_B is Boltzmann’s constant. The ratio g(ν _i ) / g(ν₁) is designed to ensure that A₁₁(θ) = 1 as we constraint the first row of A(θ) to be ones.

$$\begin{array}{ccc}{\mathbf{A}}_{i2}\left(\theta \right)\hfill & =\hfill & {\left(\frac{{\nu }_i}{{\nu }_1}\right)}^{{\kappa }_s},\hfill \\ {\mathbf{A}}_{i3}\left(\theta \right)\hfill & =\hfill & \frac{\text{exp}(h{\nu }_1/k_{\mathrm{B}}{T}_1)-1}{\text{exp}(h{\nu }_i/k_{\mathrm{B}}{T}_1)-1}{\left(\frac{{\nu }_i}{{\nu }_1}\right)}^{1+{\kappa }_d},\text{and}\hfill \\ {\mathbf{A}}_{i4}\left(\theta \right)\hfill & =\hfill & {\left(\frac{{\nu }_i}{{\nu }_1}\right)}^{{\kappa }_f},\hfill \end{array}$$

where T₁ = 18.1K is the assumed thermodynamical temperature of the dust grains, and column 2 corresponds to synchrotron, column 3 to galactic dust, and column 4 is free–free emission. There are three unknown model parameters for A, for synchrotron ${\kappa }_s\in \{{\kappa }_s:-3.0\le {\kappa }_s\le -2.3\}$, the spectral indices for dust κ _d  ∈ {κ _d  : 1 ≤ κ _d  ≤ 2}, and for free–free emission ${\kappa }_f\in \{{\kappa }_f:-2.3\le {\kappa }_f\le -2.0\}$.

The sources

The distribution of s _j is modeled as a GMM with m factors. The model proposed allows for between-source dependence; the vector of sources at a pixel is a mixture of multivariate Gaussians

$$p\left(\mathbf{S}\right|\mathit{\psi })=\prod _{j=1}^J\sum _{a=1}^m{w}_{a}p({\mathbf{s}}_j|{\mathbf{\mu }}_a,Q_a)$$

(5)

where

$$p\left({\mathbf{s}}_j\right|{\mathbf{\mu }}_a,Q_a)=\sqrt{\frac{\left|Q_a\right|}{{\left(2\Pi \right)}^{n_s}}}\text{exp}\left(-\frac{1}{2}{({\mathbf{s}}_j-{\mathbf{\mu }}_a)}^T{Q}_a({\mathbf{s}}_j-{\mathbf{\mu }}_a)\right)$$

for mixture component weights w _a , mean vectors μ _a , and precision matrices Q _a , so that ψ is all the w _a , μ _a and Q _a , with a = 1,…,m. Note that, in the standard inflationary cosmological model the CMB is a single multivariate Gaussian (m = 1) while the Galactic foregrounds might require m > 1 to be correctly modeled. In order to fulfill this, we set the CMB for components 2,…,m of the multivariate mixture to be exactly zero in the implementation.

Priors

The remaining term in Equation (2) is p(ψ). We use the conjugate prior distributions [12] that facilitate the computation of the posterior and yet flexible enough to incorporate good prior information: Gaussians for the component means, Dirichlet for the component weights, and Wishart for precision matrices. In the microwave source application, background knowledge about the magnitude of the sources can be incorporated through specifying values of the parameters of these prior distributions. This prior specification follows [13], who discuss how to specify these values in more detail.

Implementing the source separation

The posterior developed in the previous section does not lend itself to an analytical solution. MCMC techniques are one approach that let us evaluate complicated integrals by sampling rather than by analytical or numerical methods. The main criticism of Bayesian source separation with sampling methods, MCMC in particular, is their computational load and slow convergence. Regarding the speed, they cannot compete with methods such as FastICA [14, 15].

There are several approaches to speed up the algorithm, such as the strategies suggested in [16]. In the image source separation problem framework, the Langevin sampling scheme has been implemented [4], as a way to obtain a faster MC algorithm.

In this study, the source separation model presented in Section “Model” is implemented by a variational Bayesian approach [10, 17, 18], that allows for more efficient inference when dealing with large data when compared with MCMC techniques. In essence, given the data D and a model with parameters θ and latent variables Z, the variational Bayes method is based on approximating the posterior distribution p(Z,θ|D) with a factorial approximation q(Z,θ|ϕ) = q(Z|ϕ _Z )q(θ|ϕ _θ ), where ϕ are the variational parameters. The approximation is fitted by minimizing the Kullback–Leibler divergence between q and p, or equivalently maximizing a lower bound on marginal log-likelihood of the data.

Attias [19] has recently developed a fully Bayesian approach to GMM with a variational approximation to the posterior that, when choosing conjugate priors, leads to the following components: Wishart densities for the precisions, Q _a ; Normal densities for the means, μ _a ; and a Dirichlet for the mixing coefficients, p; and a discrete distribution for the indicator posteriors, z _j , which indicates the component that explains information in pixel j. We further derived the variational approximation to the marginal posterior of sources, s _j , which turns out to be a multivariate Gaussian distribution. In brief

$$\begin{array}{ccc}q\left({\mathbf{s}}_j\right)\hfill & \sim \hfill & \text{MVN}(A_j^{\star },B_j^{\star })\hfill \\ q\left(\mathbf{p}\right)\hfill & \sim \hfill & D\left(\mathbf{\lambda }\right)\hfill \\ q\left({\mathbf{\mu }}_a\right|Q_a)\hfill & \sim \hfill & N({\mathbf{\xi }}_a,{\beta }_a{Q}_a)\hfill \\ q\left(Q_a\right)\hfill & \sim \hfill & W({\mathbf{\eta }}_a,V_a)\hfill \end{array}$$

and

$$\begin{array}{ll}q(z_j& =a)\propto \text{exp}\left(\Psi \left({\lambda }_a\right)-\Psi \left(\sum _{a^{\prime }}{\lambda }_{a^{\prime }}\right)\right)|V_a{|}^{\frac{1}{2}}{2}^{\frac{n_s}{2}}\\ \text{exp}\left(\frac{1}{2}\sum _{i=1}^{n_s}\Psi \left(\frac{{\eta }_a+1-i}{2}\right)\right)\text{exp}\left(-\frac{n_s}{2{\beta }_a}\right)\\ \text{exp}\left(-\frac{{\eta }_a}{2}\left({(A_j^{\star }-{\xi }_a)}^T{V}_a(A_j^{\star }-{\xi }_a)+\mathrm{tr}\left(V_a{\left(B_j^{\star }\right)}^{-1}\right)\right)\right)\end{array}$$

where MVN stands for multivariate normal distribution and Ψ denotes the digamma function. Note that q(z _j  = a) is the probability that component a is responsible for information in pixel j in sources, s _j .

The quantities of interest, i.e., the hyper-parameters to be computed, $A_j^{\star }$, $B_j^{\star }$, λ, ξ _a , β _a , η _a , and V _a for j = 1,…,J and a = 1,…,m, have the following values:

$$\begin{array}{ccc}{\left(B_j^{\star }\right)}_{\mathrm{kl}}\hfill & =\hfill & \sum _{i=1}^{n_f}{\tau }_i{A}_{\mathrm{ik}}{A}_{\mathrm{il}}+\sum _{a=1}^{m}q(z_j=a){\eta }_a{\left(V_a\right)}_{\mathrm{kl}}\hfill \\ {\left(A_j^{\star }\right)}_k\hfill & =\hfill & {\left(B_j^{\star }\right)}^{-1}\mathbf{v}\left(k\right),\text{with}\hfill \\ \mathbf{v}\left(k\right)\hfill & =\hfill & \sum _{i=1}^{n_f}{\tau }_i{d}_{\mathrm{ij}}{A}_{\mathrm{ik}}+\sum _{a=1}^{m}q(z_j=a){\eta }_a\sum _{l=1}^{n_s}{\left(V_a\right)}_{\mathrm{kl}}{\left({\xi }_a\right)}_l\hfill \\ {\lambda }_a\hfill & =\hfill & \sum _{j}q(z_j=a)+{\lambda }_a^{\text{prior}}\hfill \\ {\xi }_a\hfill & =\hfill & \frac{\sum _j\left[q\right(z_j=a\left)A_j^{\star }\right]+{\beta }_a^{\text{prior}}{\xi }_a^{\text{prior}}}{\sum _{j}q(z_j=a)+{\beta }_a^{\text{prior}}}\hfill \\ {\beta }_a\hfill & =\hfill & \sum _{j}q(z_j=a)+{\beta }_a^{\text{prior}}\hfill \\ {\eta }_a\hfill & =\hfill & \sum _{j}q(z_j=a)+{\eta }_a^{\text{prior}}\hfill \end{array}$$

$$\begin{array}{ccc}{V}_a\hfill & =\hfill & {\sum }_j\left\{q(z_j=a)\left[\left(1-\frac{2q(z_j=a)}{{\sum }_{j}q(z_j=a)}\right){\left(B_j^{\star }\right)}^{-1}+\Phi \right.\right.\hfill \\ \left.\left.+(A_j^{\star }-{\bar{\mu }}_a){(A_j^{\star }-{\bar{\mu }}_a)}^T\right]\right\}+V_a^{\text{prior}}\hfill \\ +\frac{{\beta }_a^{\text{prior}}\left[{\sum }_{j}q\right(z_j=a\left)\right][\Phi +({\bar{\mu }}_a-{\xi }_a^{\text{prior}}){({\bar{\mu }}_a-{\xi }_a^{\text{prior}})}^T}{{\beta }_a}\hfill \end{array}$$

with

$$\begin{array}{ccc}{\bar{\mu }}_a\hfill & =\hfill & \frac{{\sum }_j\left[q\right(z_j=a\left)A_j^{\star }\right]}{{\sum }_{j}q(z_j=a)}\hfill \\ {(\Phi )}_{\mathrm{kl}}\hfill & =\hfill & \frac{{\sum }_j\left\{{\left[q\right(z_j=a\left)\right]}^2{\left(B_j^{\star }\right)}_{\mathrm{kl}}^{-1}\right\}}{{\left[{\sum }_{j}q\right(z_j=a\left)\right]}^2}.\hfill \end{array}$$

Computations were carried out using Matlab.

Examples

Analysis of simulated data

The synthetic Planck data were generated using the Planck Sky Model (PSM) package. The PSM is a set of IDL codes created by the Planck WG2 team in order to provide realistic simulations of the sky at the Planck frequencies [20]. Figure 2 shows data obtained from realistic simulations of CMB, synchrotron, and galactic dust on a 512×512 patch. The original sources are shown in Figure 3. The data were generated at the nine frequencies that are observed by Planck from 30 to 857 GHz. The mixing matrix used was as defined in Section “Mixing matrix structure” with κ _s  = −2.9 and κ _d  = 2.0. Noise precisions were those published by the Planck research team [21]. After exploring several values for m, the number of components in the GMM source model was fixed to be m = 1, as it provided the best fit, taking into account the compromise between fit and number of parameters in the model.

Figure 2

Observations on the three of the nine channels (lowest, middle, and highest frequencies are shown) of the data generated from the source separation model with realistic simulations of CMB, synchrotron, and galactic dust.

Figure 3

The simulated sources used to generate the simulated data in Figure 2 .

Figure 4 shows an estimate of CMB, along with a scatter plot of this estimate against the true value, as shown in Figure 3. Such an estimate is the average of the samples obtained for the first column of A^⋆, which corresponds to CMB. We see from the scatter plot and from comparison with Figure 3 that the reconstruction of CMB is very accurate here. The same is true for the other two sources, as shown in Figure 5.

Figure 4

The posterior mean of the reconstruction of the CMB with a scatter plot of true versus posterior mean.

Figure 5

The posterior mean of the reconstruction of synchrotron and galactic dust.

Table 1 shows the mean of the parameters of the model. Regarding the between-sources dependence structure, posterior estimates of $V_{1k}^{-1},k=2,3$ are approximately 0, suggesting independence between CMB and the other sources, as expected. On the other hand, posterior estimate of $V_{23}^{-1}\ne 0$, indicating dependence between synchrotron and galactic dust.

Table 1 Mean estimate of parameters for simulated data

Analysis of a WMAP year 5 patch

The WMAP [22] was launched in 2001 and data collection activities finished in 2010. It observes 5 frequencies from 22 to 90 GHz. Figure 6 shows a patch of 5-year WMAP data.

Figure 6

Temperatures (in mK) at 20° square patch of the sky from WMAP[2]at 5 microwave frequencies (clockwise from top left) 22, 30, 40, 90, and 60 GHz).

The algorithm was implemented with four sources (CMB, synchrotron, dust, and free–free emission). The noise precisions were assumed to be the published values for WMAP detectors. The spectral density for free–free emission was fixed at −2.14 (following [11]) and the synchrotron and dust spectral indices were as in the first example. The number of components in the GMM source model were fixed to be m = 2, following the same reasoning as in the simulation study. Informative priors were placed on the GMM parameters, based on discussions on the expected marginal properties of the sources. Table 2 shows the mean of the mixture parameters of the model. Figure 7 shows the estimated CMB. The result obtained is in agreement with previous work [3], as can be appreciated in Figure 8 that shows an histogram of the differences between the estimated CMB using the approach presented here and the estimated CMB obtained in [3].

Figure 7

Estimated CMB.

Figure 8

Histogram of the (pixel-by-pixel) differences between the estimated CMB using the approach presented in this article and the estimated CMB obtained in [3]. The result obtained is in agreement with previous work

Table 2 Parameter estimated mean for WMAP data

Finally, in order to show the fit of the data to the model, Figure 9 is a scatter plot of the observed value of the d _jk with the standardized residuals, with one figure for each frequency k = 1,…,5.

Figure 9

Assessment of model fit. Scatter plot of the posterior predicted values of d _j against the standardized residual over all pixels.

Conclusion

A fully Bayesian factor analysis algorithm has been presented and applied to a multi-channel image source separation problem, where dependencies between sources are modeled as a multivariate GMM. The algorithm performs very well on simulated Planck data and has been applied to data from WMAP.

In this study, we extend previous approaches [3] by allowing the source priors to be a mixture of multivariate Gaussian distributions for each pixel.

The development of this methodology is motivated by the need to bring an efficient solution to the separation of components in the microwave radiation maps to be obtained by the satellite mission Planck which has the objective of uncovering CMB radiation. The proposed algorithm successfully incorporates a rich variety of prior information available to us in this problem in contrast to most of the previous work that assumes completely blind separation of the sources. Further, the variational approach presented here overcomes the convergence problems of the MCMC stated in [23], when dealing with large datasets such as that will be provided by the satellite mission Planck.

In the analysis of simulated data, the number of components in the GMM source model turned out to be m = 1. This means that sources are multivariate Gaussian a priori. On the other hand, for real data, the number of components is m = 2. In blind source separation problem, identifiability relies on the independence of the sources. In this study, in spite of modeling the sources as Gaussians when m = 1, identifiability is obtained because of the prior information which is incorporated to the model, given structure to the mixing matrix.

Another type of dependence is that a source is spatially correlated. Spatial dependence is most conveniently modeled by a Gaussian MRF and some preliminary work on this idea can be found in [5]. Combining with cross source correlations, one might ultimately consider a mixture of multivariate Gaussian MRF as a prior for the sources. Implementing the analysis with such a prior would be a significant challenge computationally; we hypothesize that it will be difficult to derive a well-behaved MCMC approach. Other functional approximations, such as that of [24], offer feasible alternative to computing the posterior distribution in this case.

Finally, although the technique was developed for the astrophysical source separation problem in mind, it is general and it is applicable to other source separation problems as well.