Bayesian approach with prior models which enforce sparsity in signal and image processing

Abstract

In this review article, we propose to use the Bayesian inference approach for inverse problems in signal and image processing, where we want to infer on sparse signals or images. The sparsity may be directly on the original space or in a transformed space. Here, we consider it directly on the original space (impulsive signals). To enforce the sparsity, we consider the probabilistic models and try to give an exhaustive list of such prior models and try to classify them. These models are either heavy tailed (generalized Gaussian, symmetric Weibull, Student-t or Cauchy, elastic net, generalized hyperbolic and Dirichlet) or mixture models (mixture of Gaussians, Bernoulli-Gaussian, Bernoulli-Gamma, mixture of translated Gaussians, mixture of multinomial, etc.). Depending on the prior model selected, the Bayesian computations (optimization for the joint maximum a posteriori (MAP) estimate or MCMC or variational Bayes approximations (VBA) for posterior means (PM) or complete density estimation) may become more complex. We propose these models, discuss on different possible Bayesian estimators, drive the corresponding appropriate algorithms, and discuss on their corresponding relative complexities and performances.

1 Introduction

In many generic inverse problems in signal and image processing we want to infer on an unknown signal f(t) or an unknown image f(r) with r= (x, y) through an observed signal g(s) or an observed image g(s) related between them through an operator $\mathscr{H}$ such as convolution g = h * f or any other linear or non linear transformation $g=\mathscr{H}f$. When this relation is linear and we have discretized the problem, we arrive to the relation:

$$\mathit{g}=\mathit{H}\mathit{f}+\epsilon ,$$

(1)

where f= [f₁, ..., f _n ]' represents the unknowns, g= [g₁, ..., g _m ]' the observed data, ϵ= [ϵ₁, ..., ϵ _m ]' the errors of modeling and measurement and H the matrix of the system response. We may note that, even if the noise could be neglected (ϵ= 0) and the matrix H invertible (m = n), in general, the solution $\hat{\mathit{f}}={\mathit{H}}^{-1}\mathit{g}$ is not forcibly the good solution, because this solution may be too sensitive to small changes in the data due to the ill-conditioning of this matrix. for the general case of m ≠ n, one tries to obtain a regularized solution, for example by defining it as the optimizer of a two parts criterion

$$\hat{\mathit{f}}=\text{arg}\underset{f}{\text{min}}\left\{J\left(\mathit{f}\right)={∥\mathit{g}-\mathit{H}\mathit{f}∥}^2+\lambda {∥\mathit{f}∥}^2\right\}$$

(2)

which is given by $\hat{\mathit{f}}={\left[{\mathit{H}\mathit{H}}^{\prime }+\lambda \mathit{I}\right]}^{-1}{\mathit{H}}^{\prime }\mathit{g}$. When the regularization parameter λ = 0, one gets a generalized inverse $\hat{\mathit{f}}={\left[{\mathit{H}\mathit{H}}^{\prime }\right]}^{-1}{\mathit{H}}^{\prime }\mathit{g}$ and when H invertible, one gets the normal inverse solution $\hat{\mathit{f}}={\mathit{H}}^{-1}\mathit{g}$. The regularization theory has been developed since the pioneer work of Tikhonov [1] and Tikhonov and Arsénine [2] who had introduced a quadratic regularization terms to account for some prior properties of the solution (smoothness). Since that, many different regularization terms have been proposed. In particular, in place of L₂ norm: $L_2\left(\mathit{f}\right)={∥\mathit{f}∥}_2^2={\sum }_j{\left|f_j\right|}^2$, it has been proposed to use the L₀ norm $L_0\left(\mathit{f}\right)={∥\mathit{f}∥}_0={\sum }_j\delta \left(f_j\right)$ or the L₁ norm L₁(f) = ||f||₁ = Σ _j |f _j | to enforce the sparsity of the solution [3–11]. Then, due to the fact that L₀(f) is not convex and L₁(f) is convex, but not continuous, the optimization of a criterion with these expressions becomes more difficult than the L₂ norm case. For this reason, there was a great number of works who specialized in proposing algorithms for the optimization of such criteria.

Interestingly, defining the solution of the problem (1) as the optimization of a criterion with two parts can be assimilated to a maximum a posteriori (MAP) solution in a Bayesian approach where the first term of the criterion (2) can be related to the likelihood and the second term to a prior model as we will see in the following where the main objective is to show how the Bayesian approach can go farther than the regularization in at least the following aspects:

• A better account for the noise term characteristics;

• A better and easier way for translating the prior knowledge and in particular the sparsity;

• New tools for assessing the regularization parameter, a great subject of discussion for all those work with regularization theory;

• New solutions and new tools for doing computations (optimizations and integrations).

1.1 The Bayesian approach

The Bayesian inference approach is based on the posterior law:

$$p(\mathit{f}|\mathit{g},{\mathit{\theta }}_1,{\mathit{\theta }}_2)=\frac{p(\mathit{g}|\mathit{f},{\mathit{\theta }}_1)p(\mathit{f}|{\mathit{\theta }}_2)}{p(\mathit{g}|{\mathit{\theta }}_1,{\mathit{\theta }}_2)}\propto p(\mathit{g}|\mathit{f},{\mathit{\theta }}_1)p(\mathit{f}|{\mathit{\theta }}_2)$$

(3)

where the sign ∝ stands for "proportional to", p(g|f, θ₁) is the likelihood, p(f|θ₂) the prior model, θ= (θ₁, θ₂) are their corresponding parameters (often called the hyper parameters of the problem) and p(g|θ₁, θ₂) is called the evidence of the model.

This general Bayesian approach is illustrated as follows:

In this approach, the likelihood p(g|f, θ₁) summarizes our knowledge about the noise and the model linking the observed data g to the unknowns f and the prior term p(f|θ₂) summarizes our incomplete prior knowledge about the unknowns and the posterior law p(f|g, θ) combines these two terms and contains all our state of knowledge about the unknowns f after accounting for the prior and the observed data.

As a very simple example, when the noise is assumed to be Gaussian, then the MAP solution $\hat{\mathit{f}}=\text{arg}\underset{\mathit{f}}{\text{max}}\left\{p\left(\mathit{f}\left|\mathit{g},\theta \right.\right)\right\}$ is obtained as the optimizer of the criterion J(f) = ||g- Hf||² + λ Ω(f) where the expression of Ω(f) depends on the prior law. When the prior knowledge is translated as a Gaussian probability law, then $\Omega \left(\mathit{f}\right)={∥\mathit{f}∥}_2^2$ and when it is translated as a Laplace probability law, then Ω(f) = ||f||₁ [12–14].

The first interest of using the Bayesian approach to the regularization approach is to have new tools for handling the hyper parameters [15].

1.2 Full Bayesian approach

When the parameters θ have to be estimated too, we can assign them a prior p(θ|θ₀) with fixed values for θ₀ (often called hyper-hyper-parameters) and express the joint posterior

$$p\left(\mathit{f},\theta \left|\mathit{g},{\theta }_0\right.\right)=\frac{p\left(\mathit{g}\left|\mathit{f},{\theta }_1\right.\right)p\left(\mathit{f}\left|{\theta }_2\right.\right)p\left(\theta \left|{\theta }_0\right.\right)}{p\left(\mathit{g}\left|{\theta }_0\right.\right)}$$

(4)

and then try to estimate them jointly, for example joint MAP [16]:

$$\left(\hat{\mathit{f}},\hat{\theta }\right)=\text{arg}\underset{\left(\mathit{f},\theta \right)}{\text{max}}\left\{p\left(\mathit{f},\theta \left|\mathit{g},{\theta }_0\right.\right)\right\}$$

(5)

This Full Bayesian approach is illustrated as follows:

One may also first integrate out one of them, for example f to obtain

$$p\left(\theta \left|\mathit{g},{\theta }_0\right.\right)=\int p\left(\mathit{f},\theta \left|\mathit{g},{\theta }_0\right.\right)\mathsf{\text{d}}\mathit{f},$$

(6)

estimate θ, for example by

$$\hat{\theta }=\text{arg}\underset{\theta }{\text{max}}\left\{p\left(\theta \left|\mathit{g},{\theta }_0\right.\right)\right\}$$

(7)

and then use it for the estimation of the other one using $p\left(\mathit{f}\left|\mathit{g},\hat{\theta }\right.\right)$.

This approach (called sometimes type II maximum likelihood) is illustrated as follows:

However, very often this marginalization cannot be done analytically and so the optimization for the estimation of θ cannot be achieved. In such cases, the expectation-maximization (EM) algorithms can be helpful [17]. Considering g as incomplete data, f as hidden variable, (g, f) as complete data and noting ln p(g|θ) as incomplete data log-likelihood and ln p(g, f|θ) complete data log-likelihood, the classical EM algorithm writes:

$$\left\{\begin{array}{c}\mathsf{\text{E - step}}:q\left(\theta ,{\hat{\theta }}^{\left(k\right)}\right)={\mathsf{\text{E}}}_{p\left(\mathit{f}\left|\mathit{g},\right.{\hat{\theta }}^{\left(k\right)}\right)}\left\{\text{ln}p\left(\mathit{g},\mathit{f}\left|\theta \right.\right)\right\}\hfill \\ \mathsf{\text{M - step:}}{\hat{\theta }}^{\left(k\right)}=\text{arg}{\text{max}}_{\theta }\left\{q\left(\theta ,{\hat{\theta }}^{\left(k-1\right)}\right)\right\}\hfill \end{array}\right.$$

(8)

The Bayesian version (Bayesian EM) is not very far and differs only by the introduction of p(θ):

$$\left\{\begin{array}{c}\mathsf{\text{E - step}}:q\left(\theta ,{\hat{\theta }}^{\left(k\right)}\right)={\mathsf{\text{E}}}_{p\left(\mathit{f}\left|\mathit{g},{\hat{\theta }}^{\left(k\right)}\right.\right)}\left\{\text{ln}p\left(\mathit{g},\mathit{f}\left|\theta \right.\right)+\text{ln}p\left(\theta \right)\right\}\hfill \\ \mathsf{\text{M - step}}:{\hat{\theta }}^{\left(k\right)}=\text{arg}{\text{max}}_{\theta }\left\{q\left(\theta ,{\hat{\theta }}^{\left(k-1\right)}\right)\right\}\hfill \end{array}\right.$$

(9)

This is illustrated as follows:

As we mentioned before, one of the main steps in the Bayesian approach is the prior modeling which has the role of translating our prior knowledge on the unknown signal or image in a probability law. Sparsity is one of the prior knowledge we may translate. The main objective of this article is to see what are the different possibilities.

1.3 Prior modeling

In this article, we propose different prior modeling for signals and images which can be used in a Bayesian inference approach in many inverse problems in signal and image processing where we want to infer on sparse signals or images. The sparsity may be directly on the original space or in a transformed space (see Figures 1, 2, 3, and 4). In this article, we consider the sparsity directly in the original domain.

Figure 1

Sparsity: explicite sparse signals. The signal at the right is sparse, but its derivative (signal at the left) is still more sparse.

Figure 2

Sparsity: sparse signals in a transformed domaine (Fourier or wavelet). First row: signals, second row: Fourier or wavelet transforms.

Figure 3

Sparsity: explicite sparse images. The images at the top are sparse. The images at the bottom are not sparse, but their Laplaciens are (images at top).

Figure 4

Sparsity: sparse images in a transformed domain (Fourier or wavelet). First row: images, second row: Fourier or wavelet transforms.

The prior models discussed are the following:

• generalized Gaussian (GG) with Gaussian (G) and Laplace or double exponential (DE) as particular cases;

• symmetric Weibull (W) with symmetric Rayleigh (R) and again the DE as particular cases;

• Student-t (St) with Cauchy (C) as particular case;

• Elastic net prior model;

• generalized hyperbolic model;

• Dirichlet and symmetric Dirichlet;

• Mixture of two centered Gaussians (MoG2), one with very small and one with a large variances;

• Bernoulli-Gaussian (BG), also called Spike and slab;

• Mixture of two Gammas (MoGamm);

• Bernoulli-Gamma (BGamma);

• Mixture of three Gaussians (MoG3), one centered with very small variance and two symmetrically centered on positive and negative axes and large variances;

• Mixture of one Gaussian and two Gammas (MoGGammas), and in a more summary the case of

• Bernoulli-Multinomial (BMult) or mixture of Dirichlet (MoD).

Some of these models are well-known [12–14, 18–26], some others less. In general, we can classify them into two categories: (i) simple non Gaussian models with heavy tails and (ii) mixture models with hidden variables which result to hierarchical models.

In the Section 2, we give more details about the sparsity and all these prior models which enforce the sparsity.

1.4 Bayesian computation

The second main step in the Bayesian approach is to do the computations. Depending on the prior model selected, the Bayesian computations needed are:

• For simple prior models:

  • Simple optimization of p(f|θ, g) for the MAP:

    

  • Joint optimization p(f, θ|g) for joint MAP:

    

  • Generation of samples from the conditionals p(f|θ, g) and p(θ|f, g) for the MCMC Gibbs sampling methods,

    

  • Variational approximation (VA) of the joint p(f, θ|g) by a separable

    $$q\left(\mathit{f},\theta \left|\mathit{g}\right.\right)=q_1\left(\mathit{f}\left|\tilde{\theta }\right.,\mathit{g}\right)q_2\left(\theta \left|\tilde{\mathit{f}},\mathit{g}\right.\right)$$

    and then using them for estimation

    

• For hierarchical prior models with hidden variables z:

  • Joint optimization p(f, z, θ|g) for joint MAP,

    

  • Generation of samples from the conditionals p(f|z, θ, g), p(θ|z, f, g) and p(z|f, θ, g) for the MCMC Gibbs sampling methods:

    

  • Variational approximation (VA) of the joint p(f, z, θ|g) by a separable

    $$q\left(\mathit{f},\mathit{z},\theta \left|\mathit{g}\right.\right)=q_1\left(\mathit{f}\left|\tilde{z},\tilde{\theta },\mathit{g}\right.\right)q_2\left(\mathit{z}\left|\tilde{\mathit{f}},\tilde{\theta },\mathit{g}\right.\right)q_3\left(\theta \left|\tilde{\mathit{z}},\tilde{\mathit{f}},\mathit{g}\right.\right)$$

    and then using them for estimation

    

The second main objective of this article is to discuss on the relative complexities and performances of the algorithms obtained with the proposed prior law.

The rest of the article is organized as follows:

In Section 2, we present in details the proposed prior models and discuss their properties. For example, we will see that the Student-t model can be interpreted as an infinite mixture with a variance hidden variable or that the BG model can be considered as the degenerate case of a MoG2 where one of the variances go to zero. Also, we will examine the less known models of MoG3 and MoGGammas where the heavy tails are obtained by combining a centered Gaussian and two large variance non-centered Gaussians or Gammas.

In Section 3, we examine the expression of the posterior laws that we obtain using these priors and discuss then on complexity of the Bayesian computation of the algorithms. In particular for the mixture models, we give details of the joint estimation of the signal and the hidden variable as well as the hyper parameters (parameters of the mixtures and the noise) for unsupervised cases.

In Section 4, we give more details on the variational Bayesian approximation method, first for the general case and then for the case of mixture laws and more specifically the case of the Student-t considered as a continuous mixture.

Finally, we present the main conclusions of this article in Section 5.

2 Prior models enforcing sparsity

First, as we mentioned, the sparsity is a property which can be described either directly for the signal itself or after some transformation, for example on the derivative of the signal, or in more general on the coefficients of the projection of the signal on any basis or any set of functions.

Different prior models have been used to enforce sparsity.

2.1 Generalized Gaussian (GG), Gaussian (G) and double exponentials (DE) models

This is the simplest and the most used model (see for example, [27]). Its expression is:

$$p\left(\mathit{f}\left|\gamma ,\beta \right.\right)=\prod _j\mathcal{G}\mathcal{G}\left(f_j\left|\gamma ,\beta \right.\right)\propto \text{exp}\left\{-\gamma \sum _j{\left|f_j\right|}^{\beta }\right\}$$

(10)

where

$$\mathcal{G}\mathcal{G}\left(f_j\left|\gamma ,\beta \right.\right)=\frac{\beta \gamma }{2\Gamma \left(1/\beta \right)}\text{exp}\left\{-\gamma {\left|f_j\right|}^{\beta }\right\}.$$

(11)

Two particular cases are of importance:

• β = 2 (Gaussian):

  $$\begin{array}{ll}\hfill p\left(\mathit{f}\left|\gamma \right.\right)=\prod _j\mathcal{N}\left(f_j\left|0,1/\left(2\gamma \right)\right.\right)& \propto \text{exp}\left\{-\gamma \sum _j{\left|f_j\right|}^2\right\}\\ \propto \text{exp}\left\{-\gamma {∥f∥}_2^2\right\}\end{array}$$

  (12)

• β = 1 (double exponential or Laplace):

  $$\begin{array}{ll}\hfill p\left(\mathit{f}\left|\gamma \right.\right)=\prod _j\mathcal{D}ℰ\left(f_j\left|\gamma \right.\right)& \propto \text{exp}\left\{-\gamma \sum _j\left|f_j\right|\right\}\\ \propto \text{exp}\left\{-\gamma {∥\mathit{f}∥}_1\right\}\end{array}$$

  (13)

The general shape of these priors are shown in Figure 5, where the cases β = 1 and 0 < β < 1, which are of great interest for sparsity enforcing are compared to the Gaussian case β = 2.

Figure 5

Generalized Gaussian family. The probability density function p(x) is shown in the left and - ln p(x) is shown in the right.

2.2 Symmetric Weibull (W) and symmetric Rayleigh (R) models

The second model we consider is the symmetric Weibull probability density function (pdf):

$$\begin{array}{ll}\hfill p\left(\mathit{f}\left|\gamma ,\beta \right.\right)& =\prod _j\mathcal{W}\left(f_j\left|\gamma ,\beta \right.\right)\\ \propto \text{exp}\left\{-\gamma \sum _j{\left|f_j\right|}^{\beta }+\left(\beta -1\right)\text{log}\left|f_j\right|\right\}\end{array}$$

(14)

where

$$\mathcal{W}\left(f_j\left|\gamma ,\beta \right.\right)=c{\left|f_j\right|}^{\left(\beta -1\right)}\text{exp}\left\{-\gamma {\left|f_j\right|}^{\beta }\right\}$$

(15)

and where γ > 0 and β > 0, and the particular cases of β = 1 is the double exponential and β = 2 is the symmetric Rayleigh distribution:

$$p\left(\mathit{f}\left|\gamma ,\beta \right.\right)=\prod _jℛ\left(f_j\left|\gamma \right.\right)\propto \text{exp}\left\{-\gamma \sum _j{\left|f_j\right|}^2+\text{log}\left|f_j\right|\right\}$$

(16)

the cases where 0 < β < 1 are of great interest for sparsity enforcing. This family of models are illustrated on Figure 6.

Figure 6

Symmetric Weibull family. The probability density function p(x) is shown in the left and - ln p(x) is shown in the right.

2.3 Student-t (St) and Cauchy (C) models

The second simplest model is the Student-t model:

$$p\left(\mathit{f}\left|\nu \right.\right)=\prod _j\mathcal{S}t\left(f_j\left|\nu \right.\right)\propto \text{exp}\left\{-\frac{\nu +1}{2}\sum _j\text{log}\left(1+f_j^2/\nu \right)\right\}$$

(17)

where

$$\mathcal{S}t\left(f_j\left|\nu \right.\right)=\frac{1}{\sqrt{\pi \nu }}\frac{\Gamma \left(\left(\nu +1\right)/2\right)}{\Gamma \left(\nu /2\right)}{\left(1+f_j^2/\nu \right)}^{-\left(\nu +1\right)/2}$$

(18)

Knowing that

$$\mathcal{S}t\left(f_j\left|\nu \right.\right)={\int }_0^{\infty }\mathcal{N}\left(f_j\left|0,1/{\tau }_j\right.\right)\mathcal{G}\left({\tau }_j\left|\nu \right./2,\nu /2\right)\mathsf{\text{d}}{\tau }_j$$

(19)

we can write this model via the positive hidden variables τ _j :

$$\begin{array}{ll}\hfill p\left(\mathit{f},\tau \right)& ={\prod }_{j}p\left(f_j\left|{\tau }_j\right.\right)={\prod }_j\mathcal{N}\left(f_j\left|0,1/{\tau }_j\right.\right)\\ \propto \text{exp}\left\{-\frac{1}{2}{\sum }_j{\tau }_j{f}_j^2\right\}\\ \hfill p\left({\tau }_j\left|a,b\right.\right)& =\mathcal{G}\left({\tau }_j\left|a,b\right.\right)\propto {{\tau }_j}^{\left(a-1\right)}\text{exp}\left\{-b{\tau }_j\right\}\\ \mathsf{\text{with}}a=b=\nu /2\end{array}$$

(20)

Cauchy model is obtained when ν = 1:

$$p\left(\mathit{f}\right)=\prod _j\mathcal{C}\left(f_j\right)\propto \text{exp}\left\{-\sum _j\text{log}\left(1+f_j^2\right)\right\}$$

(21)

This family of models are illustrated on Figure 7.

Figure 7

Student-t and Cauchy family. The probability density function p(x) is shown in the left and - ln p(x) is shown in the right.

2.4 Elastic Net (EN) prior model

A prior model inspired from elastic net regression literature [28] is:

$$p\left(\mathit{f}\left|\nu \right.\right)=\prod _jℰ\mathcal{N}\left(f_j\left|\nu \right.\right)\propto \text{exp}\left\{-\sum _j\left({\gamma }_1\left|f_j\left|+{\gamma }_2{f}_j^2\right.\right.\right)\right\}$$

(22)

where

$$ℰ\mathcal{N}(f_j|\nu )=\mathcal{N}(0,1/{\gamma }_1)\mathcal{D}ℰ({\gamma }_1)\propto \exp \left\{-{\gamma }_1\left|f_j\right|-{\gamma }_2{f}_j^2)\right\}$$

(23)

which is a product of a Gaussian and a double exponential pdfs. This family of models are illustrated on Figure 8.

Figure 8

Elastic net family. The probability density function p(x) is shown in the left and - ln p(x) is shown in the right.

2.5 Generalized hyperbolic (GH) prior model

Another general prior model which can be used is:

$$\begin{array}{ll}\hfill p\left(\mathit{f}\left|\delta ,\nu ,\beta \right.\right)& ={\prod }_j{\left({\delta }^2+f_j^2\right)}^{\left(\nu -1/2\right)/2}\text{exp}\left\{\beta x\right\}\\ K_{\nu -1/2}\left(\alpha \sqrt{{\delta }^2+f_j^2}\right)\end{array}$$

(24)

where K_ν-1/2is the second kind Bessel function of order (ν - 1/2). This family of models are illustrated on Figure 9.

Figure 9

Generalized hyperbolic family. The probability density function p(x) is shown in the left and - ln p(x) is shown in the right.

2.6 Dirichlet (D) and symmetric Dirichlet (SD) models

When f _j are positive and sums to one, we can use the Dirichlet model

$$\mathcal{D}\left(\mathit{f}\left|\alpha \right.\right)\propto \prod _j{f}_j^{{\alpha }_j-1}\mathsf{\text{with}}{f}_j>0,\sum _j{f}_j=1$$

(25)

where α= {α₁, ..., α _N } with α _j > 0. The proportionality constant is

$$B\left(\alpha \right)=\frac{{\prod }_j\Gamma \left({\alpha }_j\right)}{\Gamma \left({\sum }_j\Gamma \left({\alpha }_j\right)\right)}$$

(26)

It is noted that the support of this distribution is [0,1]^Nand ||f||₁ = Σ _j f _j = 1.

It is also interesting to note that the domain of the Dirichlet distribution is itself a probability distribution, specifically a N-dimensional discrete distribution and the set of points in the support of a N-dimensional Dirichlet distribution is the open standard N - 1-simplex, which is a generalization of a triangle, embedded in the next-higher dimension.

A very common special case is the symmetric Dirichlet (SD) distribution, where all of the elements making up the parameter vector α have the same value α called the concentration parameter:

$$\mathcal{D}\left(\mathit{f}\left|\alpha \right.\right)\propto \prod _j{f}_j^{\alpha -1}\mathsf{\text{with}}{f}_j>\mathsf{\text{0,}}\sum _j{f}_j=1$$

(27)

When α > 1, the symmetric Dirichlet distribution is equivalent to a uniform distribution over the open standard standard N - 1-simplex, i.e., it is uniform over all points in its support. α > 1 prefer variants that are dense, evenly-distributed distributions, i.e., all probabilities f _j returned are similar to each other. α < 1 prefer sparse distributions, i.e., most of the probabilities f _j returned will be close to 0, and the vast majority of the mass will be concentrated in a few of them. This is the case on which we are interested. An illustration of this family of models are illustrated on Figure 10.

Figure 10

Dirichlet family. The probability density function p(x) is shown in the left and - ln p(x) is shown in the right.

2.7 Mixture of two Gaussians (MoG2) model

The mixture models are also very commonly used as prior models. In particular the mixture of two Gaussians (MoG2) model:

$$p\left(\mathit{f}\left|\lambda ,v_1,v_0\right.\right)=\prod _j\left(\lambda \mathcal{N}\left(f_j\left|0,v_1\right.\right)+\left(1-\lambda \right)\mathcal{N}\left(f_j\left|0,v_0\right.\right)\right)$$

(28)

which can also be expressed through the binary valued hidden variables z _j ∈ {0,1}

$$\left\{\begin{array}{c}\begin{array}{c}p\left(\mathit{f}\left|\mathit{z}\right.\right)={\prod }_{j}p\left(f_j\left|z_j\right.\right)={\prod }_j\mathcal{N}\left(f_j\left|0,v_{z_j}\right.\right)\\ \propto \text{exp}\left\{-\frac{1}{2}{\sum }_j\frac{f_j^2}{v_{z_j}}\right\}\end{array}\hfill \\ P\left(z_j=1\right)=\lambda ,P\left(z_j=0\right)=1-\lambda \hfill \end{array}\right.$$

(29)

In general v₁ >> v₀ and λ measures the sparsity (0 < λ << 1). This family of models are illustrated on Figure 11.

Figure 11

Mixture of two Gaussians family. The probability density function p(x) is shown in the left and - ln p(x) is shown in the right.

2.8 Bernoulli-Gaussian (BG) model

The Bernoulli-Gaussian model can be considered as the particular case of the MoG2 with the particular degenerate case of v₀ = 0:

$$p\left(\mathit{f}\left|\lambda ,v\right.\right)=\prod _{j}p\left(f_j\right)=\prod _j\left(\lambda \mathcal{N}\left(f_j\left|0,v\right.\right)+\left(1-\lambda \right)\delta \left(f_j\right)\right)$$

(30)

which can also be written as

$$\left\{\begin{array}{c}\begin{array}{c}p\left(\mathit{f}\left|\mathit{z}\right.\right)={\prod }_{j}p\left(f_j\left|z_j\right.\right)={\prod }_j{\left[\mathcal{N}\left(f_j\left|0,v\right.\right)\right]}^{\delta \left(z_j\right)}\hfill \\ {{\prod }_j\left[\delta \left(f_j\right)\right]}^{\delta \left(1-z_j\right)}\hfill \end{array}\\ P\left(z_j=1\right)=\lambda ,P\left(z_j=0\right)=1-\lambda \end{array}\right.$$

(31)

This model has also been called spike and slab. This family of models are illustrated on Figure 12.

Figure 12

Bernouilli-Gaussian family. The probability density function p(x) is shown in the left and - ln p(x) is shown in the right.

2.9 Mixture of three Gaussians (MoG3) model

Another mixture model proposed is using a Mixture of three Gaussians, one centered at zero and two symmetrically placed:

$$\begin{array}{ll}\hfill p\left(\mathit{f}\left|\lambda ,v_0,v_{+1},v_{-1},\beta \right.\right)& ={\prod }_j\left[\left(1-\lambda \right)\mathcal{N}\left(f_j\left|0,v_0\right.\right)\right.\\ +\left(\lambda /2\right)\mathcal{N}\left(f_j\left|+\beta ,v_{+1}\right.\right)\\ \left.+\left(\lambda /2\right)\mathcal{N}\left(f_j\left|-\beta ,v_{-1}\right.\right)\right]\end{array}$$

(32)

which can also be expressed through the ternary valued hidden variables z _j ∈ {-1, 0, +1}

$$\left\{\begin{array}{c}p\left(\mathit{f}\left|\mathit{z}\right.\right)={\prod }_{j}p\left(f_j\left|z_j\right.\right)={\prod }_j\mathcal{N}\left(f_j\left|z_j\beta ,v_{z_j}\right.\right)\hfill \\ P\left(z_j=1\right)=\lambda /2,\hfill \\ \begin{array}{c}P\left(z_j=-1\right)=\lambda /2,\\ P\left(z_j=0\right)=1-\lambda .\end{array}\hfill \end{array}\right.$$

(33)

In general v₊₁ = v_-1 = v >> v₀ and λ measures the sparsity (0 < λ << 1). This family of models are illustrated on Figure 13.

Figure 13

Mixture of three Gaussians family. The probability density function p(x) is shown in the left and - ln p(x) is shown in the right.

2.10 Mixture of one Gaussian and two Gammas (MoGGammas) model

Another mixture model proposed is using a mixture of one central Gaussian and two symmetric Gammas:

$$\begin{array}{ll}\hfill p\left(\mathit{f}\left|\lambda ,v_0,\alpha ,\beta \right.\right)& ={\prod }_j\left[\left(1-\lambda \right)\mathcal{N}\left(f_j\left|0,v_0\right.\right)\right.\\ +\left(\lambda /2\right)\mathcal{G}\left(f_j\left|\alpha ,\beta \right.\right)\\ \left.+\left(\lambda /2\right)\mathcal{G}\left(-f_j\left|\alpha ,\beta \right.\right)\right]\end{array}$$

(34)

which can also be expressed through the ternary valued hidden variables z _j ∈ {-1, 0, +1}

$$\left\{\begin{array}{c}p\left(\mathit{f}\left|\mathit{z}\right.\right)={\prod }_j\left(f_j\left|z_j\right.\right)={\left[\mathcal{N}\left(f_j\left|0,v_0\right.\right)\right]}^{{\sum }_j\delta \left(z_j\right)}\times \hfill \\ {\left[\mathcal{G}\left(f_j\left|\alpha ,\beta \right.\right)\right]}^{{\sum }_j\delta \left(z_j-1\right)}\times \hfill \\ {\left[\mathcal{G}\left(-f_j\left|\alpha ,\beta \right.\right)\right]}^{{\sum }_j\delta \left(z_j+1\right)}\hfill \\ P\left(z_j=1\right)=\lambda /2,\hfill \\ P\left(z_j=-1\right)=\lambda /2,\hfill \\ P\left(z_j=0\right)=1-\lambda .\hfill \end{array}\right.$$

(35)

This family of models are illustrated on Figure 14.

Figure 14

Mixture of one Gaussian and two Gammas family. The probability density function p(x) is shown in the left and - lnp(x) is shown in the right.

2.11 Bernoulli-Gamma (BGamma) model

As in the BG model, when we want to enforce both sparsity and positivity, we can use the BGamma model:

$$p\left(\mathit{f}\left|\lambda ,\alpha \right.,\beta \right)=\prod _j\left[\lambda \delta \left(f_j\right)+\left(1-\lambda \right)\mathcal{G}\left(f_j\left|\alpha ,\beta \right.\right)\right]$$

(36)

or

$$\left\{\begin{array}{c}\begin{array}{c}p\left(\mathit{f}\left|\mathit{z}\right.\right)={\prod }_{j}p\left(f_j\left|z_j\right.\right)={\prod }_j\left[z_j\mathcal{G}\left(f_j\left|\alpha ,\beta \right.\right)\right]\\ {\prod }_j\left[\left(1-z_j\right)\delta \left(f_j\right)\right]\end{array}\hfill \\ P\left(z_j=1\right)=\lambda ,P\left(z_j=0\right)=1-\lambda \hfill \end{array}\right.$$

(37)

A particular case of this model is Bernoulli-exponential (BExponential) which obtained when α = 1. These families of models are illustrated on Figure 15 and Figure 16.

Figure 15

Bernouilli-Gamma family. The probability density function p(x) is shown in the left and - ln p(x) is shown in the right.

Figure 16

Mixture of 2 Gammas family. The probability density function p(x) is shown in the left and - ln p(x) is shown in the right.

2.12 Mixture of Dirichlet (MoD) model

• Mixture of Dirichlet model

  $$p\left(\mathit{f}\left|\lambda ,{\alpha }_1,{\alpha }_2\right.\right)=\lambda \mathcal{D}\left(\mathit{f}\left|{\alpha }_1\right.\right)+\left(1-\lambda \right)\mathcal{D}\left(\mathit{f}\left|{\alpha }_2\right.\right)$$

  (38)

where

$$\mathcal{D}\left(\mathit{f}\left|\alpha \right.\right)\propto \prod _j{f}_j^{\alpha -1}\mathsf{\text{with}}{f}_j>0,\sum _j{f}_j=1$$

(39)

is the symmetric Dirichlet distribution. We need to choose α₁ > 1 for dense part and 0 < α₂ < 1 for the sparse part.

2.13 Bernoulli-multinomial (BMultinomial) model

As in the BG or BGamma model, when we know that the signal is sparse and can only take one of the K discrete values {a₁, ..., a _K }, we can use the BMultinomial model:

$$p\left(\mathit{f}\left|\lambda ,\mathit{a},\alpha \right.\right)=\prod _j\lambda ℳult\left(f_j\left|\mathit{a},\alpha \right.\right)+\left(1-\lambda \right)\delta \left(f_j\right)$$

(40)

where a= {a₁, ..., a _K } and α= {α₁, ..., α _K } with ∑ _k α _k = 1 and

$$ℳult\left(f_j\left|\mathit{a},\alpha \right.\right)=\frac{n!}{a_1!\dots a_K!}\prod _k{\alpha }_k^{a_j}$$

or

$$\left\{\begin{array}{c}\begin{array}{c}p\left(\mathit{f}\left|\mathit{z}\right.\right)={\prod }_{j}p\left(f_j\left|z_j\right.\right)={\prod }_j\left[z_jℳult\left(f_j\left|\alpha \right.\right)\right]\hfill \\ {\prod }_j\left[\left(1-z_j\right)\delta \left(f_j\right)\right]\hfill \end{array}\\ P\left(z_j=1\right)=\lambda ,P\left(z_j=0\right)=1-\lambda \end{array}\right.$$

(41)

3 Bayesian inference with sparsity enforcing priors

The priors proposed can be used in a Bayesian approach to infer on f given the observed data g through the posterior law given in Equation (3). First let assume the error ϵ to be centered, Gaussian and white: $\epsilon ~\mathcal{N}\left(\epsilon \left|0,v_{\epsilon }I\right.\right)$. Then, using the forward model (1) we have

$$p\left(\mathit{g}\left|\mathit{f}\right.\right)=\mathcal{N}\left(\mathit{H}\mathit{f},v_{\epsilon }\mathit{I}\right)\propto \text{exp}\left\{\frac{1}{2v_{\epsilon }}{∥\mathit{g}-\mathit{H}\mathit{f}∥}^2\right\}$$

(42)

Now, we consider different priors.

3.1 Simple prior models

Given p(g|f) and any simple prior law p(f), the posterior law is written:

$$p\left(\mathit{f}\left|\mathit{g}\right.\right)\propto p\left(\mathit{g}\left|\mathit{f}\right.\right)p\left(\mathit{f}\right)\propto \text{exp}\left\{J\left(\mathit{f}\right)\right\}$$

(43)

with

$$J\left(\mathit{f}\right)=\frac{1}{2v_{\epsilon }}{∥\mathit{g}-\mathit{H}\mathit{f}∥}^2+\Omega \left(\mathit{f}\right)$$

(44)

where Ω(f) = -ln p(f) and so the Maximum A Posteriori (MAP) solution is expressed as the minimizer of this criterion which has two parts: the first part is due to the likelihood and the second part is due to the prior:

Thus, depending on the choice of the prior we obtain different expressions for Ω(f). For example for the GG model of (10) we get

$$\Omega (\mathit{f})=\gamma {\displaystyle \sum _j{\left|f_j\right|}^{\beta }).}$$

(45)

For the symmetric Weibull model (14) we get

$$\Omega \left(\mathit{f}\right)=-\gamma \sum _j{\left|f_j\right|}^{\beta }+\left(\beta -1\right)\text{log}\left|f_j\right|.$$

(46)

For the Student-t model (17) we get

$$\Omega \left(\mathit{f}\right)=\frac{\nu +1}{2}\sum _j\text{log}\left(1+f_j^2/v\right).$$

(47)

For the elastic net model we get

$$\Omega \left(\mathit{f}\right)=\sum _j\left[{\gamma }_1\left|f_j\right|+{\gamma }_2{f}_j^2\right]$$

(48)

and for the Dirichlet model we get

$$\Omega \left(\mathit{f}\right)=\sum _j{f}_j^{\alpha -1},f_j>0,\sum _j{f}_j=1.$$

(49)

For each of these cases, we may discuss on the unimodality and convexity of the criterion J(f) which depends mainly on its Hessian

$$\begin{array}{ll}\hfill \mathbf{\Delta }J\left(\mathit{f}\right)& =\left[\frac{{\partial }^{2}J\left(\mathit{f}\right)}{\partial f_j\partial f_i}\right]={\mathit{H}}^{\prime }\mathit{H}+\left[\frac{{\partial }^2\Omega \left(\mathit{f}\right)}{\partial f_i\partial f_j}\right]\\ ={\mathit{H}}^{\prime }\mathit{H}+\left[\frac{{\partial }^2\Omega \left(\mathit{f}\right)}{\partial f_j^2}\right]\end{array}$$

(50)

We may look at each case to examine the range of the parameters for which this Hessian matrix is positive definite.

The optimization is done iteratively:

Update operation can be additive, multiplicative or more complex. Updating steps α^(k)can be fixed or computed adaptively at each step (steepest descent for example). δ f^(k)can be, for example proportional to the gradient, in which case, we have

We may also consider to estimate some of these parameters by assigning them appropriate priors and then express the joint p(f, θ|g, θ₀) as given in Equation (4) and then try to estimate them jointly, for example joint MAP:

or alternate optimization:

We may also want to explore this joint posterior by generating samples from it. This can be done, for example, through the following Gibbs sampling scheme:

When a great number of samples are thus generated, we may compute their means, variances or any other statistics about them.

Finally, we may try to approximate this joint posterior by a simpler one, for example by a separable q(f, θ) = q₁(f) q₂(θ) using the variational approximation (VA). The main idea and the main basic steps to achieve this is more detailed in the following section. Here, however, we present the result on the following scheme:

To illustrate the differences, we may consider the simple case of a linear forward model and Gaussian priors:

$$\left\{\begin{array}{c}p\left(\mathit{g}\left|\mathit{f},v_{\epsilon }\right.\right)=\mathcal{N}\left(\mathit{H}\mathit{f},v_{\epsilon }\mathit{I}\right)\hfill \\ p\left(\mathit{f}\left|v_f\right.\right)=\mathcal{N}\left(0,v_f\mathit{I}\right)\hfill \end{array}\right.$$

(51)

In this case, if we know θ= (v _ϵ , v _f ), then

$$\begin{array}{l}p\left(\mathit{f}\left|\mathit{g},v_{\epsilon },v_f\right.\right)=\mathcal{N}\left(\hat{\mu },\hat{\Sigma }\right)\\ \mathsf{\text{with:}}\\ \hat{\mu }={\left[{\mathit{H}}^{\prime }\mathit{H}+\lambda I\right]}^{-1}{\mathit{H}}^{\prime }\mathit{g}\\ \hat{\Sigma }={\left[{\mathit{H}}^{\prime }\mathit{H}+\lambda \mathit{I}\right]}^{-1}\end{array}$$

(52)

with $\lambda =\frac{v_{\epsilon }}{v_f}$. So, we have $\hat{\mathit{f}}=\hat{\mu }$ which can be computed by optimizing J(f) = ||g- H f||² + λ||f||². A gradient based algorithm is shown below:

Now putting inverse Gamma priors on v _ϵ and v _f , or equivalently Gamma priors on τ _ϵ = 1/v _ϵ and τ _f = 1/v _f :

$$\left\{\begin{array}{c}p\left({\tau }_{\epsilon }\left|{\alpha }_{\tau 0},{\beta }_{\tau 0}\right.\right)=\mathcal{G}\left({\alpha }_{\epsilon 0},{\beta }_{\epsilon 0}\right)\hfill \\ p\left({\tau }_f\left|{\alpha }_{f0},{\beta }_{f0}\right.\right)=\mathcal{G}\left({\alpha }_{f0},{\beta }_{f0}\right)\hfill \end{array}\right.$$

(53)

we have

$$\left\{\begin{array}{c}p\left({\tau }_{\epsilon }\left|\mathit{f},\mathit{g},{\alpha }_{\tau 0},{\beta }_{\tau 0}\right.\right)=\mathcal{G}\left({\hat{\alpha }}_{\epsilon },{\hat{\beta }}_{\epsilon }\right)\hfill \\ p\left({\tau }_f\left|\mathit{f},{\alpha }_{f0},{\beta }_{f0}\right.\right)=\mathcal{G}\left({\hat{\alpha }}_f,{\hat{\beta }}_f\right)\hfill \end{array}\right.$$

(54)

with

$$\left\{\begin{array}{c}{\hat{\alpha }}_{\epsilon }={\alpha }_{\epsilon 0}+1/2\hfill \\ {\hat{\beta }}_{\epsilon }={\beta }_{\epsilon 0}+{∥\mathit{g}-\mathit{H}\mathit{f}∥}^2/2\hfill \\ {\hat{\tau }}_{\epsilon }={\hat{\alpha }}_{\tau }/{\hat{\beta }}_{\tau }=\frac{2{\alpha }_{\epsilon 0}+1}{2{\beta }_{\epsilon 0}+{∥\mathit{g}-\mathit{H}\mathit{f}∥}^2}\hfill \\ {\hat{\alpha }}_f={\alpha }_{f0}+1/2\hfill \\ {\hat{\beta }}_f={\beta }_{f0}+{∥\mathit{f}∥}^2/2\hfill \\ {\hat{\tau }}_f={\hat{\beta }}_f/{\hat{\alpha }}_f=\frac{2{\alpha }_{f0}+1}{2{\beta }_{f0}+{∥\mathit{f}∥}^2}\hfill \end{array}\right.$$

(55)

and $\hat{\lambda }=\frac{{\hat{\tau }}_f}{{\hat{\tau }}_{\epsilon }}$. Then, the alternate optimization of the JMAP estimate algorithm becomes

The Gibbs sampling algorithm becomes

The VBA algorithm becomes

$$\left\{\begin{array}{c}q\left(\mathit{f}\right)=\mathcal{N}\left(\hat{\mu },\hat{\Sigma }\right)\hfill \\ \tilde{\Sigma }={\left[{\mathit{H}}^{\prime }\mathit{H}+\lambda \mathit{I}\right]}^{-1}\hfill \\ \begin{array}{c}\tilde{\mu }=\tilde{\Sigma }{\mathit{H}}^{\prime }\mathit{g}\end{array}\hfill \\ q\left({\tau }_{\epsilon }\right)=\mathcal{G}\left({\tilde{\alpha }}_{\epsilon },{\tilde{\beta }}_{\epsilon }\right)\hfill \\ {\tilde{\alpha }}_{\epsilon }={\alpha }_{\epsilon 0}+1/2\hfill \\ \begin{array}{c}{\tilde{\beta }}_{\epsilon }={\beta }_{\epsilon 0}+{∥\mathit{g}-\mathit{H}<\mathit{f}>∥}^2/2\end{array}\hfill \\ q\left({\tau }_f\right)=\mathcal{G}\left({\tilde{\alpha }}_f,{\tilde{\beta }}_f\right)\hfill \\ {\tilde{\alpha }}_f={\alpha }_{f0}+1/2\hfill \\ \begin{array}{c}{\tilde{\beta }}_f={\beta }_{f0}+\frac{1}{2}∥⟨{\mathit{f}}^2⟩∥={\beta }_{f0}+\frac{1}{2}{\sum }_j⟨f_j^2⟩\end{array}\hfill \\ <\mathit{f}>=\tilde{\mu }\hfill \\ ⟨{\mathit{f}}^2⟩={\tilde{\mu }}^2+\mathsf{\text{diag}}\left[\tilde{\Sigma }\right]\hfill \end{array}\right.$$

(56)

and $\hat{\lambda }=\frac{{\hat{\tau }}_f}{{\hat{\tau }}_{\epsilon }}$:

We recently implemented these algorithms for different applications such as: synthetic aperture radar (SAR) Imaging [29], ...

3.2 Mixture models

For the mixture models, and in general for the models which can be expressed via the hidden variables, we want to estimate jointly the original unknowns f and the hidden variables: τ in Cauchy model, z in MoG2, BG or BGam models and z in MoG3 or MoGGammas. Let examine these a little in details.

3.3 Student-t and Cauchy models

In this case the joint prior law can be written as:

$$\begin{array}{l}p\left(\mathit{f},\tau \right)={\prod }_{j}p\left(f_j\left|{\tau }_j\right.\right)p\left({\tau }_j\right)={\prod }_j\mathcal{N}\left(f_j\left|0,1/{\tau }_j\right.\right)p\left({\tau }_j\right)\\ \propto \text{exp}\left\{-\frac{1}{2}{\sum }_j{\tau }_j{f}_j^2+a\text{ln}{\tau }_j-b{\tau }_j\right\}\mathsf{\text{with}}a\mathsf{\text{ = }}b=\nu /2\end{array}$$

(57)

such that

$$p\left(\mathit{f},\tau \left|\mathit{g}\right.\right)\propto p\left(\mathit{g}\left|\mathit{f}\right.\right)p\left(\mathit{f},\tau \right)\propto \text{exp}\left\{-J\left(\mathit{f},\tau \right)\right\}$$

(58)

where

$$j\left(\mathit{f},\tau \right)=\frac{1}{2v_{\epsilon }}{∥\mathit{g}-\mathit{H}\mathit{f}∥}^2+\sum _j\frac{1}{2}{\tau }_j{f}_j^2-a\text{ln}{\tau }_j+b{\tau }_j$$

(59)

Joint optimization of this criterion, alternatively with respect to f(with fixed τ)

$$\begin{array}{ll}\hfill \hat{\mathit{f}}& =\text{arg}{\text{min}}_{\mathit{f}}\left\{J\left(\mathit{f},\tau \right)\right\}\\ =\text{arg}{\text{min}}_{\mathit{f}}\left\{\frac{1}{2v_{\epsilon }}{∥\mathit{g}-\mathit{H}\mathit{f}∥}^2+{\sum }_j\frac{1}{2}{\tau }_j{f}_j^2\right\}\end{array}$$

(60)

and with respect to τ(with fixed f)

$$\begin{array}{ll}\hfill \hat{\tau }& =\text{arg}{\text{min}}_{\tau }\left\{J\left(\mathit{f},\tau \right)\right\}\\ =\text{arg}{\text{min}}_{\tau }\left\{{\sum }_j\frac{1}{2}{\tau }_j{f}_j^2-a\text{ln}{\tau }_j+b{\tau }_j\right\}\end{array}$$

(61)

results in the following iterative algorithm:

$$\left\{\begin{array}{c}\hat{\mathit{f}}={\left[{\mathit{H}}^{\prime }\mathit{H}+v_{\epsilon }\mathit{D}\left(\hat{\tau }\right)\right]}^{-1}{\mathit{H}}^{\prime }\mathit{g}\hfill \\ {\hat{\tau }}_j=\varphi \left({\hat{\mathit{f}}}_j\right)=\frac{a}{\widehat{f_j^2}+b}\hfill \\ \mathit{D}\left(\hat{\tau }\right)=\mathsf{\text{diag}}\left[1/\widehat{{\tau }_j},j=1,...,n\right]\hfill \end{array}\right.$$

(62)

Note that, τ _j is the inverse of a variance and we have $1/{\tau }_j=\frac{f_j^2+b}{a}$. We can interpret this as an iterative quadratic regularization inversion followed by the estimation of variances τ _j which are used in the next iteration to define the variance matrix D(τ).

Here too, we may study the conditions on which the joint criterion is uni-modal and its alternate optimization converges to its unique solution.

We may also consider a Gibbs sampling scheme

$$\left\{\begin{array}{c}\mathit{f}~p\left(\mathit{f}\left|\tau ,\mathit{g}\right.\right)\propto p\left(\mathit{g}\left|\mathit{f}\right.\right)p\left(\mathit{f}\left|\mathit{u}\right.\right)=\mathcal{N}\left(\mathit{f}\left|\hat{\mathit{f}},\hat{\Sigma }\right.\right)\hfill \\ \tau ~p\left(\tau \left|\mathit{f},\mathit{g}\right.\right)\propto p\left(\mathit{f}\left|\tau \right.\right)p\left(\tau \right)={\prod }_j\mathcal{G}\left({\tau }_j\left|\hat{\alpha },\hat{\beta }\right.\right)\hfill \end{array}\right.$$

(63)

where

$$\left\{\begin{array}{c}\hat{\Sigma }={\left[{\mathit{H}}^{\prime }\mathit{H}+v_{\epsilon }\mathit{D}\left(\tau \right)\right]}^{-1}\hfill \\ \hat{\mathit{f}}=\hat{\Sigma }{\mathit{H}}^{\prime }\mathit{g}\hfill \end{array}\right.$$

(64)

and

$$\left\{\begin{array}{c}\hat{\alpha }=\frac{1}{2}{\hat{f}}_j+a=\frac{1}{2}{\hat{f}}_j+\nu /2\hfill \\ \hat{\beta }=b=\nu /2\hfill \end{array}\right.$$

(65)

For the VBA, we have

$$\left\{\begin{array}{c}p\left(\mathit{g}\left|\mathit{f},v_{\epsilon }\right.\right)=\mathcal{N}\left(g\left|\mathit{H}\mathit{f},v_{\epsilon }\mathit{I}\right.\right),{\tau }_{\epsilon }=1/v_{\epsilon }\hfill \\ p\left({\tau }_{\epsilon }\right)=\mathcal{G}\left({\tau }_{\epsilon }\left|{\alpha }_{\epsilon 0}\right.,{\beta }_{\epsilon 0}\right)\hfill \\ p\left(\mathit{f}\left|\mathit{v}\right.\right)={\prod }_{j}p\left(f_j\left|v_j\right.\right)={\prod }_j\mathcal{N}\left(f_j\left|0,v_j\right.\right)=\mathcal{N}\left(\mathit{f}\left|0,\mathit{V}\right.\right)\hfill \\ \mathit{V}=\mathsf{\text{diag}}\left[v\right],{\tau }_j=1/v_j,\tau =\mathsf{\text{diag}}\left[\tau \right]={\mathit{V}}^{-1}\hfill \\ p\left(\tau \right)={\prod }_j\mathcal{G}\left({\tau }_j\left|{\alpha }_0,{\beta }_0\right.\right)\hfill \end{array}\right.$$

(66)

$$\left\{\begin{array}{c}\tilde{q}\left(\mathit{f}\right)=\mathcal{N}\left(\mathit{f}\left|\tilde{\mu },\tilde{\Sigma }\right.\right)\hfill \\ \tilde{\mu }=\tilde{\Sigma }{\mathit{H}}^{\prime }\mathit{g}\hfill \\ \tilde{\Sigma }={\left({\tilde{\tau }}_{\epsilon }{\mathit{H}}^{\prime }\mathit{H}+\tilde{\mathit{V}}\right)}^{-1},\mathsf{\text{with}}\tilde{\mathit{V}}=\mathsf{\text{diag[}}\tilde{v}\mathsf{\text{]}}\hfill \end{array}\right.$$

(67)

$$\left\{\begin{array}{c}\tilde{q}\left({\tau }_{\epsilon }\right)=\mathcal{G}\left({\tau }_{\epsilon }\left|{\tilde{\alpha }}_{\epsilon }\right.,{\tilde{\beta }}_{\epsilon }\right),\hfill \\ {\tilde{\alpha }}_{\epsilon }={\alpha }_{\epsilon 0}+\left(n+1\right)/2\hfill \\ {\tilde{\beta }}_{\epsilon }={\beta }_{\epsilon 0}+1/2\hfill \\ \tilde{\tau }={\tilde{\alpha }}_{\tau }/{\tilde{\beta }}_{\tau }\hfill \\ \tilde{q}\left({\tau }_j\right)=\mathcal{G}\left({\tau }_j\left|{\tilde{\alpha }}_j,{\tilde{\beta }}_j\right.\right)\hfill \\ {\tilde{\alpha }}_j={\alpha }_{00}+1/2\hfill \\ {\tilde{\beta }}_j={\beta }_{00}+<f_j^2>/2\hfill \\ {\tilde{z}}_j={\tilde{\beta }}_j/{\tilde{\alpha }}_j\hfill \end{array}\right.$$

(68)

3.4 Mixture of two Gaussians (MoG2) model

In this case, following the same arguments, we obtain:

$$p\left(\mathit{f},\mathit{z}\left|g\right.\right)\propto p\left(\mathit{g}\left|\mathit{f}\right.\right)p\left(\mathit{f},\mathit{z}\right)\propto \text{exp}\left\{-J\left(\mathit{f},\mathit{z}\right)\right\}$$

(69)

where

$$\begin{array}{ll}\hfill J\left(\mathit{f},\mathit{z}\right)=& \frac{1}{2v_{\epsilon }}{∥\mathit{g}-\mathit{H}\mathit{f}∥}^2\\ +{\sum }_j\frac{f_j^2}{2v_{z_j}}+z_j\text{ln}\lambda +\left(1-z_j\right)\text{ln}\left(1-\lambda \right)\end{array}$$

(70)

Again, in this case also, the optimization of this criterion, alternatively with respect to f and z results in the following iterative algorithm:

$$\left\{\begin{array}{c}\hat{\mathit{f}}={\left[{\mathit{H}}^{\prime }\mathit{H}+v_{\epsilon }\mathit{D}\left(\mathit{z}\right)\right]}^{-1}{\mathit{H}}^{\prime }\mathit{g}\hfill \\ {\hat{z}}_j=\varphi \left(f_j\right)=\left\{\begin{array}{c}1,\mathsf{\text{if}}\widehat{f_j^2}\ge \left(v_1-v_0\right)\text{ln}\frac{1-\lambda }{\lambda }\hfill \\ 0,\mathsf{\text{if}}\widehat{f_j^2}<\left(v_1-v_0\right)\text{ln}\frac{1-\lambda }{\lambda }\hfill \end{array}\right.\hfill \\ \mathit{D}\left(\hat{\mathit{z}}\right)=\mathsf{\text{diag}}\left[v_{\widehat{z_j}},j=1,..,n\right]\hfill \end{array}\right.$$

(71)

Here too, we may also consider a Gibbs sampling scheme

$$\left\{\begin{array}{c}\mathit{f}~p\left(\mathit{f}\left|\mathit{z},\mathit{g}\right.\right)\propto p\left(\mathit{g}\left|\mathit{f}\right.\right)p\left(\mathit{f}\left|\mathit{u}\right.\right)=\mathcal{N}\left(\mathit{f}\left|\hat{\mathit{f}},\hat{\Sigma }\right.\right)\hfill \\ \mathit{z}~p\left(\mathit{z}\left|\mathit{f},\mathit{g}\right.\right)\propto p\left(\mathit{f}\left|\mathit{z}\right.\right)p\left(\mathit{z}\right)={\prod }_{j}P\left(z_j=k\left|f_j\right.\right)\hfill \end{array}\right.$$

(72)

where

$$\left\{\begin{array}{c}\hat{\Sigma }={\left[{\mathit{H}}^{\prime }\mathit{H}+v_{\epsilon }\mathit{D}\left(\mathit{z}\right)\right]}^{-1}\hfill \\ \hat{\mathit{f}}=\hat{\Sigma }{\mathit{H}}^{\prime }\mathit{g}\hfill \end{array}\right.$$

(73)

and

$$\left\{\begin{array}{c}P\left(z_j=1\left|f_j\right.\right)=1,\mathsf{\text{if}}{f}_j^2\ge \left(v_1-v_0\right)\text{ln}\frac{1-\lambda }{\lambda }\hfill \\ P\left(z_j=0\left|f_j\right.\right)=1,\mathsf{\text{if}}{f}_j^2<\left(v_1-v_0\right)\text{ln}\frac{1-\lambda }{\lambda }\hfill \end{array}\right.$$

(74)

3.5 BG model

For the case of BG we have to be more careful, because the joint probability laws are degenerated. Two approaches are then possible:

i) Considering them as the particular case of the MoG models where the variance v₀ is fixed to a small value or reduced gradually during the iterations.

ii) Trying first to integrate out f from the expression of p(f, z|g) to obtain p(z|g) and optimize it with respect to z(detection step) and then use it for the estimation step.

To go further in detail of the second approach, we may remark that for the given z, the expression of p(f, z|g) as a function of f is Gaussian and so it can be easily integrated out and we obtain:

$$\begin{array}{ll}\hfill p\left(\mathit{z}\left|\mathit{g}\right.\right)& \propto p\left(\mathit{g}\left|\mathit{z}\right.\right)p\left(\mathit{z}\right)\\ \propto \mathcal{N}\left(\mathit{g}\left|0\right.,\mathit{H}\left(v\mathsf{\text{diag[}}{z}_j\mathsf{\text{,}}j\mathsf{\text{ = 1,}}...\mathsf{\text{,}}n\mathsf{\text{]}}\right){\mathit{H}}^{\prime }+v_{\epsilon }\mathit{I}\right)\times \\ {\lambda }^{{\sum }_j{z}_j}{\left(1-\lambda \right)}^{{\sum }_j\left(1-z_j\right)}\end{array}$$

(75)

Now writing the expression of $ℒ\left(\mathit{z}\right)=-\text{ln}p\left(\mathit{z}\left|\mathit{g}\right.\right)$ and keeping only all terms depending on z we obtain:

$$ℒ\left(\mathit{z}\right)=-{\mathit{g}}^{\prime }{\mathit{B}}^{-1}\left(\mathit{z}\right)\mathit{g}-\text{ln}\left|\mathit{B}\left(\mathit{z}\right)\right|-2n\text{ln}\frac{1-\lambda }{\lambda }$$

(76)

where B(z) = H(v diag [z _j , j = 1, ..., n])H' + v _ϵ I. We see the complexity of this expression which needs the inversion of the matrix B and its optimization which is a combinatorial optimization needing to evaluate this expression 2ⁿtimes.

However, we may also remark that when z obtained, the estimation of f is easy. We have:

$$\hat{\mathit{f}}={\mathit{H}}^{\prime }{\mathit{B}}^{-1}\mathit{g}.$$

(77)

which needs again the inversion of the matrix B.

The exact computations of $\hat{\mathit{z}}$ and $\hat{\mathit{f}}$ are often too costly, one may try to obtain approximate solutions. Many approximations have been proposed. A good overview of these methods can be found in [30, Chap. 5] and also in [31, 32].

3.6 BGamma and MoGGammas model

In these cases, it is no more possible to integrate out f analytically as it was the case with Gaussians. One strategy here is to use the MCMC methods to generate samples from the joint posterior. The second approach is to approximate the joint posterior by a simpler one, for example by a separable one on f and the hidden variables z in the BGamma or the MoGGammas cases. Very often then we can do the computations analytically. However, it may happens that, even after these separable approximations, still we need to use the MCMC methods on some of variables. Detailed explanation of these general methods is out of focus of this article. See [30, 33, 34]. Here, we just give the details for the case of the Gaussian mixtures (MoG2 or MoG3).

4 Variational Bayesian approximation for the case of mixture laws

To start and to be complete as to propose an unsupervised method, we include also the estimation of the parameters θ and write the joint posterior law of all the unknowns:

$$p\left(\mathit{f},\mathit{z},\theta \left|\mathit{g}\right.\right)\propto p\left(\mathit{g}\left|\mathit{f},\theta \right.\right)p\left(\mathit{f}\left|\mathit{z}\right.,\theta \right)p\left(\mathit{z}\left|\theta \right.\right)p\left(\theta \right)$$

(78)

which can also be written as

$$q\left(\mathit{f},\mathit{z},\theta \left|\mathit{g}\right.\right)=p\left(\mathit{f}\left|\mathit{z},\theta ;\mathit{g}\right.\right)p\left(\mathit{z}\left|\theta ;\mathit{g}\right.\right)p\left(\theta \left|\mathit{g}\right.\right)$$

(79)

where

$$p\left(\mathit{f}\left|\mathit{z}\right.,\theta ;\mathit{g}\right)=p\left(\mathit{g}\left|\mathit{f},\theta \right.\right)p\left(\mathit{f}\left|\mathit{z}\right.,\theta \right)/p\left(\mathit{g}\left|\mathit{z}\right.,\theta \right)$$

(80)

with

$$p\left(\mathit{g}\left|\mathit{z},\theta \right.\right)=\int p\left(\mathit{g}\left|\mathit{f},\theta \right.\right)p\left(\mathit{f}\left|\mathit{z},\theta \right.\right)\mathsf{\text{d}}\mathit{f}$$

and

$$p\left(\mathit{z}\left|\theta ;\mathit{g}\right.\right)=p\left(\mathit{g}\left|\mathit{z},\theta \right.\right)p\left(\mathit{z}\left|\theta \right.\right)/p\left(\mathit{g}\left|\theta \right.\right)$$

(81)

with

$$p\left(\mathit{g}\left|\theta \right.\right)=\int p\left(\mathit{g}\left|\mathit{z},\right.\theta \right)p\left(\mathit{z}\left|\theta \right.\right)\mathsf{\text{d}}\mathit{z}$$

or

$$p\left(\mathit{g}\left|\theta \right.\right)=\sum _{z}p\left(\mathit{g}\left|\mathit{z},\theta \right.\right)p\left(\mathit{z}\left|\theta \right.\right)$$

when z are discrete valued, and finally

$$p\left(\theta \left|\mathit{g}\right.\right)=p\left(\mathit{g}\left|\theta \right.\right)p\left(\theta \right)/p\left(\mathit{g}\right)$$

(82)

with

$$p\left(\mathit{g}\right)=\int p\left(\mathit{g}\left|\theta \right.\right)p\left(\theta \right)\mathsf{\text{d}}\theta$$

One can also write:

$$p\left(\mathit{z}\left|\theta ,\mathit{g}\right.\right)=\int p\left(\mathit{f},\mathit{z}\left|\theta ,\mathit{g}\right.\right)\mathsf{\text{d}}\mathit{f}$$

(83)

and

$$p\left(\theta \left|\mathit{g}\right.\right)=\int \int p\left(\mathit{f},\mathit{z},\theta \left|\mathit{g}\right.\right)\mathsf{\text{d}}f\mathsf{\text{d}}\mathit{z}=\int p\left(\mathit{z}\left|\theta ;\mathit{g}\right.\right)\mathsf{\text{d}}\mathit{z}$$

(84)

or

$$p\left(\theta \left|\mathit{g}\right.\right)=\sum _z\int p\left(\mathit{f},\mathit{z},\theta \left|\mathit{g}\right.\right)\mathsf{\text{d}}f=\sum _{z}p\left(\mathit{z}\left|\theta ;\mathit{g}\right.\right)$$

(85)

when z are discrete valued.

We see that the first term

$$p\left(\mathit{f}\left|\mathit{z},\theta ,\mathit{g}\right.\right)\propto p\left(\mathit{g}\left|\mathit{f},\theta \right.\right)p\left(\mathit{f}\left|\mathit{z},\theta \right.\right)$$

(86)

will be easy to handle because it is the product of two Gaussians and so it is a multivariate Gaussian. But the two others are not.

The main idea behind the VBA is to approximate the joint posterior p(f, z, θ|g) by a separable one, for example

$$q\left(\mathit{f},\mathit{z},\theta \left|\mathit{g}\right.\right)=q_1\left(\mathit{f}\left|\mathit{g}\right.\right)q_2\left(\mathit{z}\left|\mathit{g}\right.\right)q_3\left(\theta \left|\mathit{g}\right.\right)$$

(87)

illustrated here:

and where the expressions of q(f, z, θ|g) is obtained by minimizing the Kullback-Leibler divergence

$$\mathsf{\text{KL}}\left(q:p\right)=\int q\text{ln}\frac{q}{p}={⟨\text{ln}\frac{q}{p}⟩}_q$$

(88)

It is then easy to show that

$$\mathsf{\text{KL}}\left(q:p\right)=\text{ln}p\left(\mathit{g}\left|ℳ\right.\right)-ℱ\left(q\right)$$

(89)

where $p\left(\mathit{g}\left|ℳ\right.\right)$ is the likelihood of the model

$$p\left(\mathit{g}\left|ℳ\right.\right)=\int \int \int p\left(\mathit{f},\mathit{z},\theta ,\mathit{g}\left|ℳ\right.\right)\mathsf{\text{d}}\mathit{f}\mathsf{\text{d}}\mathit{z}\mathsf{\text{d}}\theta$$

(90)

with

$$p\left(\mathit{f},\mathit{z},\theta ,\mathit{g}\left|ℳ\right.\right)=p\left(\mathit{g}\left|\mathit{f},\theta \right.\right)p\left(\mathit{f}\left|\mathit{z},\theta \right.\right)p\left(\mathit{z}\left|\theta \right.\right)p\left(\theta \right)$$

(91)

and $ℱ\left(q\right)$ is the free energy associated to q defined as

$$ℱ\left(q\right)={⟨\text{ln}\frac{p\left(\mathit{f},\mathit{z},\theta ,\mathit{g}\left|ℳ\right.\right)}{q\left(\mathit{f},\mathit{z},\theta \right)}⟩}_q$$

(92)

So, for a given model $ℳ$, minimizing KL(q : p) is equivalent to maximizing $ℱ\left(q\right)$ and when optimized, $ℱ\left(q^{*}\right)$ gives a lower bound for $\text{ln}p\left(\mathit{g}\left|ℳ\right.\right)$.

Without any other constraint than the normalization of q, an alternate optimization of $ℱ\left(q\right)$ with respect to q₁, q₂, and q₃ results in

$$\begin{array}{c}{q}_1\left(\mathit{f}\right)\propto \text{exp}\left\{-{⟨\text{ln}p\left(\mathit{f},\mathit{z},\theta ,\mathit{g}\right)⟩}_{q\left(\mathit{z}\right)q\left(\theta \right)}\right\}\\ q_2\left(\mathit{z}\right)\propto \text{exp}\left\{-{⟨\text{ln}p\left(\mathit{f},\mathit{z},\theta ,\mathit{g}\right)⟩}_{q\left(\mathit{f}\right)q\left(\theta \right)}\right\}\\ q_3\left(\theta \right)\propto \text{exp}\left\{-{⟨\text{ln}p\left(\mathit{f},\mathit{z},\theta ,\mathit{g}\right)⟩}_{q\left(\mathit{f}\right)q\left(\mathit{z}\right)}\right\}\end{array}$$

Note that these relations represent an implicit solution for q₁(f), q₂(z), and q₃(θ) which need, at each iteration, the expression of the expectations in the right hand of exponentials. If p(g|f, z, θ₁) is a member of an exponential family and if all the priors p(f|z, θ₂), p(z|θ₃), p(θ₁), p(θ₂), and p(θ₃) are conjugate priors, then it is to see that these expressions leads to standard distributions for which the required expectations are easily evaluated. In that case, we may note

$$q\left(\mathit{f},\mathit{z},\theta \left|\mathit{g}\right.\right)=q_1\left(\mathit{f}\left|\tilde{\mathit{z}},\tilde{\theta };\mathit{g}\right.\right)q_2\left(\mathit{z}\left|\tilde{\mathit{f}},\tilde{\theta };\mathit{g}\right.\right)q_3\left(\theta \left|\tilde{\mathit{f}},\tilde{\mathit{z}};\mathit{g}\right.\right)$$

(93)

where the tilded quantities $\tilde{\mathit{z}},\tilde{\mathit{f}}$ and $\tilde{\theta }$ are, respectively functions of $\left(\tilde{\mathit{f}},\tilde{\theta }\right),\left(\tilde{\mathit{z}},\tilde{\theta }\right)$ and $\left(\tilde{\mathit{f}},\tilde{\mathit{z}}\right)$:

and where the alternate optimization results to alternate updating of the parameters $\left(\tilde{\mathit{z}},\tilde{\theta }\right)$ for q₁, the parameters $\left(\tilde{\mathit{f}},\tilde{\theta }\right)$ of q₂ and the parameters $\left(\tilde{\mathit{f}},\tilde{\mathit{z}}\right)$ of q₃.

Finally, we may note that, to monitor the convergence of the algorithm, we may evaluate the free energy

$$\begin{array}{ll}\hfill ℱ\left(q\right)& ={⟨\text{ln}\frac{p\left(\mathit{f},\mathit{z},\theta ,\mathit{g}\left|ℳ\right.\right)}{q\left(\mathit{f},\mathit{z},\theta \right)}⟩}_q\\ ={⟨\text{ln}p\left(\mathit{f},\mathit{z},\theta ,g\right)\left|ℳ\right.⟩}_q+{⟨-\text{ln}q\left(\mathit{f},\mathit{z},\theta \right)⟩}_q\\ ={⟨\text{ln}p\left(\mathit{g}\left|\mathit{f},\mathit{z},\theta \right.\right)⟩}_q+{⟨\text{ln}p\left(\mathit{f}\left|\mathit{z},\theta \right.\right)⟩}_q+{⟨\text{ln}p\left(\mathit{z}\left|\theta \right.\right)⟩}_q\\ +{⟨-\text{ln}q\left(\mathit{f}\right)⟩}_q+{⟨-\text{ln}q\left(\mathit{z}\right)⟩}_q+{⟨-\text{ln}q\left(\theta \right)⟩}_q\end{array}$$

(94)

where all the expectations are with respect to q.

Other decompositions are also possible:

$$\begin{array}{ll}\hfill q\left(\mathit{f},\mathit{z},\theta \left|\mathit{g}\right.\right)& =q_1\left(\mathit{f}\left|\tilde{\mathit{z}},\tilde{\theta };\mathit{g}\right.\right)\\ {\prod }_j{q}_{2j}\left(z_j\left|\tilde{\mathit{f}},{\tilde{\mathit{z}}}_{\left(-j\right)},\tilde{\theta };\mathit{g}\right.\right)\\ {\prod }_j{q}_{3l}\left({\theta }_l\left|\tilde{\mathit{f}},\tilde{\mathit{z}},{\tilde{\theta }}_{\left(-l\right)};\mathit{g}\right.\right)\end{array}$$

(95)

illustrated here:

or even by:

$$\begin{array}{ll}\hfill q\left(\mathit{f},\mathit{z},\theta \left|\mathit{g}\right.\right)& ={\prod }_j{q}_{1j}\left(f_j\left|{\tilde{\mathit{f}}}_{\left(-j\right)},\tilde{\mathit{z}},\tilde{\theta };\mathit{g}\right.\right)\\ {\prod }_j{q}_{2j}\left(z_j\left|\tilde{\mathit{f}},\tilde{\mathit{z}}\left(-j\right),\theta ;\mathit{g}\right.\right)\\ {\prod }_l{q}_{3l}\left({\theta }_l\left|\tilde{\mathit{f}},\tilde{\mathit{z}},{\tilde{\theta }}_{\left(-l\right)};\mathit{g}\right.\right)\end{array}$$

(96)

illustrated here:

Here, we consider the second case (Equation (95)) and give some more details on it. First to simplify the notations, we write it as:

$$q\left(\mathit{f},\mathit{z},\theta \right)=q_1\left(\mathit{f}\right)\prod _j{q}_{2j}\left(z_j\right)\prod _l{q}_{3l}\left({\theta }_l\right)$$

(97)

where it can be shown that:

$$\begin{array}{ll}\hfill q_1\left(\mathit{f}\right)& \propto \text{exp}\left\{-{⟨\text{ln}p\left(\mathit{f},\mathit{z},\theta ,\mathit{g}\right)⟩}_{q_2\left(\mathit{z}\right)q_3\left(\theta \right)}\right\}\\ \hfill q_{2j}\left(z_j\right)& \propto \text{exp}\left\{-{⟨\text{ln}p\left(\mathit{f},\mathit{z},\theta ,\mathit{g}\right)⟩}_{q1\left(\mathit{f}\right)q_3\left(\theta \right)q_2\left({\mathit{z}}_{\left(-j\right)}\right)}\right\}\\ \hfill q_{3l}\left({\theta }_l\right)& \propto \text{exp}\left\{-{⟨\text{ln}p\left(\mathit{f},\mathit{z},\theta ,\mathit{g}\right)⟩}_{q_1}{\left(\mathit{f}\right)}_{q_2}{\left(\mathit{z}\right)}_{q_3}\left({\theta }_{\left(-l\right)}\right)\right\}\end{array}$$

where p(f, z, θ, g) = p(g|f, θ)p(f|z, θ)p(z|θ)p(θ) and where q₂(z) = Π _j q_2j(z _j ), q₃(θ) = Π _l q_3l(θ _l ), q₂(z_(-j)) = Π_i≠jq_2j(z _j ), 〈.〉 _q means expected value with respect to q.

In that case, with appropriate models for the priors (exponential families) and hyper parameters (conjugate priors), we see that q(f) is a multivariate Gaussian $g\left(\mathit{f}\right)=\mathcal{N}\left(\mathit{f}\left|\tilde{\mu },\tilde{\Sigma }\right.\right)$, q(θ _l ) are either Gaussians (for the means) or Inverse Gammas (for the variances) and q(z _j ) are discrete distributions whose expressions can be written easily.

To illustrate this in more detail, we consider the case of the Student-t model.

4.1 Student-t model

In this case, we have the following relations for the forward model and the prior laws:

$$\left\{\begin{array}{c}p\left(\mathit{g}\left|\mathit{f},v_{\epsilon }\right.\right)=\mathcal{N}\left(\mathit{g}\left|\mathit{H}\mathit{f},v_{\epsilon }\mathit{I}\right.\right),\tau =1/v_{\epsilon }\hfill \\ p\left(\mathit{f}\left|\mathit{z}\right.\right)={\prod }_{j}p\left(f_j\left|z_j\right.\right)={\prod }_j\mathcal{N}\left(z_j\left|0,z_j\right.\right)=\mathcal{N}\left(\mathit{f}\left|0,\mathit{Z}\right.\right)\hfill \\ \mathit{Z}=\mathsf{\text{diag}}\left[\mathit{z}\right],a_j=1/z_j,\mathit{A}=\mathsf{\text{diag}}\left[\mathit{a}\right]={\mathit{Z}}^{-1}\hfill \\ p\left(\mathit{a}\right)={\prod }_j\mathcal{G}\left(a_j\left|{\alpha }_0,{\beta }_0\right.\right)\hfill \\ p\left(\tau \right)=\mathcal{G}\left(\tau \left|{\alpha }_{\tau 0},{\beta }_{\tau 0}\right.\right)\hfill \end{array}\right.$$

(98)

Then, we obtain the following expressions for the VBA:

$$\left\{\begin{array}{c}\tilde{q}\left(\mathit{f}\right)=\mathcal{N}\left(\mathit{f}\left|\tilde{\mu },\tilde{\Sigma }\right.\right)\hfill \\ \tilde{\mu }=<\tau >\tilde{\Sigma }{\mathit{H}}^{\prime }\mathit{g}\hfill \\ \tilde{\Sigma }={\left(<\tau >{\mathit{H}}^{\prime }\mathit{H}+\tilde{\mathit{Z}}\right)}^{-1},\mathsf{\text{with}}\tilde{\mathit{Z}}={\tilde{\mathit{A}}}^{-1}=\mathsf{\text{diag[}}\tilde{a}\mathsf{\text{]}}\hfill \end{array}\right.$$

(99)

$$\left\{\begin{array}{c}\tilde{q}\left(\tau \right)=\mathcal{G}\left(\tau \left|{\tilde{\alpha }}_{\tau },{\tilde{\beta }}_{\tau }\right.\right),\hfill \\ {\tilde{\alpha }}_{\tau }={\alpha }_{\tau 0}+\left(n+1\right)/2\hfill \\ \begin{array}{c}{\tilde{\beta }}_{\tau }={\beta }_{\tau 0}+1/2\left[{∥\mathit{g}∥}^2-2<\mathit{f}{>}^{\prime }{\mathit{H}}^{\prime }\mathit{g}+{\mathit{H}}^{\prime }<\mathit{f}{\mathit{f}}^{\prime }>\mathit{H}\right]\end{array}\hfill \\ \tilde{q}\left(a_j\right)=\mathcal{G}\left(a_j\left|{\tilde{\alpha }}_j,{\tilde{\beta }}_j\right.\right)\hfill \\ {\tilde{\alpha }}_j={\alpha }_{00}+1/2\hfill \\ {\tilde{\beta }}_j={\beta }_{00}+<f_j^2>/2\hfill \end{array}\right.$$

(100)

where the expressions of the expectations needed are:

$$\left\{\begin{array}{c}<\mathit{f}>=\tilde{\mu }\hfill \\ <{\mathit{ff}}^{\prime }>=\Sigma +{\mu \mu }^{\prime }\hfill \\ <f_j^2>={\left[\Sigma \right]}_{jj}+{\mu }_j^2\hfill \\ <\tau >=\tilde{\tau }={\tilde{\alpha }}_{\tau }/{\tilde{\beta }}_{\tau }\hfill \\ <a_j>={\tilde{a}}_j={\tilde{\alpha }}_j/{\tilde{\beta }}_j\hfill \end{array}\right.$$

(101)

We can also express the free energy expression:

$$\begin{array}{ll}\hfill ℱ\left(q\right)& =⟨\text{ln}\frac{p\left(\mathit{f},\mathit{a},\tau ,\mathit{g}\left|ℳ\right.\right)}{q\left(\mathit{f},\mathit{a},\tau \right)}⟩\\ =⟨\text{ln}p\left(\mathit{g}\left|\mathit{f},\mathit{a}\right.,\tau \right)⟩+⟨\text{ln}p\left(\mathit{f}\left|\mathit{a},\tau \right.\right)⟩+⟨\text{ln}p\left(\mathit{a}\left|\tau \right.\right)⟩\\ +⟨-\text{ln}q\left(\mathit{f}\right)⟩+⟨-\text{ln}q\left(\mathit{a}\right)⟩+⟨-\text{ln}q\left(\tau \right)⟩\end{array}$$

(102)

where

$$\begin{array}{l}〈\ln p(\mathit{g}|\mathit{f},\tau )〉=\frac{n}{2}\left(<\ln \tau >-\ln (2\pi )\right)\\ -\frac{1}{2}\left\{<\tau >{\mathit{g}}^{\prime }\mathit{g}-2<\mathit{f}{>}^{\prime }{\mathit{H}}^{\prime }g+{\mathit{H}}^{\prime }<\mathit{f}{\mathit{f}}^{\prime }>\mathit{H}\right\}\\ 〈-\ln p(\mathit{f}|\mathit{a})〉=-\frac{n+1}{2}\ln (2\pi )\\ -\frac{1}{2}\left\{{\displaystyle {\sum }_j<\ln {\alpha }_j><{\alpha }_j><f_j^2>}\right\}\\ 〈-\ln p(\mathit{a})〉=-(n+1){\alpha }_{{\epsilon }_0}\ln ({\beta }_{{\epsilon }_0})\\ +({\alpha }_{{\epsilon }_0}-1){\displaystyle {\sum }_j[<\ln {\alpha }_j>-\beta <{\alpha }_j)]-(n+1)\ln \Gamma (\alpha )}\\ 〈p(\tau ))〉=c\ln d+(c-1)<\ln \tau )>-d<\tau >-\ln \Gamma (c)\end{array}$$

and

$$\begin{array}{l}〈-\ln q(\mathit{f})〉=-\frac{n+1}{2}(1+\ln (2\pi ))-\frac{1}{2}\ln \left|{\Sigma }_j\right|\\ 〈-\ln q(\mathit{a})〉=-{\displaystyle {\sum }_j[{\tilde{\alpha }}_j\ln ({\tilde{\beta }}_j)+({\tilde{\alpha }}_j-1)<\ln {\tilde{\alpha }}_j>}\\ -{\tilde{\beta }}_j<{\alpha }_j>-\ln \Gamma ({\tilde{\alpha }}_j)]\\ 〈q(\tau ))〉=\tilde{c}\ln \tilde{d}+(\tilde{c}-1)<\ln \tau )>-\tilde{d}<\tau >-\ln \Gamma (\tilde{c})\end{array}$$

In these equations,

$$\left\{\begin{array}{c}<\text{ln}{a}_j>=\psi \left({ã}_j\right)-\text{ln}{\tilde{b}}_j\hfill \\ <\text{ln}\tau >=\psi \left(\tilde{c}\right)-\text{ln}\tilde{d}\hfill \\ \psi \left(a\right)=\frac{\partial \text{ln}\Gamma \left(a\right)}{\partial a}\hfill \end{array}\right.$$

(103)

The resulting algorithm can be summarized as follows

5 Conclusion

The sparsity is a required property in many signal and image processing applications. In this article, first we reviewed the main steps of the Bayesian approach for inverse problems in signal and image processing. Then we presented in a synthetic way the different prior models which can be used to enforce the sparsity. These models have been presented in two categories: simple and hierarchical with hidden variables. For each of these prior models, we discuss their properties and the way to use them in a Bayesian approach resulting to many different inversion algorithms.

We have applied these Bayesian algorithms in many different applications such as X-ray computed tomography [35, 36], optical diffraction tomography [37–39], positron emission tomography [40], Microwave imaging [41, 42], Sources separation [43–46], spectrometry [47, 48], Hyper spectral imaging [49], super resolution [50–52], image fusion [53], image segmentation [54], synthetic aperture radar (SAR) imaging [29]. To save the place and be very synthetic, we did not give here any simulation results or any results on different applications of these methods. These can be found in different articles just referenced.