Single image haze removal considering sensor blur and noise

Abstract

Images of outdoor scenes are usually degraded under bad weather conditions, which results in a hazy image. To date, most haze removal methods based on a single image have ignored the effects of sensor blur and noise. Therefore, in this paper, a three-stage algorithm for haze removal, considering sensor blur and noise, is proposed. In the first stage, we preprocess the degraded image and eliminate the blur/noise interference to estimate the hazy image. In the second stage, we estimate the transmission and atmospheric light by the dark channel prior method. In the third stage, a regularized method is proposed to recover the underlying image. Experimental results with both simulated and real data demonstrate that the proposed algorithm is effective, based on both the visual effect and quantitative assessment.

1. Introduction

As is well known, the light received by a sensor from scene points is often absorbed and scattered by a complex medium (e.g., dust, mist, or fumes). Therefore, images of outdoor scenes are often degraded, with fog, mist, or haze. The scene contents in such degraded images will not be easily visible. However, most of the outdoor computer vision systems, such as those used in surveillance and transportation, need to carry out meaningful scene analysis, extract useful information, and detect image features. To this end, it is imperative to remove the effects of bad weather from these images.

The amount of irradiance scattering and haze depends on the unknown scene depth, which makes haze removal complicated. Most haze removal methods require multiple images or additional prior information. The methods in [1, 2] remove haze using multiple images under different degrees of polarization. In [3–5], the haze-free image is obtained from multiple images with different weather conditions. By inputting depth prior information or 3D models, the methods in [6, 7] can also restore haze-free images.

Recently, single image haze removal algorithms have become very popular. According to the assumption that the airlight in the atmospheric scattering model is constant [8], Tan [9] developed a haze removal method using the Markov random field framework. Under the assumption that shading and transmission are locally statistically uncorrelated, Fattal [10] computed the scene albedo and medium transmission to achieve haze removal. This method is very effective for thin haze, but it cannot easily recover a degraded image covered in dense haze. Subsequently, He et al. [11, 12] proposed a novel dark channel prior method. Using this dark channel prior method, a haze-free image can be restored with few halo artifacts, even if objects are distant in a very hazy image. This dark channel prior method is a major breakthrough for haze removal from a single image and is the state of the art until now. There have also been many other single image visibility enhancement algorithms proposed lately, such as [13–15].

All the works described above ignore the problem of noise while recovering the haze-free image. However, noise is a universal phenomenon (shown in [16–22], etc.) and, in practice, is a significant issue in haze removal. Most haze removal methods considering noise, such as [19–21], require multiple images. Recently, Matlin and Milanfar [22] have proposed the removal of haze and noise from a single image by the use of the BM3D denoising algorithm and an iterative regression method.

Sensor blur is also an important degradation factor, especially for thin haze. However, to the best of our knowledge, a haze removal method considering sensor blur has not appeared in the literature to date.

In this paper, we present a haze image model considering both sensor blur and noise. Based on this degradation model, a three-stage haze removal algorithm is proposed. In the first stage, we preprocess the degraded image to obtain an estimated hazy image. In the second stage, this hazy image is processed to estimate the atmospheric light and the transmission by the dark channel prior method [11, 12]. In the third stage, a non-local regularized method combining the transmission is proposed to recover the underlying image with the known atmospheric light and transmission.

This paper is organized as follows. Section 2 describes the related models and approaches. In Section 3, a three-stage algorithm for haze removal considering sensor blur and noise is proposed. The experiments are presented in Section 4. Section 5 concludes the paper.

2. Background

2.1 The haze image model

The ‘haze image model’ (Equation 1) has been widely used to describe haze formation in computer vision [11, 12].

$$I\left(x\right)=u\left(x\right)t\left(x\right)+A_{\mathrm{ir}}\left(1-t\left(x\right)\right)$$

(1)

where I is the observed hazy image, u is the real scene radiance, A _ir is the global airlight, and t is the medium transmission coefficient (standing for the amount of light that is not scattered and arrives at the imaging instrument). The term A _ir(1 - t(x)) is therefore called the local atmospheric light, and the first term u(x)t(x) on the right-hand side is called the direct attenuation.

2.2 The dark channel prior method

In terms of a solution for the haze image model (Equation 1), the dark channel prior method [11, 12] has been the state of the art until now. A brief review is presented next.

Outdoor haze-free images often contain colorful objects or surfaces, dark objects or surfaces, and shadows. Consequently, in most of the patches, at least one of the color channels (red, green, blue (RGB)) will contain one or more low-intensity pixels and may even be close to zero. Based on the above idea, the concept of the dark channel [11, 12] was proposed, which is defined as follows:

$$u^{\mathrm{dark}}\left(x\right)=\underset{y\in \Omega \left(x\right)}{min}\left(\underset{c\in \left\{r,g,b\right\}}{min}{u}^c\left(y\right)\right)$$

(2)

where Ω(x) is a local neighbor centered at pixel x, and u ^c is one of the color channels (RGB). For a haze-free image, the dark channel tends to zero (Equation 3), which is called the dark channel prior.

$$u^{\mathrm{dark}}\to 0$$

(3)

However, the formation of hazy images is affected by atmospheric light (see Equation 1). According to Equation 1 and the dark channel prior, the dark channel of the hazy image yields:

$$I^{\mathrm{dark}}\left(x\right)\to \underset{y\in \Omega \left(x\right)}{min}\left(\underset{c\in \left\{r,g,b\right\}}{min}{A}_{\mathrm{ir}}^c\left(1-t\left(y\right)\right)\right)$$

(4)

The global airlight A _ir can be estimated by detecting the region with the deepest haze using the dark channel. The top 0.1% brightest pixels in the dark channel are picked out. Corresponding to these pixels, the highest intensities in the input hazy image are taken as the airlight. Primarily, the transmission t(x) is assumed as a constant in the local neighbor Ω(x). Therefore, the transmission t(x) can be coarsely estimated as follows:

$$\tilde{t}\left(x\right)=1-w\underset{c\in \left\{r,g,b\right\}}{min}\left(\underset{y\in \Omega \left(x\right)}{min}\left(\frac{I^c\left(y\right)}{A_{\mathrm{ir}}^c}\right)\right)$$

(5)

In order to retain a small quantity of haze in a distant object, an adjustable parameter w (0 < w ≤ 1) is selected in Equation 5. The transmission is then refined using a soft matting framework [23]. Finally, the transmission and the atmospheric light are estimated. A haze-free image u is restored, according to Equation 1, yielding:

$$u\left(x\right)=\frac{I\left(x\right)-A_{\mathrm{ir}}}{max\left(t\left(x\right),t_0\right)}+A_{\mathrm{ir}}$$

(6)

where a typical value of t ₀ is 0.1 [11, 12].

3. The proposed algorithm

3.1 The degradation mode

In the past works, most scholars have only considered the haze image model (Equation 1); few studies have considered noise in a hazy image. In this paper, taking both sensor blur and noise interference into account, the image degradation model is described as follows:

$$I\left(x\right)=h\left(x\right)*\left[u\left(x\right)t\left(x\right)+A_{\mathit{ir}}\left(1-t\left(x\right)\right)\right]+n\left(x\right),$$

(7)

where h is the point spread function, and n is the noise distribution function. In this paper, zero-mean Gaussian noise with standard deviation of σ is considered. The other parameters are the same as Equation 1. Our goal is to recover the underlying image u. Based on this degradation model, a three-stage haze removal algorithm (termed DC-NL) is proposed.

3.2 The three-stage haze removal algorithm

The proposed algorithm has three stages, as shown in Figure 1. In the first stage, a non-local means filter and non-local TV framework are used to preprocess the degraded image. In the second stage, the dark channel prior method is used to estimate the transmission and the atmospheric light. In the third stage, a regularized method combining the transmission is used to obtain the final result. The full details of the three-stage algorithm are described next.

Figure 1

The three-stage procedure.

3.2.1 The preprocessing methods

The non-local means filter [24, 25] is a simple yet efficient approach for denoising; thus, considering noise in a hazy image, it is selected as the preprocessing for denoising in the first stage. A brief summary is now presented.

The key idea of the non-local means filter is based on similar patches. It restores the current pixel intensity using the pixels in the similar neighbors, as shown:

$$I^{\prime }\left(x\right)=\frac{1}{C\left(x\right)}{\displaystyle {\int }_{\Omega }w\left(x,y\right)I\left(y\right)\mathit{dy}}$$

(8)

where I is the current processing image, and Ω is the search window centered at x, usually taken as 21 × 21. The weight function w(x, y) is computed between the similar intensities based on a regional comparison, and C(x) is the normalizing factor, yielding:

$$w\left(x,y\right)=exp\left(-\frac{\left(G\ast {\left|I\left(x+·\right)-I\left(y+·\right)\right|}^2\right)\left(0\right)}{a^2}\right)$$

(9)

$$C\left(x\right)={\displaystyle {\int }_{\Omega }w\left(x,y\right)\mathit{dy}}$$

(10)

where G is the Gaussian kernel function, and a is a filtering parameter controlling the decay of the exponential function. In general, a corresponds to the noise level [26]. In our experiments, it is set to $\sqrt{2}$ multiplied by the standard deviation. The standard deviation is estimated by using the medial absolute deviation of the wavelet coefficients [26, 27]. From [24], we recall that:

$$\left(G\ast {\left|I\left(x+·\right)-I\left(y+·\right)\right|}^2\right)\left(0\right)={\displaystyle {\int }_{\Omega }G\left(\xi \right){\left|I\left(x+\xi \right)-I\left(y+\xi \right)\right|}^2\mathit{d\xi }}$$

(11)

The weight is dominant if and only if the patch centered at y is similar to the current patch at x.

When sensor blur and noise are simultaneously considered in a hazy image, the non-local means filter is invalid for the preprocessing. Therefore, the effective non-local TV framework [26] is used to estimate the hazy image from Equation 7. A brief review of the non-local TV framework is now given.

The non-local deblurring cost function is as follows:

$$I^{\prime }=arg\underset{I^{\prime }}{min}\left\{J_{\mathrm{NL}/\mathrm{TV}}\left(I^{\prime }\right)+\frac{\lambda }{2}{{\displaystyle \int \left(I-h\ast I^{\prime }\right)}}^2\right\}$$

(12)

where I ^′ stands for the hazy image in Equation 7, i.e.,

$$\left\{\begin{array}{l}I=h\ast I^{\prime }+n\\ I^{\prime }=\mathit{ut}+A_{\mathrm{ir}}\left(1-t\right)\end{array}\right.$$

(13)

In Equation 12, J _NL/TV is the non-local TV regularization function, having the form:

$$\begin{array}{c}{J}_{\mathrm{NL}/\mathrm{TV}}\left(I^{\prime }\right)={\displaystyle \int {\left|{\nabla }_w{I}^{\prime }\right|}^2}\hfill \\ ={\displaystyle {\int }_{\Omega }\sqrt{{\displaystyle {\int }_{\Omega }{\left(I^{\prime }\left(x\right)-I^{\prime }\left(y\right)\right)}^{2}w\left(x,y\right)\mathit{dy}}}\mathit{dx}}\hfill \end{array}$$

(14)

where w(x, y) is the same non-local weight function as the one in the non-local means filter, and it is computed in the same way as in Equation 9. ∇  _w I ^′ is a non-local gradient, which is detailed in [28], and is defined as:

$$\left({\nabla }_w{I}^{\prime }\right)\left(x,y\right)=\left(I^{\prime }\left(y\right)-I^{\prime }\left(x\right)\right)\sqrt{w\left(x,y\right)}$$

(15)

The non-local divergence ${\mathrm{div}}_w\overrightarrow{f}$ is defined as:

$$\left({\mathrm{div}}_w\overrightarrow{f}\right)\left(x\right)={\displaystyle {\int }_{\Omega }\left(f\left(x,y\right)-f\left(y,x\right)\right)\sqrt{w\left(x,y\right)}\mathit{dy}}$$

(16)

Therefore, no matter what interference is considered in the hazy image, the hazy image I ^′ can be estimated by the preprocessing in the first stage.

3.2.2 The non-local regularized method for haze removal

With the estimated hazy image I ^′ from the first stage, the dark channel prior method is then used to estimate the transmission and the atmospheric light. In addition, an initial result u ₀ can be obtained according to Equation 1. Since the noise level and the degree of blurring are unknown, the preprocessing may not be complete, which sometimes results in u ₀ being unsatisfactory. Therefore, we take a variational idea to solve the hazy image model, considering sensor blur and noise (Equation 7).

When the transmission and the atmospheric light are estimated after the above two stages, the cost function is defined as:

$$E\left(u\right)=J\left(u\right)+\frac{\lambda }{2}{{\displaystyle \int \left\{h\ast \left[\mathit{ut}+A_{\mathrm{ir}}\left(1-t\right)\right]-I\right\}}}^2$$

(17)

where J(u) is the regularized function, λ is a regularized parameter, and the other parameters are the same as in Equation 7. When considering only the noise in recovering a hazy image, the point spread function h is the constant ‘1’ in the cost function (Equation 17).

In this paper, a non-local regularized function based on a non-local idea is proposed. Before this regularized function is introduced, we first define a non-local gradient operator combining the transmission.

Since the formation of a hazy image is related to the transmission, the weight function in the regularized function should be associated with the transmission. That is to say, the weights are dominant if the transmission in the compared window looks like that in the current window. Based on the above analysis, a non-local gradient operator referring to the transmission is defined as:

$$\left({\nabla }_{{\mathrm{ww}}_1}u\right)\left(x,y\right):=\left(u\left(y\right)-u\left(x\right)\right)\sqrt{w\left(x,y\right)w_1\left(x,y\right)}$$

(18)

where ‘ww₁’ is used to emphasize the difference with the non-local operator ‘∇  _w ’, and w(x, y) is computed using the image intensity in the patch by Equation 9, i.e.,

$$w\left(x,y\right)=exp\left(-\frac{\left(G\ast |u\left(x+·\right)-u\left(y+·\right){|}^2(0\right)}{a^2}\right)$$

(19)

w ₁(x, y) is also a non-local weight function, but it is computed by the transmission in the similar window, yielding:

$$w_1\left(x,y\right)=exp\left(-\frac{\left(G\ast |t\left(x+·\right)-t\left(y+·\right){|}^2(0\right)}{a_1^2}\right)$$

(20)

where G is a Gaussian function, and a ₁ is a filter parameter controlling the decay of the exponential function. The parameters in Equations 19 and 20 are the same as the ones in Equation 9.

The regularization functions can now be denoted as:

$$\begin{array}{c}J\left(u\right)={\displaystyle \int \left|{\nabla }_{{\mathrm{ww}}_1}u\right|}\hfill \\ ={\displaystyle {\int }_{\Omega }\sqrt{{\displaystyle {\int }_{\Omega }{\left(u\left(x\right)-u\left(y\right)\right)}^{2}w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}\mathit{dx}}}\hfill \end{array}$$

(21)

Through the weight functions of w(x, y) and w ₁(x, y), the regularized function J can simultaneously preserve the similarities of the intensity and the transmission.

We use gradient descent to solve the minimization problem (Equation 17) and to get the iterative equation, shown as Equation 22:

$$u_{m+1}=u_m+\mathit{\Delta \beta }\left\{-L{u}_m+\lambda t\tilde{h}\ast \left\{I-h\ast \left[u_{m}t+A_{\mathrm{ir}}\left(1-t\right)\right]\right\}\right\},m\ge 0$$

(22)

where Δβ is the step, $\tilde{h}$ is the conjugate operator of h, and L is the corresponding gradient operator with respect to the regularized function J. Next, we select the first variation approach, which is used to discover the minimum of the energy function [29]. The operator L is derived in detail in the ‘Appendix’ and has the following form:

$$\mathit{Lu}={\displaystyle {\int }_{\Omega }\left(u\left(x\right)-u\left(y\right)\right)w\left(x,y\right)w_1\left(x,y\right)\left[\frac{1}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(x\right)\right|}+\frac{1}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(y\right)\right|}\right]}\mathit{dy}$$

(23)

4. Experimental results and discussion

In this section, extensive simulated experiments and real-life experiments with the proposed algorithm are presented. The results verify the effectiveness of the proposed three-stage algorithm. A discussion and an analysis of the results are also presented.

4.1 Experiment setting

In the simulated experiments, a hazy image is selected, as shown in Figure 2a. In the real-life experiments, three real data sets are selected, as shown in Figures 3a, 4a, and 5a. Since the clear images (the original haze-free images) in both the simulated and real-life experiments are hard to obtain directly, the common quantity evaluation index based on the reference image cannot be used to assess the recovery results. Therefore, a non-reference image quality assessment index, the Q-metric [30, 31], is selected as the objective evaluation. For the above index, the higher the value is, the better the recovery image is.

Figure 2

Simulated noise-only experiment. (a) The original hazy image. (b)The hazy image considering noise only. (c) DC. (d) IR. (e) DC-NL.

Figure 3

The first real data set. (a) The original true degraded data. (b) DC. (c) IR with denoising in the first stage. (d) DC-NL with denoising in the first stage. (e) IR with deblurring and denoising in the first stage. (f) DC-NL with deblurring and denoising in the first stage.

Figure 4

The second real data set. (a) The original true degraded data. (b) DC. (c) IR with denoising in the first stage. (d) DC-NL with denoising in the first stage. (e) IR with deblurring and denoising in the first stage. (f) DC-NL with deblurring and denoising in the first stage.

Figure 5

The third real data set. (a) The original true degraded data. (b) DC. (c) IR with denoising in the first stage. (d) DC-NL with denoising in the first stage. (e) IR with deblurring and denoising in the first stage. (f) DC-NL with deblurring and denoising in the first stage.

In this paper, ‘DC’ stands for the result directly from the dark channel [11, 12], and ‘IR’ stands for the initial result from the second stage, i.e., the result of dark channel after preprocessing. The final recovery result of the proposed algorithm is abbreviated to ‘DC-NL’. In all the experiments, the regularization parameter and the other parameters in the process are heuristically adjusted until the best result is obtained.

4.2 Experimental results

4.2.1 Simulated experiments

When the noise and blur parameters are known, the noise and blur interference can be largely eliminated in the haze removal, and the efficiency of the proposed haze removal algorithm can therefore be better demonstrated. For this reason, we first perform simulated experiments. For the simulated experiments, the original hazy image with a size of 140 × 250 is shown in Figure 2a. Three cases are considered: ‘noise only’, ‘blur only’, and ‘noise and blur’. Zero-mean Gaussian noise with a standard deviation of σ = 5 and a 5 × 5 Gaussian blur with standard deviation of σ = 0.5 are the experiment parameters. The filter parameter a ₁ is selected as 0.005, 0.012, and 0.016 in the three cases, respectively. The visual results are shown in Figures 2, 6, 7, 8, 9, 10. The quantitative assessment results using the Q-metric index are shown in Table 1 and Figure 11, which illustrate that the proposed three-stage algorithm produces the best results. A detailed analysis on the three experiments is presented in the following section.

Figure 6

Simulated blur-only experiment. (a) The hazy image considering sensor blur (blur only). (b) DC. (c) IR. (d) DC-NL.

Figure 7

The first detailed regions comparison for Figure 6. Detailed regions (a, b, c, d) from Figure 6a,b,c,d, respectively.

Figure 8

The Second detailed regions comparison for Figure 6. Detailed regions (a, b, c, d) from Figure 6a,b,c,d, respectively.

Figure 9

Simulated blur-and-noise experiment. (a) The hazy image considering sensor blur and noise. (b) DC. (c) IR. (d) DC-NL.

Figure 10

Detailed regions comparison for Figure 9. Detailed regions (a, b, c, d) from Figure 9a,b,c,d, respectively.

Table 1 The quantitative evaluation results for the three simulated experiments

Figure 11

The quantitative evaluation results for the three simulated experiments.

In the case of noise only, the degraded image is shown in Figure 2b. Since the dark channel prior method removes haze regardless of noise, the effect of the noise still exists, even in homogeneous regions, which results in the rough recovery result (noted as DC) in Figure 2c. Since preprocessing is used in the proposed algorithm, the noise interference is weakened after the first stage. Therefore, IR in Figure 2d is better than DC. Although the visual effect of the final result (DC-NL) obtained by the proposed algorithm in Figure 2e is nearly the same as IR, the quantitative evaluation result of DC-NL using the Q-metric index is better than IR, as shown in Table 1. This is because the regularized method and the iterative means are used in the third stage, which preserves the edges while suppressing the noise and removing the haze.

In the case of blur only, the degraded image is shown in Figure 6a. From Figure 6, based on visual observation, the recovery result (DC-NL) in Figure 6d is better. The intensity distributions of DC and IR in the sky region are not particularly homogeneous, as shown in Figure 6b,c, respectively. Serious halo artifacts can also be seen around the edges in their recovery results. The reason for this may be as follows: Because of the blur interference, the dark channel prior method is used to remove haze directly, which results in the halo artifacts in DC. Although the preprocessing is used in the first stage, the deblurring effect is affected by the existence of the haze. Therefore, halo artifacts also exist in IR. However, the proposed algorithm not only reduces the halo artifacts but also preserves the edges well. In order to display the comparative visual results, detailed regions from Figure 6 are shown in Figures 7 and 8. From Figure 7d, it can be seen that the edge of the beam is better preserved. The visual effect in Figure 6 is consistent with the quantitative evaluation results in Table 1.

In the case of noise and blur, the degraded image is shown in Figure 9a. From Figure 9b, for the blur and noise interference, DC produces ‘pseudo-edge’ effects in the homogeneous regions. Even if the preprocessing in the first stage is used, the deblurring and denoising effects are affected by the existence of the haze, which results in a few pseudo-edge effects in the homogeneous regions. In the proposed algorithm, since the regularized method and iterative means are used in the third stage, this preserves the edges and reduces the pseudo-edge effects (see the detailed regions in Figure 10). In addition, we can also see that DC-NL produces the highest Q-metric, as shown in Table 1.

We now make a lateral comparison with other methods. The cases of noise only and blur only are considered. The degraded images are shown in Figures 2b and 6a, respectively. The results using the methods of Fattal [10] and Milanfar [22] are shown in Figure 12. The quantitative evaluation results using the Q-metric index are also attached. In the case of noise only, the recovery result by Fattal [10] is relatively poor for the noise interference. Since Milanfar [22] used BM3D to preprocess the degraded image, the result in Figure 12b is similar to the one processed by the proposed method in Figure 2e. Only the proposed method removes haze considering sensor blur; thus, the result in Figure 6d is better than the ones obtained by the methods of Fattal [10] and Milanfar [22] in Figure 12c,d, respectively.

Figure 12

Comparison with other methods. (a) Fattal's result [10] with noise only (Q-metric = 64.44). (b) Milanfar's result [22] with noise only (Q-metric = 91.69). (c) Fattal's result [10] with blur only (Q-metric = 71.67). (d) Milanfar's result [22] with blur only (Q-metric = 73.28).

4.2.2 Real-life experiments

To verify the performance of the proposed algorithm with real data, three real-life experiments were performed. Since the sensor blur and the noise parameters are unknown, two preprocessing methods are used in the first stage.

The first real data set, with a size of 200 × 380, is shown in Figure 3a, and the recovery results are presented in Figure 3b,c,d,e,f. From Figure 3b, it can be seen that the intensity distribution on the outside wall is seriously uneven, and noise still exists in the DC result. In order to better display the comparative visual results, detailed regions from Figure 3b,c,d,e,f are shown in Figure 13a,b,c,d,e, respectively. Here, it can be seen that the results obtained by the proposed algorithm are better than DC. Although the visual diversity between panels d and f is low in Figure 3, DC-NL with deblurring and denoising in the first stage produces the highest Q-metric, as shown in Table 2. In our degradation model, since noise and blur are both considered in the haze image model, when the non-local TV is used to denoise/deblur, the textured regions may become blurred, which has also been discussed in [26]. Thus, the proposed method may result in an image in which the textured regions become blurred. In order to overcome this problem, an improved adaptive method will be explored in our future work.

Figure 13

Detailed regions comparison for Figure 3. Detailed regions (a, b, c, d, e) from Figure 3b,c,d,e,f, respectively.

Table 2 The quantitative evaluation results using the Q-metric index in Figure 3

For the second real-life experiment, a real data set with a size of 250 × 250 is shown in Figure 4a. The recovery results, corresponding detailed regions, and the quantitative evaluation results are shown in Figures 4 and 14 and Table 3, respectively. Since real-life images often have a degree of noise and sensor blur, this makes the dark channel prior method invalid, which results in the pseudo-edges in DC. When preprocessing is used in the first stage, both IR and DC-NL can reduce the amount of pseudo-edges, as shown in Figure 4c,d,e,f. In addition, DC-NL preserves the edges well, which can be seen in Figure 14d,f. It can be seen that the visual effect of DC-NL is the most pleasing, and its Q-metric is highest when the non-local TV framework is used for preprocessing in the first stage.

Figure 14

Detailed regions comparison for Figure 4. Detailed regions (a, b, c, d, e, f) from Figure 4a,b,c,d,e,f, respectively.

Table 3 The quantitative evaluation results using the Q-metric index in Figure 4

The third real data set, with a size of 200 × 200, features dense haze, as shown in Figure 5a. From Figure 5b, it can be seen that halo artifacts and pseudo-edge effects exist in the DC result. Compared to IR and DC-NL in Figure 5e,f, with deblurring and denoising simultaneously in the first stage, IR and DC-NL in Figure 5c,d, with denoising in the first stage, are relatively improved, which is consistent with the quantitative evaluation results in Table 4. Although the recovery results in Figure 5e,f are improved, some halo artifacts still exist, which becomes apparent in the edges of the people. However, the halo artifacts in the edges of the people in Figure 5c,d are clearly reduced. In order to illustrate this point, detailed regions from Figure 5c,d,e,f are shown in Figure 15. Possible reasons for this occurrence may be as follows: Firstly, since the imaging instrument itself can be considered accurate, sensor blur can be ignored. Alternatively, the sensor blur effect exists, but the dense fog seriously suppresses this effect, which affects the deblurring in the first stage. Therefore, the results with the deblurring preprocessing may be a bit rough. To verify the efficiency of the proposed algorithm, a histogram of the quantitative evaluation results for the above real-life experiments is shown in Figure 16.

Table 4 The quantitative evaluation results using the Q-metric index in Figure 5

Figure 15

Detailed regions comparison for Figure 5. Detailed regions (a, b, c, d) from Figure 5c,d,e,f, respectively.

Figure 16

The quantitative evaluation results for the real-life experiments.

For the above real-life experiments results, we want to give a further detailed explanation. Since real-life images often have a degree of noise and blur coming from the imaging sensor, the dark channel prior method is effectively invalid. Therefore, the recovery results (DC) by the dark channel prior method are relatively poor. Through the preprocessing in the first stage, some of the initial results are satisfactory. However, since the noise level and the degree of blurring are unknown in the real-life data, the preprocessing in the first stage may not be complete, which makes some of the initial results rough. This is because IR is directly obtained by solving the haze image model equation in the second stage. However, the proposed algorithm presents a variational model to recover the underlying image in the third stage, which can overcome the shortcomings of IR. Therefore, the proposed algorithm can efficiently recover a haze-free image while deblurring and suppressing the noise.

Finally, three popular haze images are used to further evaluate the proposed method. The haze images are from [10], with sizes of 441 × 450, 512 × 384, and 512 × 384, respectively. In order to save space, only the restored results (DC-NL) are shown in Figure 17. Again, it can be seen that the proposed method effectively recovers haze-free images.

Figure 17

Haze removal results for some popular haze images. Top, the input hazy images. Bottom, the restored haze-free images.

5. Conclusions

In this paper, a haze image model, considering sensor blur and noise, is presented for the first time. Based on this model, a three-stage haze removal algorithm is proposed.

In the first stage, a non-local means filter and non-local TV framework are used to preprocess the degraded image to obtain an estimated hazy image. In the second stage, based on this estimated image, the atmospheric light and the transmission are estimated, and the IR is obtained by the dark channel prior method. In the third stage, a non-local regularized method combining the transmission is proposed to recover the underlying image. Extensive experiments were performed on both simulated data and real-life data, and the experimental results demonstrate that the proposed algorithm is effective, based on both the visual effect and quantitative assessment.

Appendix

Here, the gradient operator L corresponding to J(u) in Equation 21 is derived in detail. First, set a real valued function:

$$\Phi \left(\tau \right)=J\left(u+\mathit{\tau v}\right)\left(\tau \in \mathbb{R}\right)$$

(24)

where v is any smooth function. Since u is the minimum value of J(u), τ = 0 is the minimum of Φ(τ), i.e.,

$${\Phi }^{\prime }\left(0\right)=0$$

(25)

From the expression of J(u), having:

$$\Phi \left(\tau \right)={\displaystyle {\int }_{\Omega }\sqrt{{\displaystyle {\int }_{\Omega }{\left(u\left(x\right)+\mathit{\tau v}\left(x\right)-u\left(y\right)-\mathit{\tau v}\left(y\right)\right)}^{2}w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}\mathit{dx}}}$$

(26)

Compute this derivative with respect to τ, thus:

$${\Phi }^{\prime }\left(\tau \right)={\displaystyle {\int }_{\Omega }\frac{{\displaystyle {\int }_{\Omega }\left(u\left(x\right)+\mathit{\tau v}\left(x\right)-u\left(y\right)-\mathit{\tau v}\left(y\right)\right)\left(v\left(x\right)-v\left(y\right)\right)w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}{\sqrt{{\displaystyle {\int }_{\Omega }{\left(u\left(x\right)+\mathit{\tau v}\left(x\right)-u\left(y\right)-\mathit{\tau v}\left(y\right)\right)}^{2}w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}}\mathit{dx}}$$

(27)

Let τ = 0, and deduce by Equations 25 and 27 that:

$$0={\Phi }^{\prime }\left(0\right)={\displaystyle {\int }_{\Omega }\frac{{\displaystyle {\int }_{\Omega }\left(u\left(x\right)-u\left(y\right)\right)\left(v\left(x\right)-v\left(y\right)\right)w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}{\sqrt{{\displaystyle {\int }_{\Omega }{\left(u\left(x\right)-u\left(y\right)\right)}^{2}w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}}\mathit{dx}}$$

(28)

Write the above Equation 28 into two parts:

$$\begin{array}{c}I+\mathit{II}={\displaystyle {\int }_{\Omega }\frac{{\displaystyle {\int }_{\Omega }\left(u\left(x\right)-u\left(y\right)\right)v\left(x\right)w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}{\sqrt{{\displaystyle {\int }_{\Omega }{\left(u\left(x\right)-u\left(y\right)\right)}^{2}w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}}\mathit{dx}}\hfill \\ +{\displaystyle {\int }_{\Omega }\frac{{\displaystyle {\int }_{\Omega }\left(u\left(y\right)-u\left(x\right)\right)v\left(y\right)w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}{\sqrt{{\displaystyle {\int }_{\Omega }{\left(u\left(x\right)-u\left(y\right)\right)}^{2}w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}}\mathit{dx}}\hfill \end{array}$$

(29)

Calculate these two parts respectively, yielding:

$$\begin{array}{c}I={\displaystyle {\int }_{\Omega }\frac{{\displaystyle {\int }_{\Omega }\left(u\left(x\right)-u\left(y\right)\right)w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}{\sqrt{{\displaystyle {\int }_{\Omega }{\left(u\left(x\right)-u\left(y\right)\right)}^{2}w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}}v\left(x\right)\mathit{dx}}\hfill \\ ={\displaystyle {\int }_{\Omega }\frac{{\displaystyle {\int }_{\Omega }\left(u\left(x\right)-u\left(y\right)\right)w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(x\right)\right|}v\left(x\right)\mathit{dx}}\hfill \end{array}$$

(30)

$$\begin{array}{c}\mathit{II}={\displaystyle {\int }_{\Omega }\frac{{\displaystyle {\int }_{\Omega }\left(u\left(y\right)-u\left(x\right)\right)v\left(y\right)w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}{\sqrt{{\displaystyle {\int }_{\Omega }{\left(u\left(x\right)-u\left(y\right)\right)}^{2}w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}}\mathit{dx}}\hfill \\ ={\displaystyle {\int }_{\Omega }\frac{{\displaystyle {\int }_{\Omega }\left(u\left(x\right)-u\left(y\right)\right)v\left(x\right)w\left(x,y\right)w_1\left(x,y\right)\mathit{dx}}}{\sqrt{{\displaystyle {\int }_{\Omega }{\left(u\left(y\right)-u\left(x\right)\right)}^{2}w\left(x,y\right)w_1\left(x,y\right)\mathit{dx}}}}\mathit{dy}}\hfill \\ ={\displaystyle {\int }_{\Omega }\frac{{\displaystyle {\int }_{\Omega }\left(u\left(x\right)-u\left(y\right)\right)v\left(x\right)w\left(x,y\right)w_1\left(x,y\right)\mathit{dx}}}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(y\right)\right|}\mathit{dy}}\hfill \\ ={\displaystyle {\int }_{\Omega }\left\{{\displaystyle {\int }_{\Omega }\frac{\left(u\left(x\right)-u\left(y\right))w\left(x,y\right)w_1(x,y\right)}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(y\right)\right|}\mathit{dy}}\right\}v\left(x\right)\mathit{dx}.}\hfill \end{array}$$

(31)

We deduce Equation 31 using the variable substitution and the symmetry of w(x, y) and w ₁(x, y) in the first step and then changing the integration order in the last step. Thus,

$$\begin{array}{c}I+\mathit{II}={\displaystyle {\int }_{\Omega }\left\{\frac{{\displaystyle {\int }_{\Omega }\left(u\left(x\right)-u\left(y\right)\right)w\left(x,y\right)w_1\left(x,y\right)\mathit{dy}}}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(x\right)\right|}\right\}}\hfill \\ +{\displaystyle {\int }_{\Omega }\frac{\left(u\left(x\right)-u\left(y\right))w\left(x,y\right)w_1(x,y\right)}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(y\right)\right|}\mathit{dy}}v\left(x\right)\mathit{dx}\hfill \\ ={\displaystyle {\int }_{\Omega }\left\{{\displaystyle {\int }_{\Omega }\frac{\left(u\left(x\right)-u\left(y\right))w\left(x,y\right)w_1(x,y\right)}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(x\right)\right|}}\mathit{dy}\right\}}\hfill \\ +{\displaystyle {\int }_{\Omega }\frac{\left(u\left(x\right)-u\left(y\right))w\left(x,y\right)w_1(x,y\right)}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(y\right)\right|}\mathit{dy}}v\left(x\right)\mathit{dx}\hfill \\ ={\displaystyle {\int }_{\Omega }\left\{{\displaystyle {\int }_{\Omega }\left(u\left(x\right)-u\left(y\right)\right)w\left(x,y\right)}{w}_1\left(x,y\right)\right\}}\hfill \\ \left[\frac{1}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(x\right)\right|}+\frac{1}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(y\right)\right|}\right]\mathit{dy}v\left(x\right)\mathit{dx}.\hfill \end{array}$$

(32)

According to the arbitrariness of the test function v, from Equations 28 and 32, we have:

$$\begin{array}{c}\mathit{Lu}={\displaystyle {\int }_{\Omega }\left(u\left(x\right)-u\left(y\right)\right)w\left(x,y\right)}{w}_1\left(x,y\right)\hfill \\ x\left[\frac{1}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(x\right)\right|}+\frac{1}{\left|{\nabla }_{{\mathrm{ww}}_1}u\left(y\right)\right|}\right]\mathit{dy}\hfill \end{array}$$

(33)