Realistic camera noise modeling with application to improved HDR synthesis

Abstract

Due to the ongoing miniaturization of digital camera sensors and the steady increase of the “number of megapixels”, individual sensor elements of the camera become more sensitive to noise, even deteriorating the final image quality. To go around this problem, sophisticated processing algorithms in the devices, can help to maximally exploit the knowledge on the sensor characteristics (e.g., in terms of noise), and offer a better image reconstruction. Although a lot of research focuses on rather simplistic noise models, such as stationary additive white Gaussian noise, only limited attention has gone to more realistic digital camera noise models. In this article, we first present a digital camera noise model that takes several processing steps in the camera into account, such as sensor signal amplification, clipping, post-processing,.. We then apply this noise model to the reconstruction problem of high dynamic range (HDR) images from a small set of low dynamic range (LDR) exposures of a static scene. In literature, HDR reconstruction is mostly performed by computing a weighted average, in which the weights are directly related to the observer pixel intensities of the LDR image. In this work, we derive a Bayesian probabilistic formulation of a weighting function that is near-optimal in the MSE sense (or SNR sense) of the reconstructed HDR image, by assuming exponentially distributed irradiance values. We define the weighting function as the probability that the observed pixel intensity is approximately unbiased. The weighting function can be directly computed based on the noise model parameters, which gives rise to different symmetric and asymmetric shapes when electronic noise or photon noise is dominant. We also explain how to deal with the case that some of the noise model parameters are unknown and explain how the camera response function can be estimated using the presented noise model. Finally, experimental results are provided to support our findings.

Introduction

The modeling of realistic camera noise is a subject that has not extensively been investigated, compared to the overwhelming amount of attention that the problem of stationary additive white Gaussian noise (AWGN) removal from images receives. A white stationary Gaussian noise assumption leads to simple and elegant denoising methods (such as wavelet shrinkage methods [1–5], total variation [6], anisotropic diffusion [7, 8], NLMeans [9–11]). However, when applied to realistic problems (e.g., the suppression of noise from CCD/CMOS measures taken with mobile phones or other consumer cameras), these techniques often yield poor results [11–13]. The main problem is the noise model mismatch, which causes the techniques to either over- or underestimate the noise level. In realistic circumstances, noise is not an additive Gaussian process, not white nor stationary. For example, due to quantum-mechanical aspects inherent to the measurement of light, the sensor signals, in the form of analog voltages, are subject to statistical fluctuations with variances proportional to the “ideal”^a signal (i.e., the one we would like to measure). This phenomenon is known as photon noise. Subsequently, the measured analog voltages are amplified and converted to digital signals, leading to the introduction of electronic noise and quantization noise. The overall noise is a signal-dependent mix of noise from different sources, in combination with different linear and nonlinear pre/post-processing steps performed inside the camera. Consequently, the noise can not be well described using a stationary AWGN model.

In literature, it is often advocated to use so-called variance stabilizing transforms[14, 15], these are transforms that make the noise approximately “independent” and additive with a constant variance. The simple and elegant denoising approaches can then efficiently be applied to the variance stabilized signal and finally the inverse stabilization transform is applied. A first problem is the fact that a signal model that works fine in a linear domain may function poorly after variance stabilization [13]. A second problem is that an accurate noise model is needed in order to build such a variance stabilization transform.

Recently, a number of (simplified) CCD/CMOS noise models and estimation techniques have been presented in [12, 16, 17]. Next to this, many researchers address individual aspects of the noise in digital camera images (such as dealing with clipping [18], Poisson/multiplicative noise [19, 20], signal-dependent/non-stationary noise [21–23]). Compared to these efforts, relatively little attention has gone to the joint modeling of the many involved factors that contribute to the noise characteristics in a digital camera. Some notable exceptions are [17, 24, 25]: Liu et al. [17] propose Bayesian estimation of the noise level function, directly from the camera reconstructed image itself. The set of possible noise level functions is thereby derived by considering the processing operations in the camera. In [24], a parametric noise model for RAW sensor data is proposed, which is then used to estimate the optimal exposure times for high dynamic range (HDR) image acquisition. A similar approach is presented in [25] for the photon-noise limited case.

In our opinion, image reconstruction techniques based on joint noise modeling techniques will (in the long term) significantly further improve the image quality, especially when physical limits of the image sensors are reached. For example, the minimal sensor size of a digital camera is limited by signal-to-noise considerations: if one would like to have a higher image resolution, one would also have to deal with a higher level of noise in the image, even up to such a degree that further increasing the image resolution does not bring any gain in image quality. Besides this, the dynamic range of a camera sensor is still many orders lower than the dynamic range that the human visual system can deal with, and improving the dynamic range will also bring an additional image quality increase.

Nowadays, to deal with these problems, camera manufacturers integrate extra post-processing operations, such as noise removal or HDR reconstruction into the camera. Their noise removal schemes are, mostly because of power consumption, hardware complexity and cost reasons, often based on the simple Gaussian noise models. Building more sophisticated and realistic noise models bridges the gap between the more and more miniaturized camera sensor designs and the increasing image quality expectations of camera end users.

Now that our goals are made clear, it is obvious that there are a lot of aspects (e.g. image noise, resolution, dynamic range, processing artifacts,..) to cover in order to build such an accurate and realistic noise model. To limit the scope of this article, firstly we will assume a point-wise relationship between the input and the output, which can be described by a so-called camera (or intensity or brightness) response function (ignoring correlations both spatially and between different color channels). Secondly, we will limit ourselves to image artifacts that are caused by noise: we will not deal with other artifacts common in digital camera, such as chromatic abberations, lens deformations, lens flares etc. Multivariate extensions of our theory and dealing with such artifacts may be a topic of future research. Also, while building sophisticated models, it is important to keep the models as simple as possible, so that techniques based on these models are still practical.

As an elegant illustration of our noise model, we will consider the problem of the reconstruction of a HDR image from a set of low dynamic range (LDR) images where the intensity range of each image covers a different part of the whole dynamic range (technically speaking, these images are captured by using different exposure times). This problem is illustrated in Figure 1. The application to HDR reconstruction is a beautiful example where many key image quality factors meet each other: first, we have the dynamic range that is limited by the camera sensor (and analog-to-digital converters), causing over- and underexposure in the images. Second, we are dealing with image noise, which is most apparent in images taken at low exposure times (or dark environments). Third, the “true” image resolution of the final HDR image is confined by the amount of image noise in the reconstructed HDR image. Fourth, color fidelity (i.e., whether the colors in the HDR image correspond well to the colors in the real scene) is an important image quality factor.

Figure 1

HDR Reconstruction Example. (a) (c) Three out of nine LDR images of a halogen lamp taken with a Nikon D90 camera at different exposure times. (d) Synthesized HDR image (displayed with tone mapping from [26]), with details of the lamp and the background well reconstructed. Note that the LDR image in (c) contains contrast reducing lens flares and diffraction artifacts (which fall outside the scope of this article).

An important concept in HDR reconstruction is the weighting function (also called certainty function), which is introduced in [27] and subsequently used in [28–30]. The weighting function in [30] is defined as being proportional to the slope of the camera response function (CRF), indicating how quickly the output pixel intensity of a camera varies for a given input intensity. Correspondingly, certainty images are computed by applying certainty functions to digital images, revealing which parts of the image are the most “reliable”. It is found that this is the case for the midtones of the image. One of the issues however, is that several choices for weighting functions have been proposed in literature and that it is not clear which function to choose under which conditions.

In this article, we will formalize the concept of the “certainty function” in terms of a realistic camera model, and we will show that defining the certainty function as the probability that the output pixel intensity is unbiased, yields close to optimal HDR reconstruction results in the mean square error sense. The characteristics of our certainty function are similar to [30], additionally the Bayesian probabilistic formulation of our certainty function permits a more straightforward application to other problems, as we will demonstrate. To arrive at these novel certainty functions, we will develop a realistic camera noise model. Our noise model will be based on similar considerations as in [24], however the main difference is that we explicitly compute the bias functions based on an exponential prior distribution for the irradiance values, whereas in [24], simple indicator functions are used to predict whether the signal has been clipped.

The remainder of this article is organized as follows: in Section “Reconstruction of HDR images”, we introduce the synthesis problem of HDR images. Our realistic camera noise model is presented in Section “Camera noise modeling”. In Section “Probabilistic formulation of the weighting function”, we use the camera model to derive the weighting function. In Section “Estimation of the CRF”, we use the obtained weighting function in order to estimate the CRF for a set of LDR images. Results and a discussion are given in Section “Results and discussion”. Finally, Section “Conclusion” concludes this article.

Reconstruction of HDR images

Before going deep into the noise modeling for digital cameras, we first give an overview of the HDR reconstruction process and the different factors (such as SNR, the CRF and choice of the weighting function) that influence the reconstruction quality. First, we will introduce a number of general concepts that will be used throughout the article. Next, we will explain a HDR reconstruction technique that is based on a quite general noise model. Finally, we will investigate the “denoising” performance in the SNR sense of such a scheme, which will give some insight in the different factors that play a role in the quality of the final HDR image and will give an indication on conditions needed to obtain a certain minimal level of image quality, in terms of SNR.

Basic concepts

We have a set of $j=1,\dots ,P$ LDR digital photographs of a static scene at our disposal. The photographs are all taken from a fixed position, for example, using a tripod. We assume that lighting changes can be ignored, such that the incident scene radiance is constant during the exposure. For every photograph we use a different exposure time Δ t _j , which will let us recover the dynamic range of the scene. Let z _ij denote the pixel intensity of photograph j at position i, where we use a one-dimensional index to denote the spatial position (as in raster scanning). The goal of HDR reconstruction is to recover the irradiance map of the scene E _i , based on the pixel intensities z _ij of the digital photographs.

In a deterministic (noise-free) world, the image irradiance can be related to the pixel intensity of each photograph by the introduction of a CRF, as follows:

$$z_{\mathit{ij}}=\gamma \left(\Delta t_j\sqrt{{\alpha }_j}{E}_i\right)$$

(1)

where the point-wise function γ(·)is the CRF and with $\sqrt{{\alpha }_j}$ a gain factor (the squareroot is used here, for notational convenience later). The CRF models several digital post-processing operations performed in the camera after A/D conversion, such as: white balance, color corrections, demosaicing, gamma correction, edge/contrast enhancement,.. In Figure 2, an example of the CRF is given for a Nikon D90 camera. In this example, the CRF was estimated from the set of LDR images from Figure 1. This estimation technique will be discussed in more detail in Section “Estimation of the CRF”. In practice, the CRF varies from camera to camera and even depends on (some of) the settings of a particular camera. The function is assumed to be point-wise (ignoring spatial information from the local neighborhood), mainly to keep the camera modeling and estimation simple. This has the drawback that some post-processing operations (such as demosaicing, sharpening, compression, color corrections, which can not be expressed in terms of a point-wise function), can not be included in the CRF. For the forthcoming analysis, we will first assume that the LDR images are saved in a RAW image format (which is possible for most high-end camera models). Thereby, the noise modeling becomes significantly easier, since post-processing operations, which severely affect the noise characteristics, do not need to be taken into account. Later on, we will show the robustness of our approach, despite this assumption, on JPEG compressed LDR images where standard post-processing operations have been applied (see Section “HDR reconstruction from JPEG compressed LDR digital photographs”). Additionally, we assume that the CRF is monotonically increasing and hence invertible. This reflects the simple fact that relative positive changes in image irradiance at a certain position (e.g., due to illumination) should result in a positive change in the pixel intensity at the same position.

Figure 2

Camera response function and irradiance probability density function. (a) Illustration of a CRF recovered from three LDR images taken with different exposure times. As indicated by the colors, each LDR image covers a different region of the dynamic range, (b) exponential probability density function f _E (E), fitted to the irradiance histograms of three HDR images.

In real-life, there is not a one-to-one mapping of E _i to z _ij , because E _i is the ideal noise-free image irradiance, while z _ij is subject to noise. In this case, the CRF is the function that maps the measured image irradiance ${Ê}_i$ onto the pixel intensity z _ij , or mathematically speaking, $z_{\mathit{ij}}=\gamma \left(\Delta t_j\sqrt{{\alpha }_j}{Ê}_i\right)$.

The concept of the CRF function is quite general: the techniques based on the CRF can also be used when all post-processing operations are switched off, for example for processing RAW sensor data. Because most camera manufacturers do not publicly share the internal processing steps of their cameras, together with their used parameter values, the CRF is usually estimated from the photographs themselves [28, 30]. Two photographs with different exposure times are sufficient for this task.

According to (1), the image irradiance can be recovered by applying the inverse CRF to the pixel intensity and subsequently by dividing by the expose time. In the logarithmic domain, this can be expressed as (see [28], Eq. 2):

$$\text{log}{E}_i=g\left(z_{\mathit{ij}}\right)-\text{log}\Delta t_j-\frac{1}{2}\text{log}{\alpha }_j,$$

(2)

with g(z) = log(γ⁻¹(z)), the logarithm of the inverse CRF. Working in the logarithmic domain offers a number of practical advantages, such as the fact that estimates are linear in this domain. For example, writing (2) for several values of j gives a linear system of equations (in log E _i and log Δ t _j ). To reconstruct a HDR image from a set of LDR images, the pixel intensities in two different LDR images can be related to each other as follows:

$$g\left(z_{\mathit{ij}}\right)-g\left(z_{i{j}^{\prime }}\right)=\text{log}\frac{\Delta t_j}{\Delta t_{j^{\prime }}}+\frac{1}{2}\text{log}\frac{{\alpha }_j}{{\alpha }_{j^{\prime }}},(j\ne j^{\prime }).$$

(3)

As an illustration, we depicted the intensities g(z _ij ) as a function of the image irradiance E _i in Figure 3a, for three real LDR images taken with increasing exposure times Δ t _j and equal gains α _j . Due to (2) and the different exposure times, there are three lines with different offsets. The difference in offsets can be computed using (3). Recovering an image with an extended dynamic range basically comes down to setting the offsets to zero (Figure 3b).

Figure 3

Illustration of the recovery of irradiance values from a set of LDR images. (a) Inverse CRF compensated pixel intensities g(z _ij ) as a function of the image irradiance E _i , for three LDR photographs taken with different exposure times. (b) Recovering an image with a HDR by subtracting the offsets $\text{log}\left(\Delta t_j/\Delta t_{j^{\prime }}\right)$ (here, j^′ = 2).

Noise model-based HDR reconstruction

In practice, the image irradiance measurements E _i are subject to statistical fluctuations called quantum or photon noise, caused by uncertainty associated by “counting” light energy quanta. Let ${Ê}_i\Delta t_j$ denote an energy measurement after an exposure of Δ t _j secs. The final pixel intensity of the LDR image j at position i is then given by $z_{\mathit{ij}}=\gamma \left({Ê}_i\sqrt{{\alpha }_j}\Delta t_j\right)$ Because of the statistical nature of ${Ê}_i$, z _ij will be a random variable as well. The probability density function of z _ij is a complicated function in general, because:

• The measurement ${Ê}_i$ is affected by several (both additive and multiplicative) sources of noise, as we will discuss in more detail in Section “Camera noise modeling”.

• A priori, little is known about the CRF γ(·)(except for the assumption of monotonicity).

Our solution to these problems is to firstly characterize the distribution of ${Ê}_i\Delta t_j$ (or equivalently g(z _ij ), if we exploit the fact that the CRF γ(·) is invertible, see Section “Basic concepts”). Secondly, we compute the distribution of z _ij through a change of variables. In Section “Camera noise modeling” we will explain that, under quite general circumstances, g(z _ij ), conditioned on the log-irradiance, is well represented by the Gaussian distribution:

$$\begin{array}{ccc}g\left(z_{\mathit{ij}}\right)\hfill & |\hfill & \text{log}{E}_i\sim \mathrm{N}\left(\text{log}\sqrt{{\alpha }_j}\Delta t_j\right.\hfill \\ \left.+\text{log}{E}_i+\nu \left(E_i,\Delta t_j\right),{\sigma }^2\left(E_i,\Delta t_j\right)\right),\hfill \end{array}$$

(4)

with ν(E _i ,Δ t _j ) a bias term and with σ²(E _i ,Δ t _j ) an irradiance and exposure time dependent variance (also see Figure 4). In Figure 5a, a probability density plot of g(z _ij ) | logE _i is given. The point clouds correspond to histograms obtained after Monte-Carlo simulation, while the solid line is the approximated PDF in (4). Here, the asymmetry of the point cloud is mostly caused by taking the logarithm of the sum of a Poissonian and Gaussian distributed random variables. The asymmetry cannot be modeled well using the Gaussian distribution, however, in this work only the first two statistical moments (mean and variance) are of most importance.

Figure 4

Overview of the camera noise model presented in this paper..

Figure 5

Probability density plots for Δt  = 1/20 s , α  = 5, β  = 1 and ${\sigma }_{\text{o},j}^2=1$.

The conditional Gaussianity of the PDF allows us to make a number of considerations, as we will show next. The HDR image can directly be reconstructed using the maximum likelihood method^b for estimating the mean of a Gaussian distribution (i.e., by maximizing the log-likelihood function of the data):

$$\begin{array}{ccc}\widehat{\text{log}{E}_i}\hfill & =\hfill & \text{arg}\underset{\text{log}{E}_i}{\text{max}}\hfill \\ -\sum _{j=1}^P\frac{{\left|g\left(z_{\mathit{ij}}\right)-\text{log}\sqrt{{\alpha }_j}\Delta t_j-\text{log}{E}_i-\nu \left(E_i,\Delta t_j\right)\right|}^2}{{\sigma }^2\left(E_i,\Delta t_j\right)}.\hfill \end{array}$$

(5)

The factor 1/σ²(E _i ,Δ t _j ) takes into account that the noise variance (and correspondingly the SNR) is both exposure time and irradiance dependent, and consequently penalizes images with a high noise variance (low SNR) in the HDR reconstruction.

In practice, the “true” bias ν(E _i Δ t _j ) is unknown and difficult to estimate from the LDR image data, due to the availability of a limited number of LDR images (typically, P < 10). We will explain later in Section “Camera noise modeling” that, for z _ij in the middle of the camera’s exposure range (e.g., z _ij  = 128 for 8-bit data), the bias is negligibly small compared to the image irradiance (|ν(E _i ,Δ t _j )| ≪ logΔ t _j + logE _i ) and hence can be ignored. A simple but different way to deal with the bias problem is to omit the term ν(E _i , Δ t _j ) and to introduce a weighting function w _ij  = w(z _ij ) as in [27–30]:

$$\begin{array}{c}\widehat{\text{log}{E}_i}=\text{arg}\underset{\text{log}{E}_i}{\text{min}}\sum _{j=1}^P\frac{w_{\mathit{ij}}}{{\sigma }^2\left(E_i,\Delta t_j\right)}\\ {\left|g\left(z_{\mathit{ij}}\right)-\text{log}\sqrt{{\alpha }_j}\Delta t_j+\text{log}{E}_i\right|}^2.\end{array}$$

(6)

A solution to this weighted least squares problem (6) is then given by:

$$\widehat{\text{log}{E}_i}=\frac{{\sum }_{j=1}^P{w}_{\mathit{ij}}{\sigma }^{-2}\left(E_i,\Delta t_j\right)\left(g\left(z_{\mathit{ij}}\right)-\text{log}\sqrt{{\alpha }_j}\Delta t_j\right)}{{\sum }_{j=1}^P{w}_{\mathit{ij}}{\sigma }^{-2}\left(E_i,\Delta t_j\right)},$$

(7)

which we will call the main HDR reconstruction formula. The estimated log-irradiance is a weighted average of g(z _ij ) − log Δ t _j , where each term can be interpreted as an estimate of the log-irradiance for each individual LDR image. A schematic overview of this reconstruction method is given in Figure 6.

Figure 6

(a) Overview of the HDR reconstruction algorithm, (b) weighting functions used for HDR reconstruction (for the proposed method, the variance-dependent weight σ ⁻² ( E _i , Δ t _j ) is included in this comparison).

Let us now turn to the choice of the weighting function, which is crucial for the correct operation of the HDR algorithm, since the weighting function determines the trade-off between dynamic range and SNR. In literature, several functions have been proposed:

• Debevec and Malik [28] propose a triangular function to emphasize the middle of the exposure range, mainly for its simplicity:

  $$w_{\mathit{ij}}\left(z\right)=\left\{\begin{array}{ll}z-z_{\text{min}}& z\le z_{\text{mid}}\\ z_{max}-z& z>z_{\text{mid}}\end{array}\right.$$

  (8)

  with $z_{\text{mid}}=\frac{1}{2}\left(z_{\text{min}}+z_{max}\right)$. We will see in Section “Dealing with unknown camera model parameters” that this weighting function is a good choice when little information is known about the camera noise characteristics.

• Mann and Picard [27, 30] select a weight related to the slope of the CRF γ, which indicates how quickly the pixel intensity z _ij varies with the input, in order to assign lower weights to coarser quantized pixel intensities. It is argued that for noisy input images, influences of quantization noise are minimal in the middle of the camera’s exposure range. Using our notations, the weighting function is defined as follows ([30], Eq. 11):

  $$w_{\mathit{ij}}\left(z\right)=\frac{\mathrm{d}z}{\mathrm{d}\text{log}{\widehat{q}}_j\left(z\right)}={\widehat{q}}_j\left(z\right)\frac{\mathrm{d}\gamma }{\mathrm{d}z}\left({\widehat{q}}_j\left(z\right)\right)$$

  (9)

  where ${\widehat{q}}_j\left(z\right)={\gamma }^{-1}\left(z\right)/\left(\Delta t_j\sqrt{{\alpha }_j}\right)$ is an estimate for the image irradiance, based on exposure j. For an identity CRF, the weights are approximately proportional to the image irradiance.

• Mitsunaga and Nayar [29] relate the weights to the SNR of the LDR image (which is assumed to be linear to the image irradiance). Thereby, the authors assume that the measurement noise is both stationary and independent of the underlying signal. This results in the following equation:

  $$w_{\mathit{ij}}\left(z\right)={\widehat{q}}_j\left(z\right)/\frac{\mathrm{d}{\widehat{q}}_j\left(z\right)}{\mathrm{d}z}\left(z\right)={\widehat{q}}_j\left(z\right)\frac{\mathrm{d}\gamma }{\mathrm{d}z}\left({\widehat{q}}_j\left(z\right)\right),$$

  (10)

  where the rule for the derivative of the inverse was applied. The weighting function proposed by Mitsunaga and Nayar is the same as the weighting function of Mann.

• In previous work [31], we proposed an exponential power function as a trade-off between noise suppression and clipping:

  $$w_{\mathit{ij}}\left(z\right)=exp\left(-\frac{\left(z-z_{\text{mid}}\right)}{b{\left(z_{\text{mid}}\right)}^a}\right),$$

  (11)

  where a and b are constants which were determined experimentally. This function also emphasizes the middle of the exposure range, but in such a way that the low and high intensities are still assigned a large weight, close to 1.

• Other weighting functions are proposed by Reinhard et al. [32], Tsin et al. [33], Kirk and Andersen [34].

We remark that in many of the above works, the authors assume AWGN that is independent of the image irradiance and image index j, which leads to a special case of (6) in which σ² (E _i ,Δ t _j ) is constant (consequently this factor can be dropped in (7)). In general, because of the dependency of the weighting factor σ⁻² (E _i ,Δ t _j ) on the unknown image irradiance E _i , $\widehat{\text{log}{E}_i}$ needs to be estimated as in an iteratively re-weighted least squares approach. During this iterative process, in the right handed side of (7), the image irradiance estimate E _i of the previous iteration is used. The image irradiance is then estimated using the currently best available estimate of the weight w _ij σ⁻² (E _i ,Δ t _j ). Once a better estimate E _i becomes available, the combined weights w _ij σ⁻² (E _i ,Δ t _j ) are updated.

Such an iterative updating scheme can be avoided (which is advantageous from a computational point of view), even without assuming AWGN: if σ⁻²(E _i ,Δ t _j ) is proportional to (E _i )^q, the factors (E _i )^q, which are independent of the summing variable j, in the numerator and denominator of (7) cancel each other. In the next section, we will perform a thorough modeling of the camera noise characteristics, to obtain explicit formulas for σ⁻² (E _i ,Δ t _j ). Surprisingly, our result in Section “Camera noise modeling” indicates that an approximation such as σ⁻² (E _i ,Δ t _j )∝(Δ t _j E _i )^q with q = 1 or 2 are quite adequate for a digital camera noise model.

Denoising performance of the HDR reconstruction formula

As we explained before, the final reconstructed HDR image is a weighted average of the different LDR images, with a correction term to compensate the different exposure times. Because of the averaging (which increases the SNR), denoising is intrinsically part of HDR reconstruction. To gain some insight in the different factors that determine the quality of the reconstructed HDR image, it is useful to investigate the “denoising performance” of the main HDR reconstruction formula (7). To start, we will perform our following analysis for a constant weighting function $w_{\mathit{ij}}=1,j=1\dots ,P$. Based on (7), we can write the variance of $\widehat{\text{log}{E}_i}$ as:

$$\begin{array}{ccc}\text{Var}\left[\widehat{\text{log}{E}_i}\right]\hfill & =\hfill & \frac{{\sum }_{j=1}^P{\sigma }^{-4}\left(E_i,\Delta t_j\right)\text{Var}\left[g\left(z_{\mathit{ij}}\right)\right]}{{\left[{\sum }_{j=1}^P{\sigma }^{-2}\left(E_i,\Delta t_j\right)\right]}^2}\hfill \\ =\hfill & {\left[\sum _{j=1}^P{\sigma }^{-2}\left(E_i,\Delta t_j\right)\right]}^{-1}.\hfill \end{array}$$

(12)

Exploiting the fact that the SNR is defined in terms of the ratio of the signal energy ((logE _i )²) and the noise energy $\left(\text{Var}\left[\widehat{\text{log}{E}_i}\right]\right)$, the signal-to-noise ratio of the reconstructed HDR image can be expressed in an elegant formula:

$$\begin{array}{ll}{\text{SNR}}_{\text{HDR}}& =10{\text{log}}_{10}\left({\left(\text{log}{E}_i\right)}^2/\text{Var}\left[\widehat{\text{log}{E}_i}\right]\right)\\ =20{\text{log}}_{10}\text{log}{E}_i\\ +10{\text{log}}_{10}\sum _{j=1}^P{\sigma }^{-2}\left(E_i,\Delta t_j\right)\left[\text{dB}\right].\end{array}$$

(13)

We will later see, that, under some conditions (more specifically, when photon noise is dominant), σ⁻²(E _i ,Δ t _j ) ≈ E _i Δ t _j , which allows us to approximate the SNR as:

$$\begin{array}{c}{\text{SNR}}_{\text{HDR}}\approx 20{\text{log}}_{10}\left(E_i\text{log}{E}_i\right)+10{\text{log}}_{10}\sum _{j=1}^P\Delta t_j\left[\text{dB}\right].\end{array}$$

(14)

such that each extra LDR image increases the SNR by maximally^c 10log₁₀Δ t _j dB (see Figure 7). For this model, three factors can increase the SNR: (1) increasing the light incidence of the scene (for example by example by turning on an additional light source), (2) increasing the exposure Δ t _j and (3) increasing the exposures P. Obviously, some of these measures come at a cost: for example, when increasing the exposure or the number of exposures, the scene stationary assumption may not hold any longer due to object motion. However, the formula SNR_HDR > 0 dB expresses necessary (although not sufficient) conditions required for practical HDR reconstruction. HDR reconstruction is not appropriate if the SNR reaches zero (SNR_HDR < 0 dB, or equivalently ${\sum }_{j=1}^P\Delta t_j<{\left(E_i\text{log}{E}_i\right)}^2$). In this case, the so-called Wyckoff signal-noise criterion [30] is violated: the exposure is not high enough to overcome the sensor noise.

Figure 7

Maximal theoretical contribution of each LDR image in the overall SNR of the reconstructed HDR image and the influence of the weighting function..

In case the weighting function is non-constant, the variance becomes:

$$\begin{array}{ccc}\text{Var}\left[\widehat{\text{log}{E}_i}\right]\hfill & =\hfill & \frac{{\sum }_{j=1}^P{\sigma }^{-2}\left(E_i,\Delta t_j\right)w_{\mathit{ij}}^2}{{\left[{\sum }_{j=1}^P{\sigma }^{-2}\left(E_i,\Delta t_j\right)w_{\mathit{ij}}\right]}^2}\hfill \\ \ge {\left[\sum _{j=1}^P{\sigma }^{-2}\left(E_i,\Delta t_j\right)\right]}^{-1}.\hfill \end{array}$$

(15)

In other words, non-constant weighting function, such as the weighting functions proposed in literature, lead to a reduced SNR_HDR. Figure 7 also illustrates how the SNR of the reconstructed HDR image is affected by using different weighting functions. We will go deeper into this topic in Section “Probabilistic formulation of the weighting function”.

Camera noise modeling

In Section “Noise model-based HDR reconstruction” we have put forward a Gaussian model for the inversely compensated pixel intensity g(z _ij ). This Gaussian model comprises next to the mean log(Δ t _j E _i ), a bias function ν(E _i Δ t _j ) and a noise variance function σ²(E _i ,Δ t _j ). For accurate HDR reconstruction, expressions for these functions are indispensable. These expressions through camera noise modeling. Our noise modeling and the way in which we deal with clipping is similar to the Poissonian–Gaussian modeling of [18, 35], with the main difference that our analysis covers the more general case in which the CRF is nonlinear (see Appendix 1). But before building a camera noise model, we first need a better understanding of the processing inside a digital camera.

In Figure 8, a simplified block scheme is given of the processing pipeline of a digital camera. First, the incident light of the scene enters the camera sensor through the lens. Because the image irradiance E _i is constant during the expose of length Δ t _j , the energy registered by the sensor will then roughly be the product of the image irradiance and the exposure time. As already mentioned, the measured energy is subject to photon noise. The photon noise is mostly noticeable in the image when taking photographs in dark environments or at low exposure times. A commonly used probability density model for the sensor noise is the Poisson distribution:

$$x_{\mathit{ij}}\sim \mathcal{P}\left(\Delta t_j{E}_i\right)$$

(16)

where x _ij is the measured intensity by the photo-detector at position i. We will model the photon noise using a Gaussian distribution with the same first and second order moments as the Poisson distribution, i.e. x _ij  ∼ N(Δ t _j E _i ,Δ t _j E _i ), which is a good approximation if Δ t _j E _i  ≫ 0. This is the case for consumer digital cameras [13] and also considerably simplifies the modeling task.^d In the next step, x _ij is amplified and quantized by the analog-to-digital converter (A/D) to B bits. More precisely, the analog voltages x _ij measured by the photo-detectors are subjected to the following operations:

• 1. Amplification, which introduces electronic noise. This electronic noise is well characterized by a Gaussian distribution $N\left(0,{\sigma }_{є}^2\right)$. We will denote by $\sqrt{{\alpha }_j}$ the amplification gain for image j, resulting in a signal with expected value $\sqrt{{\alpha }_j}\Delta t_j{E}_i$. The amplification gain is usually determined by the ISO sensitivity of the camera.

• 2. The measured signal x _ij is affected by non-uniform heating of the sensor, and becomes non-uniform, even when the scene radiance is constant. The resulting fluctuations are called fixed-pattern noise (FPN), also known as dark current non-uniformity. FPN has a variance that is quadratic in the image irradiance (mainly caused by variations by variations in the gain factor α _j , see [36]), but also contributes to the overall noise that is independent of the image irradiance (as an offset term). We will call these two noise components respectively gain FPN and offset FPN.

• 3. A/D conversion, which involves clipping of the dynamic range to the range 0–2^B − 1 and quantization. The camera sensor can only map irradiance values that are within certain minimum and maximum bounds (these bounds determine the tonal range of the sensor) and values that are outside this range are clipped. Because the tonal range of the sensor often cannot cover the whole dynamic range of the scene, some regions in the image will be over- or underexposed. In the following f(x) = max(0,min(2^B − 1,x)) will denote this clipping function.

• 4. For ideal A/D converters, the quantization noise is uniformly distributed in the range [−1/2,1/2]. In this work, we will treat quantization noise similarly to electronic noise and offset FPN. This is possible since quantization is performed directly after amplification. Therefore, the offset noise variance ${\sigma }_{\mathrm{o},j}^2$ signifies the summed contributions of of the variances of the electronic noise, offset FPN and quantization noise.

• 5. Finally, the quantized signal is subjected to several post-processing steps in the digital domain. Examples are: gamma correction, brightness and contrast adjustment, color corrections, white balance, compression/expansion etc. As we mentioned before, these operations are modeled using the CRF.

Figure 8

Simplified block scheme of a digital camera..

The different processing steps outlined above significantly alter the statistical properties of the measured intensities in Equation (16). In this work we separate linear effects from nonlinear effects and model them separately. For example, while amplification (linearly) magnifies the intensity by a factor $\sqrt{{\alpha }_j}$, the variance of the intensity is multiplied by a factor α _j . Similarly, adding offset noise (a linear additive process), increases the variance by ${\sigma }_{\mathrm{o},j}^2$. The clipping function and the CRF are both nonlinear functions. Because of the order of the steps 1–5, the nonlinear functions are positioned at the back of the chain, which simplifies the modeling task. y _ij the signal after applying steps 1–4, then we have the statistical dependency x _ij  → y _ij  → z _ij . Using the Gaussian approximation of x _ij , the PDF of y _ij can readily be found (see Figure 4):

$$y_{\mathit{ij}}\sim \mathrm{N}\left(\sqrt{{\alpha }_j}\Delta t_j{E}_i,{\sigma }_{\mathrm{o},j}^2+{\alpha }_j\Delta t_j{E}_i+{\beta }_{\mathit{ij}}{\left(\Delta t_j{E}_i\right)}^2\right),$$

(17)

where β _ij is a gain of FPN whose variance increases as a quadratic function of E _i (see step 2). To check the normality of y _ij , a normal probability plot is provided in Figure 5b. It can be noted that the Gaussian approximation is quite accurate, even for small irradiance values (e.g., Δ t _j E _i  < 100). Next, the measured pixel intensities z _ij are related to y _ij by the nonlinear relationship:

$$z_{\mathit{ij}}=\gamma \left(f\left(y_{\mathit{ij}}\right)\right).$$

(18)

Now, the noise variance σ² (E _i ,Δ t _j ) and bias ν (E _i ,Δ t _j ) can be computed from the statistical moments of logf(y _ij ) using the definitions of mathematical expectation and variance, as shown in (19).

$$\begin{array}{l}\nu \left(E_i,\Delta t_j\right)=\mathrm{E}\left[\text{log}f\left(y_{\mathit{ij}}\right)|E_i\right]-\text{log}f\left(\sqrt{{\alpha }_j}\Delta t_j{E}_i\right)\\ =C\underset{-\infty }{\overset{+\infty }{\int }}\text{log}\frac{f\left(u\right)\mathrm{d}u}{f\left(\Delta t_j{E}_i\right)}exp\\ \times \left(-\frac{1}{2}\frac{{(\sqrt{{\alpha }_j}\Delta t_j{E}_i-u)}^2}{s\left(\Delta t_j{E}_i\right)}\right)\\ {\sigma }^2\left(E_i,\Delta t_j\right)=\text{Var}\left[\text{log}f\left(y_{\mathit{ij}}\right)|E_i\right]\\ =C\underset{-\infty }{\overset{+\infty }{\int }}{\left(\text{log}f\left(u\right)\right)}^2\mathrm{d}uexp\\ \times \left(-\frac{1}{2}\frac{{(\sqrt{{\alpha }_j}\Delta t_j{E}_i-u)}^2}{s\left(\Delta t_j{E}_i\right)}\right)\\ -C\left[\underset{-\infty }{\overset{+\infty }{\int }}\text{log}f\left(u\right)\mathrm{d}uexp\right.\\ {\left.\times \left(-\frac{1}{2}\frac{{(\sqrt{{\alpha }_j}\Delta t_j{E}_i-u)}^2}{s\left(\Delta t_j{E}_i\right)}\right)\right]}^2\end{array}$$

(19)

where $s\left(x\right)={\sigma }_{\mathrm{o},j}^2+{\alpha }_{j}x+{\beta }_{\mathit{ij}}{x}^2$ and C = (2Πs(Δ t _j E _i ))^−1/2.

Although the functions in (19) can be calculated numerically, it is not directly clear how σ² (E _i ,Δ t _j ) is related to the parameters ${\sigma }_{\mathrm{o},j}^2$, α _j , β _ij and Δ t _j . Thereby it becomes very difficult to, e.g., devise techniques to estimate these parameters from the data z _ij .

In Appendix 1, we describe a technique that allows us to approximate the statistical moments of a general nonlinear function of y _ij , resulting in far simpler expressions. The premise is that the technique cannot be used when the nonlinear function is not differentiable. This is the case for the clipping function near its clipping points. If we disregard the clipping function (we will explain in a few moments how to deal with clipping), using Appendix 1, the statistical moments are well approximated by:

$$\begin{array}{ll}\nu \left(E_i,\Delta t_j\right)& \approx -\frac{1}{2}\frac{{\sigma }_{\mathrm{o},j}^2+{\alpha }_j\Delta t_j{E}_i}{{\alpha }_j\Delta t_j^2{E}_i^2}-\frac{{\beta }_{\mathit{ij}}}{2{\alpha }_j}\end{array}$$

(20)

and

$$\begin{array}{ll}{\sigma }^2\left(E_i,\Delta t_j\right)& \approx \frac{{\sigma }_{\mathrm{o},j}^2+{\alpha }_j\Delta t_j{E}_i}{{\alpha }_j\Delta t_j{E}_i}+\frac{{\beta }_{\mathit{ij}}}{{\alpha }_j}.\end{array}$$

(21)

Interestingly, the gain FPN appears as an extra bias term in (1). Since FPN does not change from one image to another, a FPN template can be constructed for each ISO setting. Gain FPN removal then simply consists of subtracting the template −β _ij /2α _j from logf(y _ij ), the logarithm of the clipped digitized signal. This bias subtraction approach is quite common in digital cameras.

Let us now analyze the noise variance (21). In case the photon noise is dominant, i.e. ${\alpha }_j\Delta t_j{E}_i\gg {\sigma }_{\mathrm{o},j}^2$ and α _j  ≫ Δ t _j E _i β _ij , inversely proportional to Δ t _j E _i . On the other hand, if the gain FPN is dominant, the noise variance is inversely proportional to $\Delta t_j^2{E}_i^2$:

$$\begin{array}{ll}{\sigma }^2\left(E_i,\Delta t_j\right)& \approx \left\{\begin{array}{ll}\frac{1}{\Delta t_j{E}_i}& \text{(photon noise limited region)}\\ \frac{{\sigma }_{\mathrm{o},j}^2}{{\alpha }_j\Delta t_j^2{E}_i^2}& \text{(offset noise limited region)}\\ \frac{{\beta }_{\mathit{ij}}}{{\alpha }_j}& \text{(gain FPN limited region)}\end{array}\right.\end{array}$$

(22)

such that σ⁻² (E _i ,Δ t _j ) can be approximated by a monomial in E _i . From Equation (22), the SNR of each image can easily be computed as $\text{SNR}=20{\text{log}}_{10}\text{log}\left(\sqrt{{\alpha }_j}\Delta t_j{E}_i\right)/\sigma \left(E_i,\Delta t_j\right)$. The SNR for the dominant photon and offset noise cases are equal for $E_i={\sigma }_{\text{o},j}^2/\left({\alpha }_j\Delta t_j\right)$, the SNR breaking point. In Figure 9, a plot of the SNR is given as a function of the image irradiance. While the “exact” SNR is a complicated nonlinear function in general (see (19)), the function can be well approximated by a piecewise linear function (in a logarithmic scale on both the horizontal and vertical axes). As explained at the end of Section “Noise model-based HDR reconstruction”, this finding allows us to develop efficient and practical HDR reconstruction algorithms based on a real camera noise model.

Figure 9

The SNR of a HDR image in presence of both offset noise and photon noise. In this example, Δt = 5 s., α = 1 and ${\sigma }_{\text{o},j}^2=100$.

One of our main issues at this stage is that the approximations (1) and (21) only hold for differentiable CRFs, hence clipping of the dynamic range is not incorporated in our model. To resolve this issue, in Appendix 2, we derive bounds for $\Delta t_j{E}_i\sqrt{{\alpha }_j}$ for which the approximations (1) and (21) are accurate (up to an arbitrary precision). If we denote the minimal and maximal intensity values (after applying the clipping function) as, respectively y_min and y_max, we have shown that the results hold (up to an arbitrary precision) in a “clipping-free” region $y_{\text{min}}^{\prime }\le \Delta t_j{E}_i\sqrt{{\alpha }_j}\le y_{\text{max}}^{\prime }$, where typically $y_{\text{min}}^{\prime }<\left(y_{\text{min}}+y_{\text{max}}\right)/2<y_{\text{max}}^{\prime }$ (with $y_{\text{min}}^{\prime }$ and $y_{\text{max}}^{\prime }$ as in Appendix 2). An illustration is given in Figure 10, for Δt = 1s, α = 1 and ${\sigma }_{\mathrm{o},j}^2=0.01$.

Figure 10

Exact biases and variances, numerically computed using ( 19), their approximations obtained through ( 1) and ( 21) and the computed bounds $[y_{\text{min}}^{\prime },y_{\text{max}}^{\prime }]$ for which these approximations are accurate..

Even though complicated processing by several nonlinear functions is involved, we can well describe the first and second statistical moments of the variables g(z _ij ) using a convenient formula that holds with great accuracy within certain bounds. This finding will turn out to be very useful for applications that depend on this noise model. In the next sections, we will take these bounds into account in the HDR reconstruction, in particular for designing an appropriate weighting function.

Probabilistic formulation of the weighting function

The camera noise model from previous section can relatively easily be applied in many practical applications. The main ingredients of this model are the approximations of the noise variance function and the bias function, together with the ranges $[y_{\text{min}}^{\prime },y_{\text{max}}^{\prime }]$ on which these approximations are accurate.

From the explanation in Section “Denoising performance of the HDR reconstruction formula”, the reader may expect that an optimal weighting function is one that is constant everywhere in the clipping-free regions (w _ij  = 1 − I(z _ij  ≤ 0) − I(z _ij  ≥ 2^B − 1), with I(·) the indicator function). However, such weighting function does not give maximal SNR because of bias effects: for example, an initially very large intensity y _ij may become less than 2^B − 1 with a certain probability, due to addition of offset noise or FPN, but without being affected by the clipping operation. On average, several of these initially large intensities cause a biased HDR image estimate, when the weighting function is not properly chosen. In this section, we will show how the camera noise model can be used to compute a near optimal weighting (or certainty) function in terms of SNR to be used in combination with the reconstruction formula (7).

MMSE-based estimation of the weighting function

Before deriving the weighting function, we make two observations on the requirements for the weighting function. As already mentioned, we would like to have an HDR reconstructed image that is unbiased: $E\left[\widehat{\text{log}{E}_i}\right]=\text{log}{E}_i$. Practically speaking, this condition means that the reconstructed log-irradiance is a true estimate of the “ideal” image log-irradiance. Because the brightness perception of the human eye is approximately logarithmic in the irradiance values (this is known as the Weber–Fechner law), we use the logarithm of the image irradiance. Using (7) and (4), the condition of unbiased estimates is equivalent to:

$$E\left[\widehat{\text{log}{E}_i}\right]=\text{log}{E}_i+\frac{{\sum }_{j=1}^P{w}_{\mathit{ij}}{\sigma }^{-2}\left(E_i,\Delta t_j\right)\nu \left(E_i,\Delta t_j\right)}{{\sum }_{j=1}^P{w}_{\mathit{ij}}{\sigma }^{-2}\left(E_i,\Delta t_j\right)}.$$

(23)

Here we simply computed the mathematical expectation of both sides of (7). Note that a sufficient condition for an unbiased estimate is given by ν(E _i ,Δ t _j ) ≠ 0 ⇒ w _ij  = 0. Our approximation theory (Appendix 2) then states that the latter condition is the case if $y_{\text{min}}^{\prime }\le \Delta t_j{E}_i\sqrt{{\alpha }_j}\le y_{\text{max}}^{\prime }$.

Next to having unbiased estimates, the SNR of $\widehat{\text{log}{E}_i}$ is an important factor, indicating the quality of the reconstructed HDR image. To take both effects into account, we will derive the Bayesian minimum mean square error (MMSE) estimator for our problem. We will minimize the MSE between the estimated log-irradiance and its true value, which is defined by:

$$\begin{array}{ccc}\text{MSE}\hfill & =\hfill & E\left[{\left(\widehat{\text{log}{E}_i}-\text{log}{E}_i\right)}^2\right]\hfill \\ =\hfill & \underset{\text{variance term}}{\underbrace{\mathit{Var}\left[\widehat{\text{log}{E}_i}\right]}}+\underset{\text{squared bias term}}{\underbrace{{\left(E\left[\widehat{\text{log}{E}_i}-\text{log}{E}_i\right]\right)}^2}}\hfill \end{array}$$

(24)

where we split up the MSE into two terms: the first term is closely related to the SNR of the reconstructed HDR image (see Equation (13)), the second term is the expected squared bias error. Next, we want to minimize the MSE with respect to the values of the weighting function: ${\text{min}}_{w_{\mathit{ij}}}\text{MSE}$. The weighting function w _ij is positive w _ij  ≥ 0, and additionally we add the constraint max _i w _ij  = 1 to get rid of the scaling ambiguity.^e For the HDR reconstruction formula from Equation (7), the different terms of the MSE can be written as follows:

$$\begin{array}{cc}\mathit{Var}\left[\widehat{\text{log}{E}_i}\right]& =\frac{{\sum }_{j=1}^P{\sigma }^{-2}\left(E_i,\Delta t_j\right)w_{\mathit{ij}}^2}{{\left[{\sum }_{j=1}^P{w}_{\mathit{ij}}{\sigma }^{-2}\left(E_i,\Delta t_j\right)\right]}^2},\end{array}$$

(25)

$$\begin{array}{cc}E\left[\widehat{\text{log}{E}_i}-\text{log}{E}_i\right]& =\frac{{\sum }_{j=1}^P{w}_{\mathit{ij}}{\sigma }^{-2}\left(E_i,\Delta t_j\right)\nu \left(E_i,\Delta t_j\right)}{{\sum }_{j=1}^P{w}_{\mathit{ij}}{\sigma }^{-2}\left(E_i,\Delta t_j\right)}.\end{array}$$

(26)

Note that both the variance and squared bias term have the same denominator, which will allow us to find an explicit linear solution to this minimization problem, as we will explain next. However, because taking the square of the bias term (26) results in several cross-terms $w_{\mathit{ij}}{w}_{i{j}^{\prime }}$, which eventually leads to a linear system of equations with a non-diagonal system matrix, we will look for a solution in which these cross terms do not appear. This will have as a practical benefit that the weighting function for each LDR image can be calculated solely based on the model parameters of the particular image, which permits offline computation of the weighting function. To achieve this mathematically, we will minimize an upper bound for the MSE. This upper bound should be selected carefully, close to the true MSE, such that the corresponding solution can be considered to be a good approximation to the solution of the original problem (Equation (24)). Using the inequality of Cauchy-Schwartz,^f such an upper bound can easily be found:

$$\begin{array}{c}\text{MSE}\le \frac{{\sum }_{j=1}^P{\left[w_{\mathit{ij}}{\sigma }^{-2}\left(E_i,\Delta t_j\right)\right]}^2\left({\sigma }^2\left(E_i,\Delta t_j\right)+P{\nu }^2\left(E_i,\Delta t_j\right)\right)}{{\left({\sum }_{j=1}^P{w}_{\mathit{ij}}{\sigma }^{-2}\left(E_i,\Delta t_j\right)\right)}^2}.\end{array}$$

(27)

In Appendix 3 the general solution for the minimization problem of (27) is derived. Using this result, we can write the optimal weighting function (minimizing (27) and relying on σ²(E _i , Δ t _j ) + P ν²(E _i , Δ t _j ) > 0)^g as:

$$w_{\mathit{ij}}=A\frac{{\sigma }^2\left(E_i,\Delta t_j\right)}{{\sigma }^2\left(E_i,\Delta t_j\right)+P{\nu }^2\left(E_i,\Delta t_j\right)}$$

(28)

with $A={\left[\underset{j^{\prime }}{max}{\sigma }^2\left(E_i,\Delta t_{j^{\prime }}\right)/\left({\sigma }^2\left(E_i,\Delta t_{\mathrm{\prime j}}\right)+P{\nu }^2\left(E_i,\Delta t_{j^{\prime }}\right)\right)\right]}^{-1}$ a normalization factor. According to (28), the optimal weights minimizing (27), can be found by computing the proportion of the noise variance σ² (E _i ,Δ t _j ) compared to the whole σ² (E _i ,Δ t _j ) + P ν² (E _i ,Δ t _j ). We remark that when the bias ν(E _i ,Δ t _j ) = 0, the corresponding weight is equal to one. When the bias increases, the weight becomes smaller. Readers familiar with Wiener filters will note that (28) resembles the classical scalar Wiener filter weight formula, however (28) is not related to the Wiener filter because P ν²(E _i ,Δ t _j ) is not a signal energy measure but a measure for the signal bias.

Although Equation (28) minimizes (27), the major issue is that the image irradiance E _i is unknown in practice. Therefore we wish to express the weighting function in terms of the observed z _ij , and not in terms of the unknown E _i . The pixel intensity z _ij (which is a random variable) statistically depends on the image irradiance E _i , through (4). Hence, we can rewrite the weighting function as:

$$\begin{array}{ll}{w}_{\mathit{ij}}^{\left(1\right)}& =\mathrm{E}\left[\left.\frac{A{\sigma }^2\left(E_i,\Delta t_j\right)}{{\sigma }^2\left(E_i,\Delta t_j\right)+P{\nu }^2\left(E_i,\Delta t_j\right)}\right|z_{\mathit{ij}}\right]\\ =\underset{0}{\overset{+\infty }{\int }}\frac{A{\sigma }^2\left(E_i,\Delta t_j\right)}{{\sigma }^2\left(E_i,\Delta t_j\right)+P{\nu }^2\left(E_i,\Delta t_j\right)}{f}_{E|z}\left(E_i\left|z_{\mathit{ij}}\right.\right)\mathrm{d}{E}_i\end{array}$$

(29)

where the conditional PDF f_E|z (E _i |z _ij ) can be computed through Bayes’ rule. This requires the specification of two other probability density functions:

• 1. The prior PDF of the image irradiance f _E (E _i ), for which we use a prior distribution of maximal entropy since little information about the image irradiance is known: the exponential distribution. The parameter of the exponential distribution (i.e., the average image irradiance) is assumed to be prior knowledge. To verify whether the exponential distribution is representative for real world images, we performed a simple experiment using three HDR images (in particular, the images from Figures 1d, 11a, and 12a). The obtained histogram and the fitted exponential distribution are shown in Figure 2b. It can be noted that there is a relatively good correspondence (i.e., the distribution well captures the high positive skew). Alternatively, the gamma distribution or even mixtures of gamma distributions may be used as in [25] (note in this respect that the exponential distribution arises as a special case).

• 2. The conditional PDF f _z|E(z _ij |E _i ), which can be computed exactly from (4) through the change of variables method. More specifically, we have:

  $$g\left(z_{\mathit{ij}}\right)\left|E_i\right.\sim \mathrm{N}\left(\text{log}\left(\sqrt{{\alpha }_j}\Delta t_j{E}_i\right)+\nu \left(E_i,\Delta t_j\right),{\sigma }^2\left(E_i,\Delta t_j\right)\right),$$

Figure 11

Desk still life: HDR reconstruction results for RAW data, after exposure compensation, while balancing and tone mapping..

Figure 12

TELIN Lab: HDR reconstruction results for LDR images stored in JPEG format, after tone mapping. The difference can be noted in the dark regions of the color chart on the left.

such that the PDF f_z|E(z _ij |E _i ) is given by:

$$\begin{array}{ccc}{f}_{z|E}\left(z_{\mathit{ij}}\left|E_i\right.\right)\hfill & =\hfill & \phi \left(g\left(z_{\mathit{ij}}\right),\text{log}\left(\sqrt{{\alpha }_j}\Delta t_j{E}_i\right)\right.\hfill \\ \left.+\nu \left(E_i,\Delta t_j\right),{\sigma }^2\left(E_i,\Delta t_j\right)\right)\left|\frac{\mathrm{d}g}{\mathrm{d}z}\right|,\hfill \end{array}$$

(30)

with φ(x,μ,σ) the Gaussian PDF.

Remark that the weighting function (29) depends on the parameters Δ t _j , α _j , β _ij , ${\sigma }_{\mathrm{o},j}^2$ and the CRF γ(x). We will explain in Section “Dealing with unknown camera model parameters” how to deal with the scenario in which some of these parameters are also unknown.

Approximate direct formula

The optimal weighting function found in the previous section, depends on the functions σ²(E _i ,Δ t _j ) and ν²(E _i ,Δ t _j ) and its practical computation requires a (numerical) integration over the image irradiance variable E _i . We therefore investigated if it is possible to find a good approximation to (29), that is somewhat easier to compute.

First, we note that, because the integration range is larger than the region $y_{\text{min}}^{\prime }\le \Delta t_j{E}_i\sqrt{{\alpha }_j}\le y_{\text{max}}^{\prime }$, the approximations (1) and (21) can not be used to simplify (29). However, within the range $y_{\text{min}}^{\prime }\le \Delta t_j{E}_i\sqrt{{\alpha }_j}\le y_{\text{max}}^{\prime }$, we have that ν(E _i ,Δ t _j ) ≈ 0, such that also A ≈ 1. Let us denote $E_{\text{min}}(i,j)=y_{\text{min}}^{\prime }(i,j)/\left(\Delta t_j\sqrt{{\alpha }_j}\right)$ and $E_{max}(i,j)=y_{\text{max}}^{\prime }(i,j)/\left(\Delta t_j\sqrt{{\alpha }_j}\right)$ (assuming E_max > E_min), then weighting function (29) can be simplified to:

$$\begin{array}{ll}{w}_{\mathit{ij}}^{\left(1\right)}& \approx \underset{0}{\overset{E_{\text{min}}(i,j)}{\int }}\frac{{\sigma }^2\left(E_i,\Delta t_j\right)}{{\sigma }^2\left(E_i,\Delta t_j\right)+P{\nu }^2\left(E_i,\Delta t_j\right)}{f}_{E|z}\left(E_i\left|z_{\mathit{ij}}\right.\right)\mathrm{d}{E}_i\\ +\underset{E_{max}(i,j)}{\overset{+\infty }{\int }}\frac{{\sigma }^2\left(E_i,\Delta t_j\right)}{{\sigma }^2\left(E_i,\Delta t_j\right)+P{\nu }^2\left(E_i,\Delta t_j\right)}{f}_{E|z}\left(E_i\left|z_{\mathit{ij}}\right.\right)\mathrm{d}{E}_i\\ +\underset{E_{\text{min}}(i,j)}{\overset{E_{max}(i,j)}{\int }}{f}_{E|z}\left(E_i\left|z_{\mathit{ij}}\right.\right)\mathrm{d}{E}_i\end{array}$$

(31)

where the approximation error can be controlled by k. We can now define a new weighting function $w_{\mathit{ij}}^{\left(2\right)}<w_{\mathit{ij}}^{\left(1\right)}/A$ that does not contain any terms in σ² (E _i ,Δ t _j ) and ν²(E _i ,Δ t _j ) and is hence easier to compute. This leads to our probabilistic formula for the weighting function:

$$\begin{array}{ll}{w}_{\mathit{ij}}^{\left(2\right)}& =\underset{E_{\text{min}}(i,j)}{\overset{E_{max}(i,j)}{\int }}{f}_{E|z}\left(E_i\left|z_{\mathit{ij}}\right.\right)\mathrm{d}{E}_i\\ =\text{Pr}\left[\left.E_{\text{min}}(i,j)<E_i<E_{max}(i,j)\right|z_{\mathit{ij}}\right]\end{array}$$

(32)

which is the probability that the image irradiance is within the bounds [E_min(i,j),E_max(i,j)], given the the observed pixel intensity z _ij . Generally, (32) has a higher cost in terms of MSE (because we disregard two terms of (32)), however the squared bias term of (24) will be approximately zero. If E _i  ∈ [E_min(i,j),E_max(i,j)] then the bias ν(E _i ,Δ t _j ) ≈ 0, so weighting function (32) can be interpreted as “the probability that the inversely compensated pixel intensity g (z _ij ) is approximately bias-free”. It is clear that HDR reconstruction using (32) will also be approximately bias-free: the bias error $\mathrm{E}\left[\left|\widehat{\text{log}{E}_i}-\text{log}{E}_i\right|\right]\approx 0$.

In Figure 13, an illustration of the weighting function is depicted, for two CRFs and for different choices of the parameters. It can be noted that increasing the offset noise level (resulting in biases with larger magnitudes |ν(E _i ,Δ t _j )|) inherently decreases the average weight. The same situation occurs for the large intensities (e.g., z _ij  > 128) when decreasing the exposure time. According to the formulas, the asymmetry can be attributed to the Poisson characteristics of the photon noise.

Figure 13

Illustration of the weighting function w(z) computed using ( 32) for two different CRFs. (a), (c) Influence of the offset noise level ${\sigma }_{\text{o},j}^2$. (b), (d), influence of the exposure time Δt.

In Appendix 4, a direct formula for (32) was derived:

$$\begin{array}{ll}1w_{\mathit{ij}}^{\left(3\right)}& =\frac{\Phi \left(y_{\text{min}}^{\prime }(i,j)\right)-\Phi \left(y_{\text{max}}^{\prime }(i,j)\right)}{\Phi \left(0\right)-1}\text{with}\Phi \left(x\right)\\ =\text{erf}\left(\frac{x-{\gamma }^{-1}\left(z_{\mathit{ij}}\right)+\lambda {\alpha }_j^{-1/2}\Delta t_j^{-1}s\left({\gamma }^{-1}\left(z_{\mathit{ij}}\right)\right)}{\sqrt{s\left({\gamma }^{-1}\left(z_{\mathit{ij}}\right)\right)}}\right).\end{array}$$

(33)

The merit of this approximation is that it is somewhat easier to implement and compute in practice. In case we can ignore the quadratic fixed pattern noise (see Section “Camera noise modeling”), Equation (33) only depends on the variables z and j. This allows offline computation and the use of a lookup table.

In Figure 14, our three different weighting functions (respectively Equations (28), (32) and (33) are compared to each other. It can be noted that, generally Equation (28) assigns the largest weight of the three weighting functions. This is because of the approximate relationship $w_{\mathit{ij}}^{\left(2\right)}<w_{\mathit{ij}}^{\left(1\right)}/A$. Weighting functions (32) and (33) are more conservative in assigning large weights to pixel intensities.

Figure 14

Comparison of three different weighting functions. We used the Fuji F400 Green CRF from [37] with Δt = 1s., α = 1 and k = 3. (Optimal) solution to (27), (Probabilistic rule) equation (32), (Approximated direct formula) equation (33).

Dealing with unknown camera model parameters

In the previous sections, we explained how to determine a parametric weighting function that mediates a trade-off between the SNR of the reconstructed HDR image on the one hand and the squared bias error on the other hand. The function depends on the parameters Δ t _j , α _j , β _ij , ${\sigma }_{\mathrm{o},j}^2$ and the CRF γ(x), hence beforehand, we assumed that the CRF was known.

In practice, the exposure time can always be assumed to be available.^h The CRF, however, is not publicly disclosed by camera manufacturers, but can be obtained through calibration procedures (e.g., using a color chart), or estimated from a set of LDR images. We will explain this in more detail in Section “Estimation of the CRF”.

If the CRF is available, the camera noise parameters ${\alpha }_j,{\beta }_{\mathit{ij}},{\sigma }_{\mathrm{o},j}^2$ can be estimated directly using a (robust) linear regression on the the inversely compensated pixel intensities γ⁻¹(z _ij ), relying on (17).

One of the main difficulties in estimating the CRF in a noise-robust way is that it is required that the weighting function is known. As we explained, to determine the weighting function, we need the CRF, leading to a chicken-and-egg problem. To resolve this issue, we will now make use of our probabilistic formulation of the weighting function. In Figure 15, it can be seen that range $[y_{\text{min}}^{\prime },y_{\text{max}}^{\prime }]$ becomes smaller (i.e. $y_{\text{min}}^{\prime }$ increases, $y_{\text{max}}^{\prime }$ decreases) when (1) the offset noise variance ${\sigma }_{\mathrm{o},j}^2$ increases, (2) the gain factor α decreases or (3) the gain fixed pattern noise factor β _ij increases. It can be shown that these observations also hold for $E_{\text{min}}(i,j)=y_{\text{min}}^{\prime }(i,j)/\left(\Delta t_j\sqrt{{\alpha }_j}\right)$ and $E_{max}(i,j)=y_{\text{max}}^{\prime }(i,j)/\left(\Delta t_j\sqrt{{\alpha }_j}\right)$.ⁱ Since weighting function (32) is computed as the conditional probability that E_min(i,j) < E _i  < E_max(i,j), and because probabilities are always positive, the overall effect is that the weights $w_{\mathit{ij}}^{\left(2\right)}$ decrease under the aforementioned conditions. This behavior of the weighting function can also be noted in Figure 13a,c: curves corresponding to increasing ${\sigma }_{\mathrm{o},j}^2$ are positioned under each other.

Figure 15

Bounds of the dynamic range $[y_{\text{min}}^{\prime },y_{\text{max}}^{\prime }]$ for which ( 23) and ( 21) are accurate (up to an error $\propto \text{erf}\left(-k/\sqrt{2}\right)+1$)..

The main idea of our approach is: when little is known about the noise model parameters ${\alpha }_j,{\beta }_{\mathit{ij}},{\sigma }_{\mathrm{o},j}^2$, we can always select reasonable upper bounds (or in case of α _j , an upper bound for the reciprocal ${\alpha }_j^{-1}$) for the values of these parameters. The obtained weighting function will then allow some uncertainty on the noise model parameters. The only restriction is that the upper bounds have to be chosen such that E_max(i,j) > E_min(i,j), otherwise all weights become zero (the input SNR would likely be too low to allow proper HDR reconstruction). By using such upper bounds, the weighting function will be more conservative in assigning weights, but still keeping the bias error (26) small in magnitude. In this sense, the certainty function establishes a novel meaning to this concept by relating to the uncertainty associated with the noise model parameters.

Practically speaking, we can fix an upper bound for one parameter, then compute an upper bound for the other parameter values, using the relationship $y_{\text{max}}^{\prime }(i,j)-y_{\text{min}}^{\prime }(i,j)=\Delta$ with Δ > 0 a positive number. In Figure 16, we computed the weighting functions according to this parameter uncertainty approach, when varying the lower bound for α for Δ = 10. The gain fixed pattern term β _ij was chosen to be 0 here, and an identity CRF γ(x) = x was assumed. A simple heuristic using the breaking point from Figure 9, then allows us to determine whether the parameter choice corresponds to the offset noise limited or photon noise limited case. Interestingly, we find that for the offset noise limited case, the weighting function is symmetrical, close to the weighting function of Debevec and Malik and almost equal to the Gaussian weighting function used in [38] for estimating CRFs. Unsurprisingly, the Gaussian weighting function arises as a limit for (33) when Δ is infinitesimally small. On the other hand, in the photon limited case, the weighting function tends to be asymmetrical with a decentralized maximum.

Figure 16

Weighting functions to be used in case camera noise parameter values are not available (only reasonable estimates on their upper bound). The weighting functions are computed using (33), for an identity CRF γ(x) = x.

Estimation of the CRF

In this section, we revisit another common problem in HDR reconstruction: the estimation of the CRF. The CRF is often not known in advance, because this requires exact knowledge of the processing steps of the digital cameras and their parameter values, information that is only at hand of camera manufacturers. Therefore it is useful to estimate the CRF in a camera-independent way. In literature, several techniques have been proposed to estimate the CRF. Mann and Picard propose a parametric regression method based on the comparagram (which is a joint histogram of z _ij versus $z_{i{j}^{\prime }}$ for j ≠ j^′) [27, 30]. Mitsunaga and Nayar [29] perform an iterative polynomial regression, where the (assumed to be unknown) exposure time ratio $\Delta t_j/\Delta t_{j^{\prime }}$ is refined in each iteration. Related iterative methods have been proposed by Tsin et al. [33]. Lin et al. [39] use a completely different technique that estimates the CRF based on RGB distributions of pixels at color edges, requiring only one exposure (P = 1). Debevec and Malik use a non-parametric method to estimate the inverse CRF g(z) [28], leaving a lot of degrees of freedom to this function. In [31], this method was improved to incorporate all available pixel intensities in the estimation (leading to more accurate estimation), while reducing computational complexity.

The problems with the existing techniques are that they are either not very robust to high levels of noise (for example, the method of Debevec and Malik does not enforce monotonicity in terms of a direct constraint to the CRF estimation problem, often leading to CRF estimates that are oscillating in presence of noise) or not very well adapted to the signal-dependent noise characteristics of the individual LDR images (e.g., due to an assumption of stationary noise).

Let us analyze the CRF estimation problem by relying on the camera model developed in the previous sections. Consider two distinct LDR images z _ij and $z_{i{j}^{\prime }}$. When z _ij and $z_{i{j}^{\prime }}$ are in the appropriate exposure range ($y_{\text{min}}^{\prime }(i,j)<{\alpha }_j\Delta t_j{E}_i<y_{\text{max}}^{\prime }(i,j)$ and $y_{\text{min}}^{\prime }(i,j^{\prime })<{\alpha }_{j^{\prime }}\Delta t_{j^{\prime }}{E}_i<y_{\text{max}}^{\prime }(i,j^{\prime })$), we have $\mathrm{E}\left[z_{\mathit{ij}}|E_i\right]\approx \gamma \left(\sqrt{{\alpha }_j}\Delta t_j{E}_i\right)$ and $\mathrm{E}\left[z_{i{j}^{\prime }}|E_i\right]\approx \gamma \left(\sqrt{{\alpha }_{j^{\prime }}}\Delta t_{j^{\prime }}{E}_i\right)$ with respective variances (see Appendix 1):

$$\begin{array}{l}\text{Var}\left[z_{\mathit{ij}}|E_i\right]\\ \approx \left({\sigma }_{\mathrm{o},j}^2+{\alpha }_j\Delta t_j{E}_i+{\beta }_{\mathit{ij}}{\left(\Delta t_j{E}_i\right)}^2\right){\left.{\left(\frac{\mathrm{\partial \gamma }}{\mathrm{\partial x}}\right)}^2\right|}_{x=\Delta t_j{E}_i}\\ \text{Var}\left[z_{i{j}^{\prime }}|E_i\right]\\ \approx \left({\sigma }_{\mathrm{o},j}^2+{\alpha }_{j^{\prime }}\Delta t_{j^{\prime }}{E}_i+{\beta }_{\mathit{ij}}{\left(\Delta t_{j^{\prime }}{E}_i\right)}^2\right){\left.{\left(\frac{\mathrm{\partial \gamma }}{\mathrm{\partial x}}\right)}^2\right|}_{x=\Delta t_{j^{\prime }}{E}_i}\end{array}$$

(34)

When considering the estimation of the CRF as a regression problem (as in [30]), optimal estimation is quite difficult due to the dependency of the variance on the (unknown) image irradiance E _i . If the conditional variances Var[z _ij |E _i ] and $\text{Var}\left[z_{i{j}^{\prime }}|E_i\right]$ are constant (e.g. $\text{Var}\left[z_{\mathit{ij}}|E_i\right]\approx {\sigma }_{\mathrm{o},j}^2$), the method of total least squares (TLS) [40] is well suited to estimate the nonlinear CRF: the TLS method performs a regression in which the orthogonal distance error of the fitted function to the observations is minimized. The TLS method is well suited for regression problems in which both the x-variables and y-variables have a known and constant variance. Unfortunately, this is not the case here. Furthermore, the variances (34) depend on the derivative of the CRF, which is to be estimated!

As an alternative solution, we can rely on the statistical conditional independence of z _ij and $z_{i{j}^{\prime }}$ (given E _i ). According to (4), the difference $g\left(z_{\mathit{ij}}\right)-g\left(z_{i{j}^{\prime }}\right)$ has the following PDF:

$$\begin{array}{ccc}g\left(z_{\mathit{ij}}\right)-g\left(z_{i{j}^{\prime }}\right)\hfill & \sim \hfill & \mathrm{N}\left(\text{log}\frac{{\alpha }_j\Delta t_j}{{\alpha }_{j^{\prime }}\Delta t_{j^{\prime }}}+\nu \left(E_i,\Delta t_j\right)\right.\hfill \\ -\nu \left(E_i,\Delta t_{j^{\prime }}\right),{\sigma }^2\left(E_i,\Delta t_j\right)\hfill \\ \left.+{\sigma }^2\left(E_i,\Delta t_{j^{\prime }}\right)\right).\hfill \end{array}$$

In the exposure range $\left\{{\alpha }_j\Delta t_j{E}_i,{\alpha }_j\Delta t_{j^{\prime }}{E}_i\right\}\subset \left[y_{\text{min}}^{\prime }(i,j),y_{\text{max}}^{\prime }(i,j^{\prime })\right]$, the bias terms can safely be ignored (provided that E _i is sufficiently large):

$$\begin{array}{ccc}g\left(z_{\mathit{ij}}\right)-g\left(z_{i{j}^{\prime }}\right)\hfill & \sim \hfill & \mathrm{N}\left(\text{log}\frac{\Delta t_j}{\Delta t_{j^{\prime }}}+\text{log}\frac{{\alpha }_j}{{\alpha }_{j^{\prime }}},{\sigma }^2\left(E_i,\Delta t_j\right)\right.\hfill \\ \left.+{\sigma }^2\left(E_i,\Delta t_{j^{\prime }}\right)\right).\hfill \end{array}$$

(35)

Maximum likelihood estimation of g(z _ij ) now amounts to the linear regression:

$$\widehat{g\left(z\right)}=\text{min}\sum _{j,j^{\prime }=1}^P\sum _{i=1}^N{w}_{\mathit{ij}}{w}_{i{j}^{\prime }}\frac{{\left|g\left(z_{\mathit{ij}}\right)-g\left(z_{i{j}^{\prime }}\right)-\text{log}\frac{\Delta t_j}{\Delta t_{j^{\prime }}}-\text{log}\frac{{\alpha }_j}{{\alpha }_{j^{\prime }}}\right|}^2}{{\sigma }^2\left(E_i,\Delta t_j\right)+{\sigma }^2\left(E_i,\Delta t_{j^{\prime }}\right)},$$

(36)

where a weighting function was introduced in analogy to Equation (6). Recall that the LDR pixel intensities are discrete ($z_{\mathit{ij}}\in 0,\dots ,2^B-1$ and $z_{i{j}^{\prime }}\in 0,\dots ,2^B-1$). Therefore, (36) can be solved by treating g(z _ij ) and $g\left(z_{i{j}^{\prime }}\right)$ as unknown variables. Solving (36) then gives a linear system of N × P equations with 2^B unknowns ($g\left(0\right),\dots ,g(2^B-1)$).

In case either offset noise or photon noise is dominant, the sum ${\sigma }^2\left(E_i,\Delta t_j\right)+{\sigma }^2\left(E_i,\Delta t_{j^{\prime }}\right)$ can be approximated by (see (22)):

$$\begin{array}{cc}{\sigma }^2\hfill & \left(E_i,\Delta t_j\right)+{\sigma }^2\left(E_i,\Delta t_{j^{\prime }}\right)\hfill \\ \approx \left\{\begin{array}{ll}\frac{1}{E_i}\frac{\Delta t_j+\Delta t_{j^{\prime }}}{\Delta t_j\Delta t_{j^{\prime }}}& \text{(photon noise dominant)}\\ \frac{{\sigma }_{\mathrm{o},j}^2}{E_i^2}\frac{\Delta t_j^2{\alpha }_j+\Delta t_{j^{\prime }}^2{\alpha }_{j^{\prime }}}{{\alpha }_j{\alpha }_{j^{\prime }}\Delta t_j\Delta t_{j^{\prime }}}& \text{(offset noise dominant)}\end{array}.\right.\hfill \end{array}$$

As we mentioned earlier, because of estimation in presence of noise, we need to explicitly enforce that g(z) is monotonic. Therefore, we include the conditions g(z) > g(z − 1),$z=1,\dots ,2^B-1$ as extra constraints to the minimization problem from (36). Reorganizing the terms in (36) then gives (in case photon noise is dominant in all of the LDR images):

$$\begin{array}{l}\text{min}\sum _{i=1}^N\frac{1}{E_i}\sum _{j,j^{\prime }=1}^P{w}_{\mathit{ij}}{w}_{i{j}^{\prime }}\frac{{\left|g\left(z_{\mathit{ij}}\right)-g\left(z_{i{j}^{\prime }}\right)-\text{log}\frac{\Delta t_j}{\Delta t_{j^{\prime }}}-\text{log}\frac{{\alpha }_j}{{\alpha }_{j^{\prime }}}\right|}^2}{\left(\Delta t_j+\Delta t_{j^{\prime }}\right)/\Delta t_j\Delta t_{j^{\prime }}}\\ +\lambda \sum _{z=0}^{2^B-1}{w}_{\text{smooth}}^2\left(z\right)\left|\frac{{\mathrm{d}}^{2}g}{\mathrm{d}{z}^2}\right|\\ \text{s.t.}g\left(z\right)\ge g(z-1)+є,z=1,\dots ,2^B-1.\end{array}$$

(37)

with λ a regularization parameter to enforce smoothness of g(z) (see [28, 31]), and with є a small positive number. In (37), the function w_smooth(z) is the weighting function, averaged over the set of images $j=1,\dots ,P$. To compute d²g/dz², the numerical second derivative is used. Optimization problem (37) can be efficiently solved using standard quadratic program (QP) solvers [41]. We note that in (37), the (unknown) image irradiance appears as position-dependent weights. A straightforward solution is then to iteratively update the CRF and the log-irradiance estimate logE _i , but to keep the processing technique simple, we propose here to simply drop the position-dependent weights 1/E _i . To conclude, the proposed CRF estimation technique differs from the technique in [31] in the following aspects: (1) an image-dependent weight factor $\left(\Delta t_j+\Delta t_{j^{\prime }}\right)/\Delta t_j\Delta t_{j^{\prime }}$ that penalizes LDR images with low exposure times (because the SNR is generally lower in these images), (2) the use of a weight function w _ij that is adapted to the underlying camera noise model and (3) the monotonicity constraint for g(z). Consequently, the CRF estimation technique will be considerably more robust against noise, with only limited increase in algorithmic complexity (the use of a QP solver compared to a sparse system solver).

Results and discussion

Comparison of different weighting functions

In this section, we compare the different weighting functions proposed in literature to our probabilistic function from Equation (32). We will only consider the influence of the choice of the weighting function on the HDR reconstruction (i.e., we do not include the CRF estimation in this experiment and all techniques use the same HDR reconstruction formula from (7). As in Equation (7), the effective weights are calculated using w _ij σ⁻²(E _i ,Δ t _j ). We computed the performance of the different weighting functions, using analytical formulas (Equations (24)–(26), for different camera gains $\sqrt{\alpha }\in \left\{0.25,0.5,1,2,4\right\}$ and different offset noise levels σ _o  ∈ {0.001,0.5,1,2,4,8,16}. We assumed four LDR images with exposure times 0.1s, 0.25s, 1s, 2s and an identity CRF γ(z) = z. The expected MSE $E\left[{\left(\widehat{\text{log}E}-\text{log}E\right)}^2\right]$ is inherently averaged over a logarithmically spaced range over the image irradiance E, covering the whole input dynamic range. We remark that an upper bound for the theoretical maximal gain compared to the triangular weighting function of Debevec and Malik is given by:

$$10{\text{log}}_{10}\frac{{\int }_0^1{z}^2\mathrm{d}z}{{\left({\int }_0^{1}z\mathrm{d}z\right)}^2}=10{\text{log}}_{10}\frac{4}{3}=1.25\text{dB}.$$

This means that, for any estimation task, no weighting function will give a SNR improvement of more than 1.25 dB, compared to the triangular weighting function. The above formula ignores the clipping of the dynamic range and the presence of signal-dependent noise, hence in practice, the actual improvement will be much smaller than 1.25 dB. The actual values are plotted in Figure 17, it can be seen that the probabilistic weighting function reaches a maximal improvement of approximately 1 dB (for larger values of ${\sigma }_{\mathrm{o},j}^2$) and outperforms the other weighting functions in most of the cases.

Figure 17

Comparison of different weighting functions in the Δ SNR sense (difference in SNR compared to the triangular function of Debevec and Malik) for γ ( z )= z . The SNR was computed using (24) and in (a) averaged as function of the gain α, in (b) averaged as function of the offset noise standard deviation σ _o .

Ground truth data with simulated noise

To have ground truth data, we generated a computer rendering of a HDR scene (see Figure 18a). The scene consists of a room with a textured walls with four paintings (all textures and paintings have a bit depth of 8). In the middle of the room, a bright point light source is positioned, creating a dynamic range of 11 stops (i.e. the ratio of the brightest and smallest possible pixel intensity is about 2¹¹). Sensor signals were simulated and artificial noise was thereby generated, according to the procedure explained in Section “Camera noise modeling”. An example of such exposes LDR images is given in Figure 18b–e. The light source causes parts of the wall, floor and ceiling to be overexposed in some of the LDR images, while the wall on the right is underexposed in other images. Results are again generated for different camera gains $\sqrt{\alpha }\in \left\{0.25,0.5,1,2,4\right\}$ and different offset noise levels σ _o  ∈ {0.001,0.5,1,2,4,8,16}, for a CRF modeling gamma correction γ(z) = z^0.4 under (1) the assumption that the CRF is known in advance, and (2) by estimating the CRF from the images. For the method of Debevec and Malik [28], the CRF was estimated from 1,000 samples, randomly sampled in the image. The regularization parameter is fixed to λ = 1 for all methods. The results are given in Figure 19. It can be seen that, in case the CRF is known, an improvement of about 1dB is obtained by adapting to the camera noise model. When also CRF estimation is included in the algorithm, the SNR further improves by 1–1.5 dB. Also notice that the weighting function from Section “Dealing with unknown camera model parameters” that includes uncertainty with respect to the noise model parameters gives slightly worse results than the proposed probabilistic weighting function.

Figure 18

Ground truth data used for the experiments. (a) HDR image with compression from [26]. (b)–(e) LDR images generated from (a) using different exposure times.

Figure 19

HDR reconstruction results using ground truth data. (a) Average PSNR results as function of gain α, (b) average PSNR results as function of the offset noise standard deviation σ_o.

A visual HDR reconstruction result is presented in Figure 20. In the image reconstructed using the proposed technique, the noise is significantly better suppressed. Although the maximal SNR improvement of the HDR image is bounded when only using position-dependent averaging (see Section “Denoising performance of the HDR reconstruction formula”), further improvement can be obtained by including spatial regularization in the reconstruction process.

Figure 20

Visual comparison of HDR reconstruction using different weighting functions ( α  = 1/64, β  = 0, ${\sigma }_{\text{o},j}^2$  = 64, j  = 1,…,4) and different CRF estimation methods.

HDR reconstruction of RAW sensor data

Next, we demonstrate our method for real digital camera images. In particular, we use the “desk still life” set of 17 RAW LDR images acquired by Hasinoff [24] using a Canon EOS 1D Mark III camera (10 mega-pixel images, with an equal ISO setting of 100).^j The 17 images are used to obtain an estimate for the ground-truth data, which is useful for objective evaluation. The spatial resolution of the images and the exposure time sampling is sufficiently high, such that different parameters of the reconstruction methods have a limited impact on the reconstruction result. Here, we use Debevec’s method[28] for creating the HDR images.

To compare different methods, we select 3 LDR images out of the 17 images, with exposure times 1/800, 1/200, 1/25 s. These exposure times are chosen to cover a large portion of the dynamic range of the scene, such that some of the LDR images exhibit high levels of noise (especially in the dark regions, see Figure 21). Although this choice is somewhat arbitrary, similar results can be obtained using slightly deviating exposure time values.

Figure 21

Desk still life: LDR images (after camera reconstruction).

First, to estimate the camera noise model parameters (see Section “Camera noise modeling”), we analyze signal-dependent noise in a similar way as in [42], i.e., by computing the local noise standard deviation as a function of the local intensity. We use a 7×7 local window to estimate the local mean and standard deviation, where we also exclude patches in which the signal is too strong (to find these patches, we compare the maximum local magnitude of the Laplacian filtered image to a fixed threshold, and we reject patches if the maximum is too large). We repeat this for the three color channels, and we directly fit the parabolic noise variance model (17) to the obtained point clouds using least squares fitting. The noise fitting results are shown in Figure 22 for the LDR image of exposure time 1/25 s. It can be seen that there is a good correspondence between the predicted noise variance and the estimated noise variance. Alternatively, affine fitting could also be used here (assuming the absence of gain FPN, β _ij  = 0 in (17), however, we found that in most LDR images a quadratic model better describes the measured noise variances.

Figure 22

Signal-dependent noise analysis of the raw sensor data: local noise variance as a function of the local intensity.

A HDR image is then constructed from the three LDR images, using the proposed approach or, alternatively, existing methods like [28, 31]. Finally, some common post-processing operations, such as white balancing, exposure correction, color correction and HDR tone mapping need to be applied to the constructed HDR images. For this post-processing stage, we employ the software program “Raw Therapee V4.0”.^k For our purposes, this program allows us to create a reconstruction profile with fixed (non automatic) parameter settings, which can then subsequently be applied to the images reconstructed using different methods. This is very useful to ease visual comparison. In Figure 11, the final results are shown, together with the HDR reconstructed from densely sampled LDR images. The signal-to-noise ratio is estimated from the log-irradiance values obtained before post-processing, by using this last image as a reference. It can be seen that by using optimized weighting functions we can obtain a significant increase in visual quality, compared to other methods which are using more “general-purpose” weighting functions.

HDR reconstruction from JPEG compressed LDR digital photographs

As a last experiment, we test our method on a set of 15 LDR images, with increasing exposure times and a fixed ISO setting of 100 and aperture of f3.5, captured using a Canon Powershot S5 camera, with post-processing and JPEG compression performed by the camera. As in the previous experiment, we use all 15 images to obtain an estimate for the ground-truth data, and we select a subset of four images (with exposure times 1/100, 1/10, 1/2 and 2 s, see Figure 23), for comparing the different HDR reconstruction methods. Because camera post-processing is applied, all methods now require CRF estimation. One of the issues arising for our method, is that the noise model parameter estimation (as used in the previous experiment) is not directly applicable. Note in this respect that applying the inverse CRF is not sufficient to obtain estimates of the RAW sensor data, because the component-wise CRF does not take color space operations (such as color corrections, white balancing,..) into account. Instead, we compute the weighting functions as explained in Section “Dealing with unknown camera model parameters” by defining upper bounds for ${\alpha }_j^{-1}$ and ${\sigma }_{\mathrm{o},j}^2$. Here we use the values $max\left({\alpha }_j^{-1}\right)=0.005$, $max\left({\sigma }_{\mathrm{o},j}^2\right)=0.02$ and max(β _ij ) = 0, which are experimentally determined. In our method, the weighting functions used for CRF estimation are based on an identity CRF (recall the chicken-and-egg problem mentioned in Section “Dealing with unknown camera model parameters”). Next, to reconstruct the HDR image, we re-compute the weighting functions using the estimated CRF using results of Appendices 1 and 2. Finally, tone mapping with fixed parameters is applied using the program “Raw Therapee V4.0”. Some visual results are depicted in Figure 12 and also estimated SNR values (obtained by comparing to the “estimated” ground-truth data) are reported. It can be seen that only small differences exist in the reconstruction results and that the proposed method performs within the same range as the other methods (in particular, the difference can be spotted in the dark regions of the color chart in Figure 12: while Debevec’s method still leaves a lot of noise in this region, the method of [31] and the proposed method do a better job in suppressing the noise). The reason that the proposed method does not outperform the other methods in the same way as in Section “HDR reconstruction from JPEG compressed LDR digital photographs” is most likely a combined effect of several factors: (1) an estimate of the CRF is required to determine the weighting functions for (32), (2) CRF estimation errors also influence the weighting functions and (3) nonlinear color space operations are not taken into account. Although further improvements are possible, e.g., by iteratively refining the weighting functions and CRFs, this experiment already shows that the proposed approach is robust enough to work with JPEG compressed images where standard post-processing operations have been applied.

Figure 23

TELIN Lab: LDR images (after camera reconstruction).

Conclusion

In this article, we presented a realistic digital camera noise model that incorporates several noise sources (such as photon noise, electronic noise, fixed pattern noise), as well as several parts of the camera processing pipeline (such as the amplifier, clipping,..). For this noise model, we derived both exact and approximate formulas for the bias function and the noise variance function, taking clipping of the dynamic range into account. Next, we investigated novel HDR image reconstruction weighting schemes relying on the realistic noise model. We paid special attention to different factors that determine the signal-to-noise ratio of the HDR image, and we used the obtained insight to derive weighting functions that are optimized for the camera noise model. This led to our probabilistic weighting function, that is defined as the probability that the considered pixel intensity is (approximately) unbiased. Because due to the clipping there is a strong coupling between the noise variance and bias functions, this weighting function offers a trade-off between maximizing the SNR of the HDR image and bias errors, its performance is close to optimal in the MSE sense. The probabilistic reasoning was then extended to derive a weighting function for the case in which (some of) the noise model parameters are unknown. Experimental results confirmed the expected improvements in reconstruction quality, especially for reconstruction from RAW sensor data. Although HDR image reconstruction is only one example of application of sophisticated noise modeling techniques, there is a wide range of other application areas to explore in which similar image quality improvements can be obtained when adapting to the noise characteristics from the imaging devices.

Appendix

Appendix 1: approximation of the statistical moments of a non-linear function of signal and noise

In this section, we develop a technique that gives simple expressions that relate the noise statistical moments to the unknown parameters. First, we start from a generative model:

$$u=\zeta (E,\nu )$$

(38)

where E is the image irradiance and ν is Gaussian noise with zero mean and unit variance N(0,1). The signal-and-noise mixing function ζ(E,ν) has the property that it is infinitely differentiable near the working point ν = 0, for all E. In our application, the mixing function is given by:

$$\zeta (E,\nu )=\gamma \left(\sqrt{\alpha }\mathrm{\Delta tE}+\sqrt{s\left(\mathrm{\Delta tE}\right)}\nu \right)$$

(39)

with $s\left(x\right)={\sigma }_{\mathrm{o},j}^2+\mathrm{\alpha x}+\beta x^2$. The conditional statistical moments of u can be computed through the McLaurin series expansion of ζ(E,ν). For the first order moment we have:

$$\begin{array}{ll}\mathrm{E}\left[u|E\right]& \approx \zeta (E,0)+\sum _{n=1}^{+\infty }{\left.\frac{{\partial }^n\zeta }{\partial {\nu }^n}\right|}_{\nu =0}\frac{{\nu }^n}{n!}\\ =\zeta (E,0)+\sum _{n=1}^{+\infty }{\left.\frac{{\partial }^{2n}\zeta }{\partial {\nu }^{2n}}\right|}_{w=0}\frac{2^{-n}}{n!}\end{array}$$

(40)

where we used the statistical moments of the Gaussian distribution. A similar expression can be given for the second order moment:

$$\begin{array}{l}\mathrm{E}\left[u^2|E\right]\approx {\zeta }^2(E,0)+\sum _{n=1}^{+\infty }{\left.\frac{{\partial }^n{\zeta }^2}{\partial {\nu }^n}\right|}_{w=0}\frac{\mathrm{E}\left[{\nu }^n\right]}{n!}\\ ={\zeta }^2(E,0)+\sum _{n=1}^{+\infty }{\left.\frac{{\partial }^{2n}{\zeta }^2}{\partial {\nu }^{2n}}\right|}_{w=0}\frac{2^{-n}}{n!}\\ ={\zeta }^2(E,0)+2\sum _{n=1}^{+\infty }\frac{2^{-n}}{n!}\\ \left(\zeta (E,0)\left({\left.\frac{{\partial }^{2n}\zeta }{\partial {\nu }^{2n}}\right|}_{\nu =0}\right)+{\left.\frac{{\partial }^{2n-1}\zeta }{\partial {\nu }^{2n-1}}\right|}_{\nu =0}\left({\left.\frac{\mathrm{\partial \zeta }}{\mathrm{\partial \nu }}\right|}_{\nu =0}\right)\right)\end{array}$$

Note that Var[u|E] = E[u²|E] − (E[u|E])². The reason for such an expansion is the fact that in practical circumstances, ζ(E,ν) is a relatively smooth function, i.e., the derivatives ∂ⁿζ/∂ νⁿ evaluated at ν = 0 decay exponentially as a function of n. For this reason, only one term of the series expansion is often sufficient in our application (provided that ζ(E,ν) is differentiable near w = 0). When only considering one term of the McLaurin series approximation, we find:

$$\begin{array}{ll}\mathrm{E}\left[u|E\right]& \approx \zeta (E,0)+\frac{1}{2}{\left.\frac{{\partial }^2\zeta }{\partial {\nu }^2}\right|}_{\nu =0}\\ \text{Var}\left[u|E\right]& \approx {\left.\frac{\mathrm{\partial \zeta }}{\mathrm{\partial \nu }}\right|}_{\nu =0}^2-\frac{1}{4}{\left.\frac{{\partial }^2\zeta }{\partial {\nu }^2}\right|}_{\nu =0}^2\approx {\left.\frac{\mathrm{\partial \zeta }}{\mathrm{\partial \nu }}\right|}_{\nu =0}^2\end{array}$$

where we used the fact that ${\left.\frac{{\partial }^2\zeta }{\partial {\nu }^2}\right|}_{\nu =0}^2\ll {\left.\frac{\mathrm{\partial \zeta }}{\mathrm{\partial \nu }}\right|}_{\nu =0}^2$. Substitution of the mixing function (39) gives explicit expressions for the conditional statistical moments of u:

$$\begin{array}{ll}\mathrm{E}\left[u|E\right]& \approx \gamma \left(\sqrt{\alpha }\mathrm{\Delta tE}\right)+\frac{s\left(\mathrm{\Delta tE}\right)}{2}{\left.\frac{{\partial }^2\gamma }{\partial x^2}\right|}_{x=\sqrt{\alpha }\mathrm{\Delta tE}}\end{array}$$

(41)

$$\begin{array}{ll}\text{Var}\left[u|E\right]& \approx s\left(\mathrm{\Delta tE}\right){\left({\left.\frac{\mathrm{\partial \gamma }}{\mathrm{\partial x}}\right|}_{x=\sqrt{\alpha }\mathrm{\Delta tE}}\right)}^2\end{array}$$

(42)

where γ(x) = ζ(x,0) is the CRF. For the special case of a logarithmic transform γ(x) = logx, we get:

$$\begin{array}{ll}\mathrm{E}\left[u|E\right]& \approx \text{log}\left(\sqrt{\alpha }\mathrm{\Delta tE}\right)-\frac{1}{2}\frac{s\left(\mathrm{\Delta tE}\right)}{\mathrm{\alpha \Delta }{t}^2{E}^2}\end{array}$$

(43)

$$\begin{array}{ll}\text{Var}\left[u|E\right]& \approx \frac{s\left(\mathrm{\Delta tE}\right)}{\mathrm{\alpha \Delta }{t}^2{E}^2}\end{array}$$

(44)

These convenient formulas form the basis of our noise model in Section “Camera noise modeling”.

Appendix 2: determining the clipping-safe region of the dynamic range

In this section, we will determine a region of the dynamic range where the approximations (23) and (21) hold, with high probability. We remark that these approximations are obtained using the technique presented in Appendix 1 and that this technique fails for working points of the clipping function, where this function is not differentiable. Obviously, this is near the saturation points of the dynamic range. We will analytically determine the effect of the clipping on the statistical moments of logf(y _ij ). First, we define the moment function as:

$$\begin{array}{l}{M}_E\left(n\right)=\mathrm{E}\left[\left.{\left(\text{log}f\left(y_{\mathit{ij}}\right)\right)}^n\right|E\right]\\ =\frac{1}{\sqrt{2\mathrm{\Pi s}\left(\mathrm{\Delta tE}\right)}}\underset{-\infty }{\overset{+\infty }{\int }}{\left(\text{log}f\left(u\right)\right)}^{n}exp\\ \left(-\frac{1}{2}\frac{{\left(\sqrt{\alpha }\mathrm{\Delta tE}-u\right)}^2}{s\left(\mathrm{\Delta tE}\right)}\right)\mathrm{d}\mathrm{u.}\end{array}$$

(45)

The bias and variance functions from (19) can be expressed in terms of M _E (n):

$$\begin{array}{ll}\nu \left(E,\mathrm{\Delta t}\right)& =M_1\left(E\right)-\text{log}f\left(\sqrt{\alpha }\mathrm{\Delta tE}\right)\\ {\sigma }^2\left(E,\mathrm{\Delta t}\right)& =M_2\left(E\right)-{\left(M_E\left(1\right)\right)}^2\end{array}$$

Suppose that the clipping function, in a general form, is defined by:

$$f\left(y\right)=\left\{\begin{array}{ll}{y}_{\text{min}}& y<y_{\text{min}}\\ y& y_{\text{min}}\le y\le y_{\text{max}}\\ y_{\text{max}}& y>y_{\text{max}}\end{array}\right.$$

we can rewrite the moment function (45) as:

$$\begin{array}{ll}{M}_n\left(E\right)& =\frac{1}{\sqrt{2\mathrm{\Pi s}\left(\mathrm{\Delta tE}\right)}}\underset{-\infty }{\overset{+\infty }{\int }}{\left(\text{log}u\right)}^{n}exp\left(-\frac{1}{2}\frac{{(\sqrt{\alpha }\mathrm{\Delta tE}-u)}^2}{s\left(\mathrm{\Delta tE}\right)}\right)\mathrm{d}u\\ -\frac{1}{2}{\left(\text{log}{y}_{\text{min}}\right)}^n\left(1+\text{erf}\left(\frac{y_{\text{min}}-\sqrt{\alpha }\mathrm{\Delta tE}}{\sqrt{2s\left(\mathrm{\Delta tE}\right)}}\right)\right)\\ -\frac{1}{2}{\left(\text{log}{y}_{\text{max}}\right)}^n\left(1+\text{erf}\left(\frac{-y_{\text{max}}+\sqrt{\alpha }\mathrm{\Delta tE}}{\sqrt{2s\left(\mathrm{\Delta tE}\right)}}\right)\right).\end{array}$$

(46)

Because the first term of M _E (n) is not related to the clipping function, we can use the technique from Appendix 1 to approximate this term (this term represents the statistical moments as if a clipping function was not present). Because of the decay of 1 + erf(−x), the second and third terms in (46) vanish in the following region of the dynamic range:

$$y_{\text{min}}+k\sqrt{s\left(\mathrm{\Delta tE}\right)}<\sqrt{\alpha }\mathrm{\Delta tE}<y_{\text{max}}-k\sqrt{s\left(\mathrm{\Delta tE}\right)}$$

(47)

with k a positive constant. Next, we can exploit the fact that $s\left(x\right)={\sigma }_{\mathrm{o},j}^2+\mathrm{\alpha x}+\beta x^2$. To proceed, we use the linear Taylor approximations:

$$\begin{array}{ll}\sqrt{s\left(\mathrm{\Delta tE}\right)}& \approx \left\{\begin{array}{l}\sqrt{s\left(y_{\text{min}}\right)}+\frac{1}{2}\frac{\left(1+2\beta y_{\text{min}}/\alpha \right)\left(\sqrt{\alpha }\mathrm{\Delta tE}-y_{\text{min}}\right)}{\sqrt{s\left(y_{\text{min}}\right)}}\\ \text{near}\sqrt{\alpha }\mathrm{\Delta tE}=y_{\text{min}}\\ \sqrt{s\left(y_{\text{max}}\right)}+\frac{1}{2}\frac{\left(1+2\beta y_{\text{max}}/\alpha \right)\left(\sqrt{\alpha }\mathrm{\Delta tE}-y_{\text{max}}\right)}{\sqrt{s\left(y_{\text{max}}\right)}}\\ \text{near}\sqrt{\alpha }\mathrm{\Delta tE}=y_{\text{max}}\end{array}\right.\end{array}$$

The sought region of the dynamic range can then be written as:

$$\begin{array}{ll}{y}_{\text{min}}^{\prime }& <\sqrt{\alpha }\mathrm{\Delta tE}<y_{\text{max}}^{\prime }\text{with}\\ y_{\text{min}}^{\prime }& =\frac{k\left(\mathrm{\alpha s}\left(y_{\text{min}}\right)-\beta y_{\text{min}}^2\right)+y_{\text{min}}\alpha \left(\sqrt{s\left(y_{\text{min}}\right)}-k/2\right)}{\alpha \sqrt{s\left(y_{\text{min}}\right)}-k\left(\sqrt{\alpha }+2\beta y_{\text{min}}\right)/2}\text{and}\\ y_{\text{max}}^{\prime }& =\frac{-k\left(\mathrm{\alpha s}\left(y_{\text{max}}\right)-\beta y_{\text{max}}^2\right)+y_{\text{max}}\alpha \left(\sqrt{s\left(y_{\text{max}}\right)}+k/2\right)}{\alpha \sqrt{s\left(y_{\text{max}}\right)}+k\left(\sqrt{\alpha }+2\beta y_{\text{max}}\right)/2}\end{array}$$

(48)

An illustration of the bounds as a function of the offset noise level ${\sigma }_{\mathrm{o},j}^2$ and the amplification factor $\sqrt{\alpha }$ is given in Figure 15. It can be seen that even at low SNRs, the region $[y_{\text{min}}^{\prime },y_{\text{max}}^{\prime }]$ still covers a sufficiently large region near the center of the dynamic range $\sqrt{\alpha }\mathrm{\Delta tE}=\left(y_{\text{min}}+y_{\text{max}}\right)/2$.

Appendix 3: optimization problem for determining the weighting function

In this section, we will solve the following optimization problem:

$$\underset{\overrightarrow{w}}{\text{min}}\frac{{\sum }_{j=1}^P{w}_j^2{\alpha }_j^2{\sigma }_j^2}{{\left({\sum }_{j=1}^P{w}_j{\alpha }_j\right)}^2}\text{such that}{∥\overrightarrow{w}∥}_{\infty }=1,$$

(49)

where ∥·∥ _∞ is the maximum norm. To deal with the constraint ${∥\overrightarrow{w}∥}_{\infty }=1$, we introduce a new constraint ${\sum }_{j=1}^P{w}_j{\alpha }_j=C$ and subsequently we fix the constant C by imposing that ${∥\overrightarrow{w}∥}_{\infty }=1$. Using Lagrange multipliers, the resulting problem can be stated as follows:

$$\underset{\overrightarrow{w}}{\text{min}}\sum _{j=1}^P{w}_j^2{\alpha }_j^2{\sigma }_j^2+\lambda \left(\sum _{j=1}^P{w}_j{\alpha }_j-C\right).$$

(50)

Taking the derivatives of the cost function with respect to w _j gives the following equations:

$$w_j{\alpha }_j{\sigma }_j^2=-\lambda /2,\text{with}j=1,\dots ,\mathrm{P.}$$

One of the issues here is that ${\sigma }_j^2$ may become zero, such that the weight w _j can not be determined. In this case, we are free to choose the weight, and for convenience we choose w _j  = 1 (such that the contribution of the corresponding LDR in the HDR reconstruction will be maximal). Imposing the constraints ${\sum }_{j=1}^P{w}_j{\alpha }_j=C$ and ${∥\overrightarrow{w}∥}_{\infty }=1$ then leads to the following solution:

$$w_j=\left\{\begin{array}{ll}1& {\sigma }_j^2=0\\ \frac{{\sigma }_j^{-2}}{{\alpha }_j}/\underset{{\sigma }_{j^{\prime }}^2\ne 0}{max}\frac{{\sigma }_{j^{\prime }}^{-2}}{{\alpha }_{j^{\prime }}^2}& {\sigma }_j^2\ne 0\end{array}\right.$$

This solution is used in Section “Probabilistic formulation of the weighting function” for computing optimal weighting functions.

Appendix 4: derivation of a direct formula for the weighting function

Based on the noise model from Equations (17) and (30), the PDF $f_{{\gamma }^{-1}\left(z\right)|E}\left({\gamma }^{-1}\left(z\right)\left|E_i\right.\right)$ can be determined as follows:

$$\begin{array}{c}{f}_{{\gamma }^{-1}\left(z\right)|E}\left({\gamma }^{-1}\left(z\right)\left|E_i\right.\right)=\frac{1}{\sqrt{2\Pi s^{\prime }\left(E_i\right)}}exp\\ \left(-\frac{1}{2}{\left(\frac{{\gamma }^{-1}\left(z\right)-\sqrt{{\alpha }_j}\Delta t_j{E}_i}{s^{\prime }}\right)}^2\right)\left|\frac{\mathrm{d}{\gamma }^{-1}\left(z\right)}{\mathrm{d}z}\right|\end{array}$$

(51)

with s^′ = Var[γ⁻¹(z)|E _i ]. Using results from Appendix 1, we approximate s^′(E _i ) as follows:

$$s^{\prime }\approx s\left({\gamma }^{-1}\left(z\right)\right)={\sigma }_{\mathrm{o},j}^2+{\alpha }_j{\gamma }^{-1}\left(z\right)+{\beta }_{\mathit{ij}}{\left({\gamma }^{-1}\left(z\right)\right)}^2.$$

(52)

The weighting function (32) can be computed using Bayes’ rule:

$$\begin{array}{ll}{w}_{\mathit{ij}}^{\left(2\right)}& =\frac{{\int }_{E_{\text{min}}(i,j)}^{E_{max}(i,j)}{f}_{{\gamma }^{-1}\left(z\right)|E}\left({\gamma }^{-1}\left(z\right)\left|E_i\right.\right)f_E\left(E_i\right)\mathrm{d}{E}_i}{{\int }_{-\infty }^{+\infty }{f}_{{\gamma }^{-1}\left(z\right)|E}\left({\gamma }^{-1}\left(z\right)\left|E_i\right.\right)f_E\left(E_i\right)\mathrm{d}{E}_i}\end{array}$$

(53)

Using (51) and the approximation (52), this weighting function amounts to the direct expression:

$$\begin{array}{ll}{w}_{\mathit{ij}}^{\left(2\right)}& \approx \frac{\Phi \left(y_{\text{min}}^{\prime }(i,j)\right)-\Phi \left(y_{\text{max}}^{\prime }(i,j)\right)}{\Phi \left(0\right)-1}\text{with}:\Phi \left(x\right)\\ =\text{erf}\left(\frac{x-{\gamma }^{-1}\left(z_{\mathit{ij}}\right)+\lambda {\alpha }_j^{-1/2}\Delta t_j^{-1}s\left({\gamma }^{-1}\left(z_{\mathit{ij}}\right)\right)}{\sqrt{s\left({\gamma }^{-1}\left(z_{\mathit{ij}}\right)\right)}}\right).\end{array}$$

Endnotes

^a We remark that due to quantum-mechanical aspects, in reality there is no such thing as an “ideal” signal, here we define the “ideal” signal as the voltage averaged over a “very” long exposure time.

^b Remark that Bayesian estimates (such as MAP, MMSE) are equally possible, given that prior information on E _i is available.

^c Due to Jensen’s inequality.

^d This is because the contribution of several noise sources can be modeled using a Gaussian distribution, with signal-dependent mean and variance.

^e Equation (13) is invariant under scaling of w _ij ; if a certain $w_{\mathit{ij}}^{\star }$ minimizes MSE then any scaled version $a{w}_{\mathit{ij}}^{\star }$ with a > 0 is also a solution to (24). By adding an extra constraint to the solution, this problem is solved.

^f More specifically, we use ${\left|{\sum }_j{a}_j{b}_j\right|}^2<{\sum }_j{a}_j^2{\sum }_l{b}_l^2$ with a _i  = w _ij σ^− 2(E _i ,Δ t _j )ν(E _i ,Δ t _j ) and b _l  = 1.

^g This follows from the fact that E _i  > 0, Δ t _j  > 0, α _j  > 0 and ${\sigma }_{\mathrm{o},j}^2>0$.

^h The exposure time is either stored in the RAW data files, or the EXIF information of the compressed JPEG files.

ⁱ To see this, note that $\sqrt{s\left(\mathrm{\Delta t\alpha }\right)}/\left(\sqrt{\mathrm{\Delta t}}\alpha \right)=\sqrt{\frac{{\sigma }_{\mathrm{o}}^2}{\mathrm{\alpha \Delta }{t}^2}+\mathrm{\Delta tE}+\frac{\beta }{\alpha }E},$ which increases monotonically in ${\sigma }_{\mathrm{o}}^2$ and β and decreases in α.

^j Available at http://people.csail.mit.edu/hasinoff/hdrnoise/.

^khttp://rawtherapee.com/.