Advanced statistical tools for enhanced quality digital imaging with realistic capture models

Editorial

Getting closer to reality in modeling image capture devices is crucial for the improvement of image quality beyond the limits of image restoration algorithms as we know them today. This calls for more accurate statistical modeling of distortions and noise coming from real capture devices (Poisson noise, internal non-linearities, space variant point spread functions due to nonideal optics, chromatic aberrations, etc.). While these effects are often not considered in the restoration algorithms, their impact on the resulting image quality is huge in practice. For example, different nonlinearities (both intrinsic to the imaging device and induced ones, e.g., to make the noise signal independent) can invalidate typically assumed noise models and can also devastate deblurring. Joint modeling of digital and nondigital components (like optics and sensors) or various sources of image distortions (such as color filter array, blur, and noise) will likely yield improvements over the traditional approach to treat them separately. Rapid progress in digital camera technology makes a huge impact on computer vision, surveillance and security systems, production of portable electronic devices, such as smart phones. Physical limits of the sensors are likely to impose trade-offs on picture quality (e.g., in terms of achievable resolution versus noise considerations) that can be dealt with only by smart and device-aware signal processing. Moreover, new challenges arise from new imaging technologies, like in three-dimensional (3D) digital cameras [1, 2], and new acquisition modalities, such as compressive sensing [3–7].

One of the central topics to this special issue is realistic modeling of noise in digital cameras. With ongoing miniaturization, sensor elements of the camera are becoming increasingly sensitive to noise. In state-of-the-art image sensors, the pixel size is approaching 1 μ m. By shrinking the pixel size towards the wavelengths of light that the camera captures, the amount of light received will be further decreased by technical barriers (diffraction effects) [8]. Hence, increasing further the sensor resolution by itself will not necessarily lead to actual gains in image quality. Also, recent improvements in sensor sensitivity allow cameras to operate in very low lighting conditions, but this boosts noise in the acquired images. The negative effects of noise can be largely suppressed by post-processing algorithms, but these require precise knowledge of the noise characteristics to achieve optimal performance. Due to the mismatch between the actual noise characteristics and typically assumed, simplified noise models, the noise level is typically over- or underestimated, affecting adversely the performance of noise reduction.

Three papers in this special issue [9–11] address the problem of camera noise modeling and different aspects of noise reduction, ranging from post-processing steps for hardware solutions to purely signal processing post-production approaches. Available technologies for reducing noise in hardware include pixel binning [12, 13], where electrical charges of neighboring pixels are combined to form a superpixel. The benefits are faster readout speeds and improved signal-to-noise ratios albeit at the price of spatial resolution loss. Binning in color images is particularly challenging due to the presence of color filter array (CFA). To maintain color fidelity, the binning process combines neighboring pixels with the same color filter, such that the resulting superpixels form again a CFA, typically the Bayer pattern. The full-color image that is recovered from these subsampled sensor data by means of classical demosaicking is not only of lower resolution (compared to the case without binning), but suffers also from characteristic pixelization artifacts. Jin and Hirakawa provide in [9] a comprehensive characterization of pixel binning for color image sensors and propose post-capture signal processing steps to eliminate the binning artifacts. By a rigorous analysis, they prove that pixelization arises due to the mismatch between binning and demosaicking and that it cannot be eliminated simply by increasing the sensor resolution, but rather by a carefully designed binning-aware demosaicking approach.

The contribution by Aiazzi et al. [10] revisits optoelectronic noise model in view of recent advances in the technology of modern CCD color cameras. The authors present an original and robust method for estimating the parameters of a mixed photon and electronic noise from a single image. The main idea behind the method is to decouple the power of the signal-dependent photon noise from that of the signal-independent electronic noise in the space defined by sample mean versus variance, such that noise parameters can be estimated elegantly, by multivariate linear regression. The paper gives a comprehensive discussion on the range of validity of the optoelectronic noise model, clipping considerations in actual cameras and nonlinearities that dominate the camera response function (CRF), which gives new interesting insights with respect to related approaches like [14, 15]. Motivated by this analysis, the authors advocate noise estimation on the CRF-corrected data, i.e., after compensating for the nonlinearities introduced by the electronic chain, showing that in this case potential problems with the range of validity of the model are avoided.

Goossens et al. [11] take a different approach and present a detailed model of the camera noise, related to [16], which not only models a mixture of photon and electronic noise but also takes explicitly into account different aspects, such as amplification (ISO sensitivity of the camera), fixed pattern noise, clipping and quantization due to A/D conversion, and different post-processing operations through the CRF. This noise model is then applied to the reconstruction of high dynamic range images from a small set of low dynamic range acquisitions of a static scene. An original contribution of this work is also a novel Bayesian formulation of the weighting function for combining low dynamic range components into a high dynamic range image.

Another central and rapidly evolving field addressed in this special issue is image restoration. No imaging device has a perfect one-to-one correspondence of spatial directions in the scene (those ideally producing a single projection dot on the output image) and spatial locations in the resulting image. Instead, each of these incoming directions gives rise to a point spread function (PSF, a small ‘cloud’), whose shape and size depends (besides the incoming light wavelengths, in refraction-based optical devices) on both the image location from where the ray starts and on the imaging device itself (e.g., on its optical response, for devices capturing visible light). Other practically important source of blur is relative movement between the camera and objects in the scene. Such movements produce dynamically evolving images, which are temporally integrated by the sensor during the exposure time interval. Whereas all of these image degradations have been traditionally modeled in the image processing literature as simple convolutions, this is only a crude first approximation. In fact, the blur phenomenon is better modeled as a 3D field of PSFs depending on the relative coordinates (and movement) of the scene with respect to the camera.

This type of degradation, to a certain extent, is unavoidable in real imaging devices, and, coupled with the also omnipresent imaging noise, poses a formidable challenge for image processing scientists and engineers. The challenge at stake consists of correctly modeling and compensating for the various degradation sources, whereas keeping computational complexity of the restoration reasonably low.

In this special issue, we include four stimulating papers addressing different problems in image restoration. Santiago et al. [17] deal with the problem of boundary handling in image restoration. Despite its practical impact, this problem is often ignored in the restoration field, as evidenced by artificial boundary conditions such as circular or mirror-like boundary conditions used in typical simulation studies. Among the published works really addressing this problem, the model-based ones usually consider an output image with a smaller spatial support than the input image. Whereas this is a conceptually correct approach, it usually leads to highly complex (and, thus, somehow unpractical in most cases) restoration algorithms. Santiago et al. [17], instead, propose here a novel practical approach based on a multilayer perceptron (the classical neural network), which learns the degradation model. More regular results are produced in the total variation sense, without a priori constraints on the structure of the image or the boundary conditions. Compared to the previous methods, they achieve robust and more visually plausible reconstructions of the image on the boundary affected areas.

Seo and Milanfar [18] consider the problem of combining a short-exposure image obtained with flash and a long-exposure image taken without flash. The second is generally blurry due to the camera shake during exposure and also noisy due to poor illumination. The goal is to obtain the best of the two of them, i.e., the natural appearance (soft shadows, warm colors, etc.) of the one without flash, and the good definition and low noise of the one with flash. They propose a generalization of the guided filter approach of He et al. [19]. The process involves adjusting an unsupervised prediction model, which has both nonlinear components (reaction-diffusion driven) and a locally linear adaptive dependency. This novel approach results in a significant visual improvement with respect to previous methods.

Miraut et al. present two papers [20, 21] dealing with spatially variant PSF characterization and its corresponding image restoration. In both papers, the authors account for the fact that the PSFs affecting images are not spatially uniform for real imaging devices, but they rather change from one location of the acquired image to another one. In cases specified by the authors, when the PSF does not experience an abrupt change, a smooth PSF field may be used for globally characterizing the blur. In the first paper [20], Miraut and Portilla propose a novel model for representing smooth PSF fields, and for dealing with them efficiently, by means of deformable filtering techniques. A practical, but still model-based and general, methodology is developed on how to perform image restoration on a given image affected by a known, given, smooth PSF field. In the second paper [21], Miraut et al. address the problem of estimating the PSF field from a single observed image. This is done for the especially favorable case of star field images, where the objects are dot-like. Here, the assumption of a smooth PSF field is correct, and in addition, local PSFs are easy to estimate from the stars (almost ideal dot-like objects). Some real examples using ground-based telescope images show the ability of the proposed algorithm to effectively characterize the PSF field.

We close this special issue by two papers [22, 23] devoted to the topic of rapidly emerging sparse reconstruction and compressed sensing approaches. Puy et al. present in [22] a ‘spread spectrum’ compressed sensing strategy. The main idea behind this approach is to apply a wide bandwidth pre-modulation (i.e., chirp modulation) to the signal of interest before projecting it onto randomly selected vectors of an orthonormal basis. Authors consider the case of Fourier imaging, where pre-modulation with a wide-band signal amounts to a convolution in the Fourier domain, spreading the power spectrum of the original signal, while preserving its norm. Consequently, coherence between the sparsity and the sensing bases is drastically reduced, leading to enhanced reconstruction quality. Enhancement in the reconstruction quality stemming from this approach is demonstrated in two concrete applications: radio interferometric imaging and magnetic resonance imaging.

Finally, Dadkhahi et al. [23] offer a different approach to sparse representations: a reprojection of a signal represented in one basis onto another basis. In particular, the authors concentrate on constructing the sparse representation of piecewise smooth signals using the discrete cosine transform (DCT) by deriving the inverse polynomial reconstruction method for the DCT expansion. They show that this approach enables recovering piecewise smooth signals from a relatively small number of DCT coefficients with high accuracy. The paper demonstrates benefits of this framework in applications of signal and image denoising and approximation.