Low Light Image Restoration: Models, Algorithms and Learning with Limited Data

Abstract

The ability to capture high quality images under low-light conditions is an important feature of most hand-held devices and surveillance cameras. Images captured under such conditions often suffer from multiple distortions such as poor contrast, low brightness, color-cast and severe noise. While adjusting camera hardware settings such as aperture width, ISO level and exposure time can improve the contrast and brightness levels in the captured image, they often introduce artifacts including shallow depth-of-field, noise and motion blur. Thus, it is important to study image processing approaches to improve the quality of low-light images. In this thesis, we study the problem of low-light image restoration. In particular, we study the design of low-light image restoration algorithms based on statistical models, deep learning architectures and learning approaches when only limited labelled training data is available.

In our statistical model approach, the low-light natural image in the band pass domain is modelled by statistically relating a Gaussian scale mixture model for the pristine image, with the low-light image, through a detail loss coefficient and Gaussian noise. The detail loss coefficient in turn is statistically described using a posterior distribution with respect to its estimate based on a prior contrast enhancement algorithm. We then design our low-light enhancement and denoising (LLEAD) method by computing the minimum mean squared error estimate of the pristine image band pass coefficients. We create the Indian Institute of Science low-light image dataset of well-lit and low-light image pairs to learn the model parameters and evaluate our enhancement method. We show through extensive experiments on multiple datasets that our method helps better enhance the contrast while simultaneously controlling the noise when compared to other classical joint contrast enhancement and denoising methods.

Deep convolutional neural networks (CNNs) based on residual learning and end-to-end multiscale learning have been successful in achieving state of the art performance in image restoration. However, their application to joint contrast enhancement and denoising under low-light conditions is challenging owing to the complex nature of the distortion process involving both loss of details and noise. We address this challenge through two lines of approaches, one which exploits the statistics of natural images and the other which exploits the structure of the distortion process. We first propose a multiscale learning approach by learning the subbands obtained in a Laplacian pyramid decomposition. We refer to our framework as low-light restoration network (LLRNet). Our approach consists of a bank of CNNs where each CNN is trained to learn to explicitly predict different subbands of the Laplacian pyramid of the well exposed image. We show through extensive experiments on multiple datasets that our approach produces better quality restored images when compared to other low-light restoration methods.

In our second line of approach, we learn a distortion model that relates a noisy low- light and ground truth image pair. The low-light image is modeled to suffer from contrast distortion and additive noise. We model the loss of contrast through a parametric function, which enables the estimation of the underlying noise. We then use a pair of CNN models to learn the noise and the parameters of a function to achieve contrast enhancement. This contrast enhancement function is modeled as a linear combination of multiple gamma transforms. We show through extensive evaluations that our low-light Image Model for Enhancement Network (LLIMENet) achieves superior restoration performance when compared to other methods on several publicly available datasets.

While CNN models are fairly successful in low-light image restoration, such approaches require a large number of paired low-light and ground truth image pairs for training. Thus, we study the problem of semi-supervised learning for low-light image restoration when limited low-light images have ground truth labels. Our main contributions in this work are twofold. We first deploy an ensemble of low-light restoration networks to restore the unlabeled images and generate a set of potential pseudo-labels. We model the contrast distortions in the labeled set to generate different sets of training data and create the ensemble of networks. We then design a contrastive self-supervised learning based image quality measure to obtain the pseudo-label among the images restored by the ensemble. We show that training the restoration network with the pseudo-labels allows us to achieve excellent restoration performance even with very few labeled pairs. Our extensive experiments on multiple datasets show the superior performance of our semi-supervised low-light image restoration compared to other approaches.

Finally, we study an even more constrained problem setting when only very few labelled image pairs are available for training. To address this challenge, we augment the available labelled data with large number of low-light and ground-truth image pairs through a CNN based model that generates low-light images. In particular, we introduce a contrast distortion auto-encoder framework that learns to disentangle the contrast distortion and content features from a low-light image. The contrast distortion features from a low-light image are then fused with the content features from another pristine image to create a low-light version of the pristine image. We achieve the disentanglement of distortion from image content through the novel use of a contrastive loss to constrain the training. We then use the generated data to train low-light restoration models. We evaluate our data generation method in the 5-shot and 10-shot labelled data settings to show the effectiveness of our models.