Tight Frames, Non-convex Regularizers, and Quantized Neural Networks for Solving Linear Inverse Problems

Abstract

The recovery of a signal/image from compressed measurements involves formulating an optimization problem and solving it using an efficient algorithm. The optimization objective involves data fidelity, which is responsible for ensuring conformity of the reconstructed signal to the measurement, and a regularization term to enforce desired priors on the signal. More recently, the optimization based solvers have been replaced by deep neural networks.

This thesis considers four aspects of inverse problems in computational imaging:

• Part-I: Generalized least-squares based data-fidelity, which makes the effective sensing matrix/forward operator a Parseval tight frame; • Part-II: Sparsity promoting non-convex regularizers instead of the l1-norm; • Part-III: Analysis of deep-unfolded networks for solving the sparse recovery problem with emphasis on stability, training efficiency, and parameter selection; and • Part-IV: Neural networks with weight quantization, considering both multi-layer perceptron and convolutional neural networks.

Given below is a summary of the research contributions pertaining to each of the aforementioned parts.

Part-I – Generalized least-squares based data-fidelity: We consider the analysis-sparse l1-minimization problem with a generalized l2-norm-based data-fidelity and show that it effectively corresponds to using a tight-frame sensing matrix. Tight-frame sensing matrices result in minimum mean-squared-error recovery given oracle knowledge of the support of the sparse vector. The new formulation offers improved performance bounds when the number of non-zero entries is large. One could develop a tight-frame variant of a known sparse recovery algorithm using the proposed formalism. We solve the analysis-sparse recovery problem in an unconstrained setting using proximal methods. Within the tight-frame sensing framework, we rescale the gradients of the data-fidelity loss in the iterative updates to further improve the accuracy of analysis-sparse recovery. Experimental results show that the proposed algorithms offer superior analysis-sparse recovery performance. Proceeding further, we also develop deep-unfolded variants, with a convolutional neural network as the sparsifying operator. On the application front, we consider compressed sensing image recovery. Experimental results on Set11, BSD68, Urban100, and DIV2K datasets show that the proposed techniques outperform the state-of-the-art techniques, with performance measured in terms of peak signal-to-noise ratio (PSNR) and structural similarity index metric (SSIM). We extend the tight-frame formulation to the image deconvolution problem. We develop a novel formulation for image restoration considering a generalized data-fidelity loss and a convex regularization function that enforces a desired image prior, and solve it using proximal gradient-descent method. We further propose the plug-and-play counterpart of the restoration technique, which allows one to leverage off-the-shelf data-driven denoisers in place of the proximal operator. Experimental validations carried out on BSD500, Brodatz, Urban100, and DIV2K datasets show that the proposed technique gives rise to superior image reconstruction quality compared with the state-of-the-art techniques, with the performance measured in terms of PSNR and SSIM, with comparable computational complexity.

Part-II – Non-convex regularizers for promoting sparsity: Here, we consider alternatives to the l1-norm based sparsity promoting penalty. Non-convex regularizers such as the minimax concave penalty (MCP) and smoothly clipped absolute deviation (SCAD) penalty are employed to obtain superior accuracy in sparse recovery. We combine several penalties to achieve robust sparse recovery, while tackling the challenge of parameter tuning through deep-unfolded networks. We propose an ensemble of proximal networks for sparse recovery, where the ensemble weights are learnt in a data-driven fashion. We found that the proposed network performs superior to or on par with the individual networks in the ensemble for synthetic data under various noise levels and sparsity conditions. We demonstrate an application to image denoising based on the convolutional sparse coding formulation. We extend the ensemble of regularizers formulation to solve the CS-IR problem, where the multi-penalty formulation results in an iterative algorithm involving proximal-averaging. We then unfold the iterative algorithm into a trainable network that facilitates learning the sparsity prior. We call the resulting network as proximal-averaging network (PAN). The proposed approach offers superior reconstruction accuracy and quality compared with the state-of-the-art unfolding techniques. We also consider image deconvolution and super-resolution localization microscopy using non-convex penalties. Super-resolution localization microscopy allows imaging beyond the diffraction limit of conventional widefield microscopy and involves combining several blurry/low-resolution images into a single high-resolution image. We propose sparsity based deconvolution on the low-resolution images and combine the results to generate the high-resolution image. Specifically, we use minimax concave penalty for promoting sparsity. For experimental validation, we consider several test phantoms. For quantitative assessment, we use performance metrics such as precision, recall, Jaccard, mean-squared error, and mean-absolute deviation. The proposed methods with a combination of tight-frame data- fidelity and minimax concave penalty result in the best reconstructions compared with the state-of-the-art techniques.

Part-III – Analysis of deep-unfolded networks: In this part, we analyze the reasons behind the performance im- provements in deep-unfolded networks (DUNs) over iterative algorithms. DUNs are inspired by conventional iterative algorithms, where an iteration is transformed into a layer/block of a network with learnable parameters. Despite their huge success, the reasons behind their superior performance over their iterative counterparts are not fully understood. We focus on enhancing the explainability of DUNs by providing potential evidence behind their superior performance over traditional iterative methods. We concentrate on the Learned Iterative Shrinkage-Thresholding Algorithm (LISTA), a foundational contribution, which achieves sparse recovery with significantly fewer layers than its iterative counterpart, ISTA. By analyzing the distribution of singular values in the learned model, we explore the distinctions between LISTA, ISTA, and the recently proposed unfolded networks tight-frame LISTA (TF-LISTA) and analytical LISTA (ALISTA). Our findings reveal that the learned matrices in LISTA are unstable because the singular values exceed unity. Stability is restored by the activation function/nonlinearity. This study illustrates the capability of neural networks to learn specific patterns not allowed in traditional optimization schemes. On the other hand, alternative optimization approaches and deep unfolded networks with very few learnable parameters, such as TF-LISTA and ALISTA, can outperform LISTA on small training datasets. This analysis not only provides a deeper understanding of DUNs but also presents advancements in network training techniques that result in performance improvements over traditional methods. We also present a novel unbiasing technique that can further improve the performance gains obtained through deep-unfolded networks. The unbiasing technique corresponds to eliminating the nonlinearity in the last layer of the deep-unfolded network. Interestingly, the performance gains are substantial with this variant. We conclude by highlighting future research directions focused on improving the explainability of deep-unfolded networks and bridging the gap between theoretical models and practical, data-driven approaches.

Part-IV – Quantized neural networks for CS-IR: In this part, we consider quantized neural networks for solving the CS-IR problem. Quantization makes neural networks efficient both in terms of memory and computation during inference, and also renders them compatible for low-precision hardware deployment. We extend the proximal averaging network from Part-II to include the quantized weights in a convolutional sparsifying operator. Our learning algorithm is based on a variant of the ADAM optimizer in which the quantizer is part of the forward pass and the gradients of the loss function are evaluated corresponding to the quantized weights while doing a book-keeping of the high-precision weights. We demonstrate applications to CS-IR and magnetic resonance image reconstruction. The proposed approach offers superior reconstruction accuracy and quality than state-of-the-art unfolding techniques and the performance degradation is minimal even when the weights are subjected to extreme quantization. We also consider generative priors for solving inverse problems. We extend the concept of quantized weights to generative priors, i.e., the generator network weights come from a learned finite alphabet. We solve non-linear inverse problems using quantized generative models, where the sensing operator is a neural network. We introduce a new meta-learning framework that makes use of proximal operators and jointly optimizes the quantized weights of the generative model, parameters of the sensing network, and the latent-space representation. Experimental validation is carried out using standard datasets – MNIST, CIFAR10, SVHN, and STL10. The results show that the performance of 32-bit networks can be achieved using 4-bit networks. The performance of 1-bit networks is about 0.7 to 2 dB inferior, while saving significantly (32×) on the model size.