Unsupervised Test-time Adaptation for Patient-Specific Deep Learning Models in Medical Imaging

Abstract

Deep learning (DL) models have achieved state-of-the-art results in multiple medical imaging applications, resulting in the widespread adoption of artificial intelligence (AI) models for radiological workflows. Despite their success, they exhibit two major weaknesses in real-world applications: a) poor generalization, and b) lack of patient optimality. The potential ‘distribution shift’ in unseen medical imaging data, due to changes in demography, acquisition hardware, etc., often leads to a reduction in performance. Solving this problem is critical in medical image analysis because this variance in performance across cases and abject failures on selected subjects will increase the burden on care-giving experts and reduce trust in AI-based applications. The aim of this thesis is to develop methods for patient-specific test-time adaptation of pretrained DL models on individual subject data. While generalization is often treated as a training time goal, methods developed in this thesis modifies the weights during inference time in an unsupervised manner specific to each subject’s data to jointly address generalization and patient optimality. While sustaining performance in individual subjects is seen as a challenge for current DL models, this line of research also offers an opportunity for next generation clinical decision support systems. AI methods that would result in patient-specific DL models can be heralded as the next big step in medical imaging.

Information Geometric Test Time Adaptation (IGTTA) for semantic segmentation: The test-time adaptation (TTA) of deep-learning-based semantic segmentation models, specific to individual patient data, was addressed in this part of the thesis. Existing TTA methods in medical imaging are often unconstrained and require prior anatomical information or additional neural networks built during the training phase, making them less practical and prone to performance deterioration. In this part of the thesis work, a novel framework based on information geometric principles was proposed to achieve generic, off-the-shelf, regularized patient-specific adaptation of models during test-time. By considering the pre-trained model and adapted models as part of statistical neuromanifolds, test-time adaptation was treated as constrained functional regularization using information geometric measures, leading to improved generalization and patient optimality. The efficacy of the proposed approach was shown for three challenging problems: a) improving the generalization of state-of-the-art models for segmenting COVID-19 anomalies in Computed Tomography (CT) images, b) cross-institutional brain tumour segmentation from magnetic resonance (MR) images, and c) segmentation of retinal layers in Optical Coherence Tomography (OCT) images. Furthermore, it was demonstrated that robust patient-specific adaptation can be achieved without adding a significant computational burden, making it the first of its kind, based on information geometric principles.

Test-Time Adaptation via Domain Transforms (TTADT): OCT imaging has emerged as the modality of choice for the diagnosis of retinal diseases. The popular, cost-effective variant of OCT - Spectral-Domain Optical Coherence Tomography (SDOCT) often suffers in image quality owing to speckle noise, making expert or automated analysis of these images challenging. This part of the thesis proposes, for the first-time inference time adaptation of deep learning models for a given test sample to achieve improved diagnostic performance. The adaptation is driven by domain transforms, a framework in which domain-specific augmentation is employed for a test sample, and the deep learning model weights are updated in a constrained and regularized manner. The performance of the proposed approach was evaluated on both simulated and real-world noisy B-mode OCT images affected by varying degrees of speckle noise from four benchmark SD-OCT datasets. Systematic studies clearly demonstrate the utility of the proposed approach in building generalized and robust deep learning models for automated retinal disease diagnosis using OCT images.

Uncertainty-Aware Test-Time Adaptation for inverse problems (UATTA): Deep learning models for inverse problems (both one-pass and unrolled versions): have these limitations: 1) inability to provide pixel-wise uncertainty quantification; 2) sensitivity to variation in forward model; and 3) suboptimal regularization over different iterations. In this part of the thesis, uncertainty aware test-time adaptation of off-the-shelf models is proposed for improved image reconstruction performance. Different pixel-wise uncertainty quantification methods were explored for their utility in test-time adaptation, along with data fidelity constraints. The results were studied to address the pertinent problem of quantitative susceptibility mapping for quantifying tissue magnetic susceptibility. The proposed framework provides reliable and accurate reconstruction of these images.

Test-Time Adaptation with Foundational Models (TTAFM): Recently, foundational models have demonstrated impressive promptable segmentation of natural images. However, these models often perform poorly with medical imaging data. In this part of the thesis, a two-way synergic approach for jointly utilizing foundational models while adapting pretrained medical imaging segmentation models is explored. The results are demonstrated on challenging medical imaging semantic segmentation tasks to show the ability of foundational models as neural surrogates, thus demonstrating the test-time adaptation achieved in real time. As these foundational models are capable of handling high levels of semantics, using them as surrogates helps reduce reliance on costly annotated data needed to achieve state-of-the-art results. In summary, this thesis developed methods to improve the generalization of deep-learning models utilized in medical imaging by personalizing them to individual patients during test time in an unsupervised manner. The developed methods demonstrate the applicability and utility on different medical imaging tasks across various modalities. This thesis is the first of its kind in medical imaging to showcase the utility of test-time adaptation methods, paving the way for further research towards the goal of precision medicine.