Leveraging Spatial Statistics for Domain Adaptation of Vision Language Models in Medical VQA

Abstract

Recent advances in Vision–Language Models (VLMs) have demonstrated strong performance in Medical Visual Question Answering (Medical VQA) task. Although they perform very well within their domains, these models often experience issues with their generalization ability on unknown clinical distribution data because of different imaging technologies and patient groups used in various medical facilities. Generalization problems faced by these models make their practical application in the field of VLM-based medical VQA systems rather difficult. To overcome this limitation we proposed our method named Spatial Semantics Aware Domain Adaptation (SSADA), which is an integrated framework that combines both finetuning and prompt-based in-context learning for domain adaptation. Our proposed approach, SSADA, includes the following three important components: (i) Mask-Aware Finetuning (MAFt) to make localization aware finetuning, (ii) Anatomy Aware Instance Normalization (AAIN) for handling intensity or distribition shift, and (iii)Weighted Multi-Modal Example Retrieval (WMMER) for semantically consistent example selection during inference. We evaluate the proposed framework on three publicly available Medical VQA benchmarks, SLAKE, VQA-Med 2019, and OmniMedVQA–RadImageNet, under cross-domain settings and compare it against standard finetuning techniques. Experimental results demonstrate the effectiveness of SSADA in improving cross-domain generalization.