Enhanced Embedding for Multimodal Medical Visual Question and Answering

Abstract

Visual Answering of questions in the field of Medical which is called as (VqA) has grown as a dominant area of research that fuse processing of natural language and vision of computer often known as CV or NLP to assist in medical decision-making. However, effective multimodal fusion between medical images and clinical questions remains a significant challenge. This thesis examines the application of the Perceiver IO architecture as an efficient multimodal aggregator for medical VQA. The work has been carried out in multiple directions. First, a classification-based framework is developed by combining Vision Transformer (ViT) and ClinicalBERT alongside a Perceiver IO aggregator to perform multimodal fusion for generating answers to medical questions. In the second approach, the florENCE TWO vision language model is finely tuned with parameter-efficient techniques to enable generative medical VQA. Finally, a hybrid architecture is introduced, where Perceiver IO is employed as a fusion module to integrate visual and textual representations, which are then used to condition the Florence-2 model for answer generation. In this thesis, the VQARAD dataset and ImageCLEF Med VQA 2019 dataset are used for the experiments. Through these explorations, the thesis examines both the classification and generative paradigms in area of medical Visual Question and answering and analyzes the importance of Perceiver IO for multimodal understanding. Several challenges encountered in hybrid modeling are discussed, along with right direction in terms of research in future, including the growing development of effective fusion strategies and the extension of the proposed approaches to larger medical VQA datasets. Overall, this thesis contributes to the understanding of efficient multimodal fusion techniques and provides insights for building both classification and more general systems in medical question along with answers on visual data.