Simultaneous Tumor Delineation and Report Generation from Brain MR Images

Abstract

Brain tumor analysis is an important application of medical image computing, where accurate segmentation and interpretation of tumor regions can support diagnosis and treatment planning. However, existing methods often address tumor segmentation and radiology report generation as separate tasks. Moreover, one of the major challenges in report generation tasks from MRI is accurate tumor localization. Most models fail to locate the lobe and hemisphere in which the tumor is located, causing incorrect report generation. In this regard, a unified 3D vision-language framework is proposed for simultaneous brain tumor segmentation and report generation from multi-modal MRI. Given T1, T2, T1C, and FLAIR volumes, the proposed model predicts clinically meaningful tumor regions and generates a textual description of tumor location and appearance. In order to encourage consistency between the predicted segmentation and generated report, the proposed model judiciously integrates a Swin UNETR-based 3D encoder-decoder, a multi-scale lesion tokenizer, auxiliary clinical grounding heads, and an iterative crossmodal refinement module. Moreover, anatomy and laterality heads are introduced, which provide clinical hints to the LLM, allowing better tumor localization. Further, an iterative refinement is incorporated so that the generated report and segmented outputs can refine each other, finally producing better outputs. Extensive experimentation on BraTS2020 and TextBraTS data sets shows that the proposed model achieves a mean Dice of 81.60% and HD95 of 6.21. In addition, the model achieves a BERTScore-F1 of 0.9226, clinical laterality F1 of 0.8459, clinical anatomy F1 of 0.7539 and clinical pathology F1 of 0.9976. These results indicate that the proposed framework can generate clinically meaningful reports while accurately localizing tumor regions and maintaining strong alignment between segmentation and textual interpretation.