A MULTI-TASK MULTI-MODAL DEEP LEARNING MODEL FOR MEDICAL IMAGE SEGMENTATION AND CLASSIFICATION

Abstract

Computer vision plays a crucial role in automated medical image analysis, supporting diagnosis, treatment planning, and disease monitoring in modern healthcare. Among the many imaging techniques, Magnetic Resonance Imaging (MRI) is widely used due to its rich anatomical detail, particularly across its four key sub-modalities: T1-weighted (T1-W), T2-weighted (T2-W), contrast-enhanced T1 (T1-CE), and fluid￾attenuated inversion recovery (FLAIR). Medical image analysis involves tasks such as segmentation, classification, and reconstruction, where deep learning has shown significant success. However, most existing approaches are limited to either single￾task or single-modality learning, preventing them from leveraging complementary information across tasks and modalities. These limitations reduce models’generalizability and hinder their practical use in real-world clinical environments. Multi-task learning (MTL) offers a promising alternative by enabling the joint optimization of multiple tasks, resulting in improved feature representation and efficiency. Similarly, multi-modal fusion enriches the feature space by integrating complementary information from multiple imaging modalities. Despite these strengths, limited research has explored joint segmentation and classification in a multi-modal MTL framework thus, leaving a significant gap in solving the information bottleneck problem. Likewise, multi-objective optimization for multiple non-convex tasks across modalities remains under-investigated, leaving critical gaps in model versatility and robustness. The objective of this study was to develop a multi￾task multi-modal deep learning model for medical image segmentation and classification. A positivist research philosophy and experimental design were adopted. First, a systematic literature review identified the strengths and gaps of existing MTL approaches. This was complemented by an expert opinion survey to refine requirements. A comparative analysis of U-Net variants was then conducted to determine the most suitable architecture for the proposed model. Based on these insights, a novel MTL pipeline and architecture were designed, incorporating a hybrid multi-modal fusion strategy to integrate features across the four MRI modalities for both tasks. The model used hard-parameter sharing and multi-objective optimization techniques to jointly train the two non-convex tasks. Experiments were conducted using several datasets, with the BraTS 2021 Task 1 and Task 2 datasets serving as the primary datasets for model training and evaluation. Task 1 was used for brain tumour segmentation, while Task 2 was used for Radiogenomic classification of MGMT promoter methylation status. The model achieved state-of-the-art results with 95.08% accuracy, 95.46% F1-score, 97.04% precision, 99.2% recall, 99.15% AUC, 96.16% Dice score, and 88.51% Jaccard Index. The results show that by integrating multi￾modal fusion and joint learning, the model effectively exploits complementary information between tasks and modalities resulting into a pareto-optimal solution. The model was then rigorously validated for generalizability and versatility, it exhibited strong generalization capabilities across diverse task environments and maintained robustness even when presented with missing modalities. This study contributes to optimization theory, information theory, and computational learning theory by proposing a unified and generalizable framework for medical image analysis. It further aligns with global and national health priorities, including the United Nations Sustainable Development Goals, Africa’s Agenda 2063, and Kenya’s Universal Health Coverage agenda. Overall, the proposed approach provides a robust and scalable framework for AI-assisted medical image analysis in clinical environments. Future work should focus on extending the framework to additional tasks and modalities, longitudinal multi-task learning, and real-world clinical deployment and validation within healthcare systems.