EUVIMEDObjevte EUVIMED
Crossref

Medical semantic knowledge‐integrated multitask learning network for report generation and neoadjuvant chemotherapy response prediction

Wei Song, Ming Fan, Lihua Li

Medical Physics · 2025

DOI: 10.1002/mp.17925 ↗

At a glance

Journal article in Medical Physics, 2025. Written by Wei Song and 2 co-authors. Institution: School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China. Major MEDLINE topics: Semantics, Neoadjuvant Therapy, Machine Learning, Neural Networks, Computer. 50 references. Funded by: National Natural Science Foundation of China, Zhejiang Provincial Natural Science Foundation of China.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Automatically generated reports and predictions of neoadjuvant chemotherapy (NAC) responses can support physicians in clinical decision‐making. However, previous research has not thoroughly considered the connection between the detailed descriptions within these reports and their impact on predicting NAC responses.

Objectives

This study aims to develop an effective medical semantic knowledge‐integrated multitask learning (MSKMTL) network to enhance the accuracy of NAC response predictions and improve the quality of automatically generated medical reports.

Methods

We propose a MSKMTL network designed to increase report generation and NAC prediction by leveraging semantic knowledge within medical reports. The knowledge base was trained with DCE‐MRI and the corresponding reports to capture the interrelationships of paired visual‐text representations. During the training stage, semantic knowledge is encoded via a pretrained BERT model and stored in the knowledge base. We use contrastive learning to align the feature space of the image with the related report. In the inference stage, MSKMTL retrieves the stored semantic knowledge by querying image features to facilitate the generation of an impression section of the report and predict the NAC response. We evaluated the performance of the proposed network using metrics associated with both report generation and prediction tasks on a breast DCE‐MRI dataset and a public IU x‐ray dataset.

Results

For report generation, the BLEU‐1, BLEU‐4, ROUGE, and METEOR scores are 0.408, 0.186, 0.426, and 0.402, respectively. For NAC prediction, the area under the curve, recall, precision, and F1 score are 0.849, 0.750, 0.795, and 0.794, respectively. The network also showed strong performance with the IU x‐ray dataset.

Conclusion

Our study shows that, compared to other methodologies, integrating medical semantic knowledge increases prediction accuracy and generates superior reports.

Subjects

MeSH terms (MEDLINE)

SemanticsNeoadjuvant TherapyMachine LearningNeural Networks, ComputerHumansMagnetic Resonance ImagingBreast NeoplasmsKnowledge Bases

Author keywords

multi‐task learningneoadjuvant chemotherapy predictionreport generationsemantic knowledge base

Authors and institutions

  • School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China.
  • Institute of Intelligent Biomedicine, Hangzhou Dianzi University, Hangzhou, China.

Funding

  • National Natural Science Foundation of China
  • Zhejiang Provincial Natural Science Foundation of China

Cite

APA

Song, W., Fan, M., & Li, L. (2025). Medical semantic knowledge‐integrated multitask learning network for report generation and neoadjuvant chemotherapy response prediction. Medical Physics. https://doi.org/10.1002/mp.17925

Vancouver

Song W, Fan M, Li L. Medical semantic knowledge‐integrated multitask learning network for report generation and neoadjuvant chemotherapy response prediction. Medical Physics. 2025. doi:10.1002/mp.17925

BibTeX
@article{Song2025,
  author = {Song, Wei and Fan, Ming and Li, Lihua},
  title = {Medical semantic knowledge‐integrated multitask learning network for report generation and neoadjuvant chemotherapy response prediction},
  journal = {Medical Physics},
  year = {2025},
  doi = {10.1002/mp.17925}
}
Start for free →