ConTEXTual Net 3D: Vision-Language Modeling in PET/CT for Visual Grounding of Positive Findings
Journal of Imaging Informatics in Medicine · 14.04.2026
DOI: 10.1007/s10278-026-01879-2 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Imaging Informatics in Medicine, online erschienen am 14.04.2026. Verfasst von Zachary Huemann und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, University of Wisconsin-Madison, 1111 Highland Ave, Madison, WI, 53705, USA. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIBIB NIH HHS, National Institute of Biomedical Imaging and Bioengineering.
Abstract (Originalsprache)
Vision-language models can connect the text description of an object to its specific location in an image through visual grounding. This has potential applications in enhanced radiology reporting. However, these models require large annotated image-text datasets, which are lacking for PET/CT. We developed an automated pipeline to generate weak image-text labels and used it to train a 3D visual grounding model. Our weak-labeling pipeline identified sentences describing positive findings in PET/CT reports by searching for mentions of standardized uptake values (SUVmax) and axial slice numbers. These were used to automatically generate lesion masks, which were paired with the corresponding text descriptions. From 25,578 PET/CT exams, we extracted 11,356 sentence-label pairs. Using this data, we trained ConTEXTual Net 3D, which takes as input a description of a lesion and generates a corresponding segmentation mask. The model’s performance was evaluated on 251 radiologist-reviewed cases and compared against LLMSeg, a 2.5D version of ConTEXTual Net, and two radiologists. We evaluated detection performance using F1 score. The weak-labeling pipeline accurately identified lesion locations in 98% of cases (246/251). ConTEXTual Net 3D achieved an F1 score of 0.80, outperforming LLMSeg (F1 = 0.22) and the 2.5D model (F1 = 0.53), though it underperformed both radiologists (F1 = 0.94 and 0.91). The model achieved better performance on 18 F-fluorodeoxyglucose (F1 = 0.78) and DCFPyL (F1 = 0.75) exams than on DOTATATE (F1 = 0.58) and 18 F-fluciclovine (F1 = 0.66) exams. In conclusion, our novel weak labeling pipeline accurately produced an annotated dataset of PET/CT image-text pairs. ConTEXTual Net 3D significantly outperformed other models but fell short of the performance of nuclear medicine physicians. Our study suggests that even larger datasets may be needed to close this performance gap.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiology, University of Wisconsin-Madison, 1111 Highland Ave, Madison, WI, 53705, USA.
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, WI, USA.
- University of Wisconsin Carbone Cancer Center, Madison, WI, USA.
- Department of Radiology, University of Wisconsin-Madison, 1111 Highland Ave, Madison, WI, 53705, USA. tbradshaw@wisc.edu.
Förderung
- NIBIB NIH HHS
- National Institute of Biomedical Imaging and Bioengineering
Zitieren
APA
Huemann, Z., Church, S., Warner, J. D., Tran, D., Tie, X., McMillan, A. B., Hu, J., Cho, S. Y., Lubner, M. G., & Bradshaw, T. J. (2026). ConTEXTual Net 3D: Vision-Language Modeling in PET/CT for Visual Grounding of Positive Findings. Journal of Imaging Informatics in Medicine. https://doi.org/10.1007/s10278-026-01879-2
Vancouver
Huemann Z, Church S, Warner JD, Tran D, Tie X, McMillan AB, et al. ConTEXTual Net 3D: Vision-Language Modeling in PET/CT for Visual Grounding of Positive Findings. Journal of Imaging Informatics in Medicine. 2026. doi:10.1007/s10278-026-01879-2
BibTeX
@article{Huemann2026,
author = {Huemann, Zachary and Church, Samuel and Warner, Joshua D and Tran, Daniel and Tie, Xin and McMillan, Alan B and Hu, Junjie and Cho, Steve Y and Lubner, Meghan G and Bradshaw, Tyler J},
title = {ConTEXTual Net 3D: Vision-Language Modeling in PET/CT for Visual Grounding of Positive Findings},
journal = {Journal of Imaging Informatics in Medicine},
year = {2026},
doi = {10.1007/s10278-026-01879-2}
}
EUVIMED