What Does DALL-E 2 Know About Radiology?
Journal of Medical Internet Research · 2023-03-16 · 25 · e43110
DOI: 10.2196/43110 ↗At a glance
Journal article in Journal of Medical Internet Research, published online on 2023-03-16. Written by Lisa C Adams and 5 co-authors. Institution: Department of Radiology, Charité Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin,…. Major MEDLINE topics: Artificial Intelligence, Radiology. Cited 74 times so far (Crossref); 17 references. The full text is freely available in PubMed Central.
Abstract (original language)
Generative models, such as DALL-E 2 (OpenAI), could represent promising future tools for image generation, augmentation, and manipulation for artificial intelligence research in radiology, provided that these models have sufficient medical domain knowledge. Herein, we show that DALL-E 2 has learned relevant representations of x-ray images, with promising capabilities in terms of zero-shot text-to-image generation of new images, the continuation of an image beyond its original boundaries, and the removal of elements; however, its capabilities for the generation of images with pathological abnormalities (eg, tumors, fractures, and inflammation) or computed tomography, magnetic resonance imaging, or ultrasound images are still limited. The use of generative models for augmenting and generating radiological data thus seems feasible, even if the further fine-tuning and adaptation of these models to their respective domains are required first.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Radiology, Charité Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany.
- Department of Radiology, Stanford University, Stanford, CA, United States.
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany.
- Department of Diagnostic and Interventional Radiology, Technical University of Munich, Munich, Germany.
- Artificial Intelligence in Medicine Program (AIM), Mass General Brigham, Harvard Medical School, Boston, MA, United States.
- Radiology and Nuclear Medicine, CARIM & GROW, Maastricht University, Maastricht, Netherlands.
ORCID profiles: Lisa C Adams, Felix Busch, Daniel Truhn, Marcus R Makowski, Hugo J W L Aerts, Keno K Bressem
Cite
APA
Adams, L. C., Busch, F., Truhn, D., Makowski, M. R., Aerts, H. J. W. L., & Bressem, K. K. (2023). What Does DALL-E 2 Know About Radiology?. Journal of Medical Internet Research, 25, e43110. https://doi.org/10.2196/43110
Vancouver
Adams LC, Busch F, Truhn D, Makowski MR, Aerts HJWL, Bressem KK. What Does DALL-E 2 Know About Radiology?. Journal of Medical Internet Research. 2023;25:e43110. doi:10.2196/43110
BibTeX
@article{Adams2023,
author = {Adams, Lisa C and Busch, Felix and Truhn, Daniel and Makowski, Marcus R and Aerts, Hugo J W L and Bressem, Keno K},
title = {What Does DALL-E 2 Know About Radiology?},
journal = {Journal of Medical Internet Research},
year = {2023},
volume = {25},
pages = {e43110},
doi = {10.2196/43110}
}
EUVIMED