EUVIMEDOpdag EUVIMED
Open access

What Does DALL-E 2 Know About Radiology?

Lisa C Adams iD, Felix Busch iD, Daniel Truhn iD, Marcus R Makowski iD, Hugo J W L Aerts iD, Keno K Bressem iD

Journal of Medical Internet Research · 2023-03-16 · 25 · e43110

DOI: 10.2196/43110 ↗

At a glance

Journal article in Journal of Medical Internet Research, published online on 2023-03-16. Written by Lisa C Adams and 5 co-authors. Institution: Department of Radiology, Charité Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin,…. Major MEDLINE topics: Artificial Intelligence, Radiology. Cited 74 times so far (Crossref); 17 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Generative models, such as DALL-E 2 (OpenAI), could represent promising future tools for image generation, augmentation, and manipulation for artificial intelligence research in radiology, provided that these models have sufficient medical domain knowledge. Herein, we show that DALL-E 2 has learned relevant representations of x-ray images, with promising capabilities in terms of zero-shot text-to-image generation of new images, the continuation of an image beyond its original boundaries, and the removal of elements; however, its capabilities for the generation of images with pathological abnormalities (eg, tumors, fractures, and inflammation) or computed tomography, magnetic resonance imaging, or ultrasound images are still limited. The use of generative models for augmenting and generating radiological data thus seems feasible, even if the further fine-tuning and adaptation of these models to their respective domains are required first.

Subjects

MeSH terms (MEDLINE)

Artificial IntelligenceRadiologyHumansTomography, X-Ray ComputedMagnetic Resonance ImagingUltrasonography

Author keywords

DALL-Eartificial intelligencecreating images from textdiagnostic imaginggenerative modelimage creationimage generationmachine learningmedical imagingradiologytext-to-imagetransformer language modelx-ray

Authors and institutions

  • Department of Radiology, Charité Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany.
  • Department of Radiology, Stanford University, Stanford, CA, United States.
  • Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany.
  • Department of Diagnostic and Interventional Radiology, Technical University of Munich, Munich, Germany.
  • Artificial Intelligence in Medicine Program (AIM), Mass General Brigham, Harvard Medical School, Boston, MA, United States.
  • Radiology and Nuclear Medicine, CARIM & GROW, Maastricht University, Maastricht, Netherlands.

ORCID profiles: Lisa C Adams, Felix Busch, Daniel Truhn, Marcus R Makowski, Hugo J W L Aerts, Keno K Bressem

Cite

APA

Adams, L. C., Busch, F., Truhn, D., Makowski, M. R., Aerts, H. J. W. L., & Bressem, K. K. (2023). What Does DALL-E 2 Know About Radiology?. Journal of Medical Internet Research, 25, e43110. https://doi.org/10.2196/43110

Vancouver

Adams LC, Busch F, Truhn D, Makowski MR, Aerts HJWL, Bressem KK. What Does DALL-E 2 Know About Radiology?. Journal of Medical Internet Research. 2023;25:e43110. doi:10.2196/43110

BibTeX
@article{Adams2023,
  author = {Adams, Lisa C and Busch, Felix and Truhn, Daniel and Makowski, Marcus R and Aerts, Hugo J W L and Bressem, Keno K},
  title = {What Does DALL-E 2 Know About Radiology?},
  journal = {Journal of Medical Internet Research},
  year = {2023},
  volume = {25},
  pages = {e43110},
  doi = {10.2196/43110}
}
Start for free →