Leveraging large language models for heuristic usability assessment of medical software: Insights with the Radiation Planning Assistant
Journal of Applied Clinical Medical Physics · 2026
DOI: 10.1002/acm2.70495 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Applied Clinical Medical Physics, 2026. Verfasst von Laurence E Court und 21 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA. Hauptthemen laut MEDLINE: Large Language Models, Software, Heuristics, Radiotherapy Planning, Computer-Assisted. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Usability engineering is essential for ensuring the safety and effectiveness of medical software, as design‐related issues are a leading cause of use errors in clinical settings. Heuristic evaluation provides a practical approach to identifying usability problems, but its outcomes depend heavily on expert interpretation. Large Language Models (LLMs), such as ChatGPT, offer a potential means to augment heuristic evaluation by generating structured, context‐aware usability feedback. This study explored the use of ChatGPT to support heuristic assessment of the Radiation Planning Assistant (RPA), a web‐based radiotherapy planning tool designed to support clinical teams in low‐ and middle‐income countries.
Methods
ChatGPT was provided with the RPA user and technical guides, training videos for each functional dashboard, and Zhang et al.’s 14 usability heuristics. The model was instructed to score each dashboard according to these heuristics, using Zhang's 0–4 severity scale, and to propose concrete interface improvements. The resulting feedback was reviewed and scored independently by the RPA developer team and by 13 users during a dedicated User Meeting. Comparative analysis was performed between ChatGPT, developer, and user ratings.
Results
ChatGPT identified 26 potential usability issues across six heuristic domains. The developer team considered nine of these actionable, though all were classified as minor (severity ≤ 2). User ratings showed wide variability, with nine suggestions achieving mean scores ≥ 1.5. Qualitative agreement between users and developers was limited, underscoring the importance of diverse perspectives in heuristic evaluation. Three suggestions—enhanced upload logs, reversible actions (“reopen request”), and stronger error prevention—were rated as potentially high priority by a minority of users. ChatGPT's ratings were consistent across dashboards.
Conclusions
While ChatGPT did not reveal any critical usability failures, its heuristic assessment proved valuable in prompting discussion, identifying minor refinements, and enriching both developer and user engagement with the RPA's interface design. This study demonstrates that LLMs can serve as an effective, low‐cost complement to conventional heuristic evaluation, supporting early‐stage usability review and stakeholder dialogue in the development of medical software.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA.
- Department of Medical Physics, University of the Free State School of Medicine, Bloemfontein, South Africa.
- Division of Medical Physics, Tygerberg Hospital, Stellenbosch University, Cape Town, South Africa.
- Department of Radiation Medicine, Groote Schuur Hospital, University of Cape Town, Cape Town, South Africa.
- Department of Radiation Oncology, Inkosi Albert Luthuli Central Hospital, Durban, South Africa.
- Department of Radiation Oncology, Charlotte Maxeke Johannesburg Academic Hospital, Wits University, Johannesburg, South Africa.
ORCID-Profile: Lourens Strauss, Andrea Marais, Thekiso Khotle, Christine V Chung, Pavel Govyadinov, Tucker Netherton
Zitieren
APA
Court, L. E., Smit, J., Strauss, L., Shaw, W., Marais, A., Trauernicht, C., Joubert, N., Smith, E., Badre, S., Lazarus, G. L., Khotle, T., Netherton, L., van Heerden, W., Cardenas, C., Serban, M., Seuntjens, J., Chung, C. V., Govyadinov, P., Khan, M., …, & Zhang, L. (2026). Leveraging large language models for heuristic usability assessment of medical software: Insights with the Radiation Planning Assistant. Journal of Applied Clinical Medical Physics. https://doi.org/10.1002/acm2.70495
Vancouver
Court LE, Smit J, Strauss L, Shaw W, Marais A, Trauernicht C, et al. Leveraging large language models for heuristic usability assessment of medical software: Insights with the Radiation Planning Assistant. Journal of Applied Clinical Medical Physics. 2026. doi:10.1002/acm2.70495
BibTeX
@article{Court2026,
author = {Court, Laurence E and Smit, Jacobus and Strauss, Lourens and Shaw, William and Marais, Andrea and Trauernicht, Christoph and Joubert, Nanette and Smith, Elaine and Badre, Shona and Lazarus, Graeme L and Khotle, Thekiso and Netherton, Lauren and van Heerden, Wanda and Cardenas, Carlos and Serban, Monica and Seuntjens, Jan and Chung, Christine V and Govyadinov, Pavel and Khan, Meena and Nair, Saurabh and Netherton, Tucker and Zhang, Lifei},
title = {Leveraging large language models for heuristic usability assessment of medical software: Insights with the Radiation Planning Assistant},
journal = {Journal of Applied Clinical Medical Physics},
year = {2026},
doi = {10.1002/acm2.70495}
}
EUVIMED