EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Controlled Single-Speaker Evaluation of Ambient Voice Technologies’ Speech-to-Text Function Using Narrated Surgical Case-Reports

Ruairi O'Kane, Daniel Stonehouse-Smith, Melody Shirazi, Rachel Hutchinson, Morgan Gregg, Riya Patel, Jadbinder Seehra, Spyridon N Papageorgiou, Martyn T Cobourne

Journal of Medical Systems · 2026

DOI: 10.1007/s10916-026-02471-5 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Systems, 2026. Verfasst von Ruairi O'Kane und 8 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Dentistry, Oral & Craniofacial Sciences, Centre for Craniofacial Development & Regeneration, King's College London, Floor 27, SE1 9RT, Lon…. Hauptthemen laut MEDLINE: Speech Recognition Software, Case Reports as Topic. Die Arbeit nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Medical Research Council, National Institute of Health and Care Research.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Ambient voice technologies (AVT) utilise automatic speech recognition (ASR) and generative artificial intelligence for speech-to-text conversion, and are advocated as efficient, accurate tools for clinical documentation. We investigated ( n = 8) AVT systems using surgical case-reports as narratives in a controlled single-speaker evaluation prioritising safety outcomes: specifically 3 commercial transcription-systems (Dragon-Medical-One, Heidi-Health, Tortus); 4 ASR speech-to-text Application Programming Interfaces (Speechmatics-Enhanced, Amazon-Medical-Transcribe, Whisper, GPT4oTranscribe); and an experimental two-stage ASR–Large Language Model (LLM) pipeline incorporating GPT4oTranscribe with GPT-5-LLM generative error correction (GPT4oTranscribe-Corrected-5). Reference-transcripts ( n = 100; 32,897 words, range = 44–449, mean = 329/per-transcript) derived from surgical case-reports were recorded and input into AVT systems as audio-recordings for transcription-output generation. Primary outcome was proportion of transcription-outputs containing at least one clinically significant Class 3 error graded for potential harm. Secondary outcomes included transcription accuracy: Domain-Word-Error-Rate (DWER) against SNOMED-CT, lexical-accuracy (ROUGE score) and semantic similarity (BERT, BART scores). To investigate impact of the experimental pipeline on clinically significant errors, raw ASR and LLM-processed transcription-outputs were compared and errors classified as resolved, remaining or newly introduced. Across ( n = 800) transcript-outputs, 30–68% (GPT4oTranscribe-Corrected-5; Amazon-Medical-Transcribe and Dragon-Medical-One, respectively) contained at least one clinically significant Class 3 error. LLM-processing reduced the proportion of transcript-outputs affected (53% to 30%), amongst 89 clinically significant errors in ASR-output, 51 were resolved, 38 remained significant and 4 were newly introduced. Reference-transcript length significantly increased odds of Class 3 errors for 4 systems (GPT4oTranscribe, Tortus, Amazon-Medical-Transcribe, Dragon-Medical-One; odds ratio 2.65–3.34/additional 100 reference-words). Errors with potential to cause severe harm or death (NHS England Level 3–4) totalled ( n = 53) across systems, concentrated in the domains of medication-type/dose (39.6%), and investigations/laboratory results (30.2%). Performance varied across systems for transcription metrics ( P < 0.001). GPT4oTranscribe-Corrected-5 (DWER = 3.60%) and Heidi-Health (DWER = 5.67%) performed best for medical terminology; Amazon-Medical-Transcribe worst (DWER = 24.03%). For lexical and semantic similarity GPT4oTranscribe-Corrected-5 performed best, followed by Heidi-Health. The ASR–LLM pipeline reduced clinically significant errors but did introduce new ones in 3% of transcript-outputs. The safety and effectiveness of these systems in clinical practice requires further evaluation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Speech Recognition SoftwareCase Reports as TopicHumansLarge Language ModelsGenerative Artificial IntelligenceNarration

Stichwörter der Autor:innen

Artificial intelligenceMedical recordsNatural language processingSpeech recognition softwareSurgeryTranscription accuracy

Autor:innen und Institutionen

  • Faculty of Dentistry, Oral & Craniofacial Sciences, Centre for Craniofacial Development & Regeneration, King's College London, Floor 27, SE1 9RT, London, UK.
  • Department of Orthodontics, Guy's & St Thomas' NHS Foundation Trust, London, UK.
  • Department of Oral & Maxillofacial Surgery, King's College Hospital NHS Foundation Trust, London, UK.
  • Department of Orthodontics, Royal Alexandra Children's Hospital, Sussex University Hospitals NHS Foundation Trust, Brighton, UK.
  • Centre of Dental Medicine, Clinic of Orthodontics and Paediatric Dentistry, University of Zurich, Zurich, Switzerland.
  • Faculty of Dentistry, Oral & Craniofacial Sciences, Centre for Craniofacial Development & Regeneration, King's College London, Floor 27, SE1 9RT, London, UK. martyn.cobourne@kcl.ac.uk.

Förderung

  • Medical Research Council
  • National Institute of Health and Care Research

Zitieren

APA

O'Kane, R., Stonehouse-Smith, D., Shirazi, M., Hutchinson, R., Gregg, M., Patel, R., Seehra, J., Papageorgiou, S. N., & Cobourne, M. T. (2026). A Controlled Single-Speaker Evaluation of Ambient Voice Technologies’ Speech-to-Text Function Using Narrated Surgical Case-Reports. Journal of Medical Systems. https://doi.org/10.1007/s10916-026-02471-5

Vancouver

O'Kane R, Stonehouse-Smith D, Shirazi M, Hutchinson R, Gregg M, Patel R, et al. A Controlled Single-Speaker Evaluation of Ambient Voice Technologies’ Speech-to-Text Function Using Narrated Surgical Case-Reports. Journal of Medical Systems. 2026. doi:10.1007/s10916-026-02471-5

BibTeX
@article{OKane2026,
  author = {O'Kane, Ruairi and Stonehouse-Smith, Daniel and Shirazi, Melody and Hutchinson, Rachel and Gregg, Morgan and Patel, Riya and Seehra, Jadbinder and Papageorgiou, Spyridon N and Cobourne, Martyn T},
  title = {A Controlled Single-Speaker Evaluation of Ambient Voice Technologies’ Speech-to-Text Function Using Narrated Surgical Case-Reports},
  journal = {Journal of Medical Systems},
  year = {2026},
  doi = {10.1007/s10916-026-02471-5}
}
Kostenlos starten →