Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models: Quantitative Study Using Large Language Models
Journal of Medical Internet Research · 2025
DOI: 10.2196/73325 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Internet Research, 2025. Verfasst von Tabinda Sarwar und 2 weiteren Autor:innen. Beteiligte Einrichtung: Royal Melbourne Institute of Technology University, Melbourne, Australia. Hauptthemen laut MEDLINE: Machine Learning, Data Accuracy, Language. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 71 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Data collected in controlled settings typically results in high-quality datasets. However, in real-world applications, the quality of data collection is often compromised. It is well established that the quality of a dataset significantly impacts the performance of machine learning models. In this context, detailed information about individuals is often recorded in progress notes. Given the critical nature of health applications, it is essential to evaluate the impact of textual data quality, as any incorrect prediction can have serious, potentially life-threatening consequences.
Objective
This study aims to quantify the quality of textual datasets and systematically evaluate the impact of varying levels of errors on feature representation and machine learning models. The primary goal is to determine whether feature representations and machine learning models are tolerant to errors and to assess whether investing additional time and computational resources to improve data quality is justified.
Methods
We developed a rudimentary error rate metric to evaluate textual dataset quality at the token level. The Mixtral large language model (LLM) was used to quantify and correct errors in low-quality datasets. The study analyzed two health care datasets: the high-quality MIMIC-III public hospital dataset (for mortality prediction) and a lower-quality private dataset from Australian aged care homes (AACHs; for depression and fall risk prediction). Errors were systematically introduced into MIMIC-III at varying rates, while the AACH dataset quality was improved using the LLM. Feature representations and machine learning models were assessed using the area under the receiver operating curve.
Results
For the sampled 35,774 and 6336 patients from the MIMIC and AACH datasets, respectively, we used Mixtral to introduce errors in MIMIC and correct errors in AACH. Mixtral correctly detected errors in 63% of progress notes, with 17% containing a single token misclassified due to medical terminology. LLMs demonstrated potential for improving progress note quality by addressing various errors. Under varying error rates (5%-20%, in 5% increments), feature representation performance was tolerant to lower error rates (
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Royal Melbourne Institute of Technology University, Melbourne, Australia.
ORCID-Profile: Tabinda Sarwar, Antonio José Jimeno Yepes, Lawrence Cavedon
Zitieren
APA
Sarwar, T., Yepes, A. J. J., & Cavedon, L. (2025). Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models: Quantitative Study Using Large Language Models. Journal of Medical Internet Research. https://doi.org/10.2196/73325
Vancouver
Sarwar T, Yepes AJJ, Cavedon L. Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models: Quantitative Study Using Large Language Models. Journal of Medical Internet Research. 2025. doi:10.2196/73325
BibTeX
@article{Sarwar2025,
author = {Sarwar, Tabinda and Yepes, Antonio José Jimeno and Cavedon, Lawrence},
title = {Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models: Quantitative Study Using Large Language Models},
journal = {Journal of Medical Internet Research},
year = {2025},
doi = {10.2196/73325}
}
EUVIMED