EUVIMEDEUVIMED entdecken
Open Access

Utilizing active learning strategies in machine-assisted annotation for clinical named entity recognition: a comprehensive analysis considering annotation costs and target effectiveness

Jiaxing Liu iD, Zoie S Y Wong

Journal of the American Medical Informatics Association · 2024

DOI: 10.1093/jamia/ocae197 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2024. Verfasst von Jiaxing Liu und 1 weiteren Autor:innen. Beteiligte Einrichtung: School of Statistics and Mathematics, Zhongnan University of Economics and Law, Wuhan, Hubei 430073, China. Hauptthemen laut MEDLINE: Supervised Machine Learning. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Natural Science Foundation of Hubei Province, Zhongnan University of Economics and Law, Fundamental Research Funds for the Central Universities.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Active learning (AL) has rarely integrated diversity-based and uncertainty-based strategies into a dynamic sampling framework for clinical named entity recognition (NER). Machine-assisted annotation is becoming popular for creating gold-standard labels. This study investigated the effectiveness of dynamic AL strategies under simulated machine-assisted annotation scenarios for clinical NER. Materials and Methods We proposed 3 new AL strategies: a diversity-based strategy (CLUSTER) based on Sentence-BERT and 2 dynamic strategies (CLC and CNBSE) capable of switching from diversity-based to uncertainty-based strategies. Using BioClinicalBERT as the foundational NER model, we conducted simulation experiments on 3 medication-related clinical NER datasets independently: i2b2 2009, n2c2 2018 (Track 2), and MADE 1.0. We compared the proposed strategies with uncertainty-based (LC and NBSE) and passive-learning (RANDOM) strategies. Performance was primarily measured by the number of edits made by the annotators to achieve a desired target effectiveness evaluated on independent test sets.

Results

When aiming for 98% overall target effectiveness, on average, CLUSTER required the fewest edits. When aiming for 99% overall target effectiveness, CNBSE required 20.4% fewer edits than NBSE did. CLUSTER and RANDOM could not achieve such a high target under the pool-based simulation experiment. For high-difficulty entities, CNBSE required 22.5% fewer edits than NBSE to achieve 99% target effectiveness, whereas neither CLUSTER nor RANDOM achieved 93% target effectiveness. Discussion and Conclusion When the target effectiveness was set high, the proposed dynamic strategy CNBSE exhibited both strong learning capabilities and low annotation costs in machine-assisted annotation. CLUSTER required the fewest edits when the target effectiveness was set low.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Supervised Machine LearningHumansNatural Language ProcessingDatasets as TopicElectronic Health Records

Stichwörter der Autor:innen

active learningclinical notesmedication extractionnamed entity recognitionpre-trained language model

Autor:innen und Institutionen

  • School of Statistics and Mathematics, Zhongnan University of Economics and Law, Wuhan, Hubei 430073, China.
  • Graduate School of Public Health, St Luke's International University, OMURA Susumu & Mieko Memorial St Luke's Center for Clinical Academia, Chuo-ku, Tokyo 104-0045, Japan.
  • The Kirby Institute, University of New South Wales, Sydney, NSW 2052, Australia.
  • School of Medical Sciences, The Unviersity of Sydney, Camperdown, NSW 2050, Australia.

ORCID-Profile: Jiaxing Liu

Förderung

  • Natural Science Foundation of Hubei Province
  • Zhongnan University of Economics and Law
  • Fundamental Research Funds for the Central Universities
  • Japan Society for the Promotion of Science KAKENHI

Zitieren

APA

Liu, J., & Wong, Z. S. Y. (2024). Utilizing active learning strategies in machine-assisted annotation for clinical named entity recognition: a comprehensive analysis considering annotation costs and target effectiveness. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocae197

Vancouver

Liu J, Wong ZSY. Utilizing active learning strategies in machine-assisted annotation for clinical named entity recognition: a comprehensive analysis considering annotation costs and target effectiveness. Journal of the American Medical Informatics Association. 2024. doi:10.1093/jamia/ocae197

BibTeX
@article{Liu2024,
  author = {Liu, Jiaxing and Wong, Zoie S Y},
  title = {Utilizing active learning strategies in machine-assisted annotation for clinical named entity recognition: a comprehensive analysis considering annotation costs and target effectiveness},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  doi = {10.1093/jamia/ocae197}
}
Kostenlos starten →