Utilizing active learning strategies in machine-assisted annotation for clinical named entity recognition: a comprehensive analysis considering annotation costs and target effectiveness
Journal of the American Medical Informatics Association · 2024
DOI: 10.1093/jamia/ocae197 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2024. Verfasst von Jiaxing Liu und 1 weiteren Autor:innen. Beteiligte Einrichtung: School of Statistics and Mathematics, Zhongnan University of Economics and Law, Wuhan, Hubei 430073, China. Hauptthemen laut MEDLINE: Supervised Machine Learning. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Natural Science Foundation of Hubei Province, Zhongnan University of Economics and Law, Fundamental Research Funds for the Central Universities.
Abstract (Originalsprache)
Objectives
Active learning (AL) has rarely integrated diversity-based and uncertainty-based strategies into a dynamic sampling framework for clinical named entity recognition (NER). Machine-assisted annotation is becoming popular for creating gold-standard labels. This study investigated the effectiveness of dynamic AL strategies under simulated machine-assisted annotation scenarios for clinical NER. Materials and Methods We proposed 3 new AL strategies: a diversity-based strategy (CLUSTER) based on Sentence-BERT and 2 dynamic strategies (CLC and CNBSE) capable of switching from diversity-based to uncertainty-based strategies. Using BioClinicalBERT as the foundational NER model, we conducted simulation experiments on 3 medication-related clinical NER datasets independently: i2b2 2009, n2c2 2018 (Track 2), and MADE 1.0. We compared the proposed strategies with uncertainty-based (LC and NBSE) and passive-learning (RANDOM) strategies. Performance was primarily measured by the number of edits made by the annotators to achieve a desired target effectiveness evaluated on independent test sets.
Results
When aiming for 98% overall target effectiveness, on average, CLUSTER required the fewest edits. When aiming for 99% overall target effectiveness, CNBSE required 20.4% fewer edits than NBSE did. CLUSTER and RANDOM could not achieve such a high target under the pool-based simulation experiment. For high-difficulty entities, CNBSE required 22.5% fewer edits than NBSE to achieve 99% target effectiveness, whereas neither CLUSTER nor RANDOM achieved 93% target effectiveness. Discussion and Conclusion When the target effectiveness was set high, the proposed dynamic strategy CNBSE exhibited both strong learning capabilities and low annotation costs in machine-assisted annotation. CLUSTER required the fewest edits when the target effectiveness was set low.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Statistics and Mathematics, Zhongnan University of Economics and Law, Wuhan, Hubei 430073, China.
- Graduate School of Public Health, St Luke's International University, OMURA Susumu & Mieko Memorial St Luke's Center for Clinical Academia, Chuo-ku, Tokyo 104-0045, Japan.
- The Kirby Institute, University of New South Wales, Sydney, NSW 2052, Australia.
- School of Medical Sciences, The Unviersity of Sydney, Camperdown, NSW 2050, Australia.
ORCID-Profile: Jiaxing Liu
Förderung
- Natural Science Foundation of Hubei Province
- Zhongnan University of Economics and Law
- Fundamental Research Funds for the Central Universities
- Japan Society for the Promotion of Science KAKENHI
Zitieren
APA
Liu, J., & Wong, Z. S. Y. (2024). Utilizing active learning strategies in machine-assisted annotation for clinical named entity recognition: a comprehensive analysis considering annotation costs and target effectiveness. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocae197
Vancouver
Liu J, Wong ZSY. Utilizing active learning strategies in machine-assisted annotation for clinical named entity recognition: a comprehensive analysis considering annotation costs and target effectiveness. Journal of the American Medical Informatics Association. 2024. doi:10.1093/jamia/ocae197
BibTeX
@article{Liu2024,
author = {Liu, Jiaxing and Wong, Zoie S Y},
title = {Utilizing active learning strategies in machine-assisted annotation for clinical named entity recognition: a comprehensive analysis considering annotation costs and target effectiveness},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
doi = {10.1093/jamia/ocae197}
}
EUVIMED