EUVIMEDEUVIMED entdecken
LeitlinieOpen Access

Synthesized annotation guidelines are knowledge-lite boosters for clinical information extraction

Enshuo Hsu iD, Martin Ugbala, Krishna Kumar Kookal, Kawtar Zouaidi, Nicholas L Rider, Muhammad F Walji, Kirk Roberts iD

Journal of the American Medical Informatics Association · 2026

DOI: 10.1093/jamia/ocag084 ↗

Kurz gefasst

Leitlinie in der Zeitschrift Journal of the American Medical Informatics Association, 2026. Verfasst von Enshuo Hsu und 6 weiteren Autor:innen. Beteiligte Einrichtung: McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX 77030, United States. Hauptthemen laut MEDLINE: Large Language Models, Guidelines as Topic, Information Storage and Retrieval, Electronic Health Records. Die Arbeit nennt 36 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: National Institute of Allergy & Infectious Diseases, NLM NIH HHS, NIAID NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Generative information extraction using large language models (LLMs), particularly through prompting combined with few-shot learning, has become a popular method. In many ways such prompts with examples resemble the annotation guidelines long used for manual labeling of data for information extraction, and indeed studies have demonstrated the direct use of these guidelines as effective prompts. However, constructing annotation guidelines is both labor- and knowledge-intensive. Instead, this paper proposes to leverage LLMs’ impressive ability to automatically create such annotation guidelines. Materials and Methods Specifically, we propose a zero-shot hierarchical prompt engineering method that harvests the knowledge summarization and text generation capacity of LLMs to synthesize annotation guidelines to improve downstream LLMs while requiring minimal human input.

Results

Zero-shot clinical named entity recognition benchmarks, 2012 i2b2 EVENT, 2012 i2b2 TIMEX, 2014 i2b2, and 2018 n2c2 showed improvements of 0.2% to 25.86% for Llama 3.1 and 5.82% to 16.13% for GPT-OSS in strict F1 scores from the no-guideline baseline. The LLM-synthesized guidelines showed equivalent or better performance compared to human-written guidelines by 0.23% to 10.00% in most tasks.

Discussion

LLMs generate high-quality annotation guidelines following a consistent pattern (eg, title, entity types, examples) without human guidance, indicating that a representation of such a concept has been encoded during the pre-training. Nuances in definitions, however, still require adjustment by researchers to align with the project.

Conclusion

This study proposes a novel hierarchical prompt engineering method that requires minimal knowledge transfer from a human expert and is applicable to multiple biomedical domains.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsGuidelines as TopicInformation Storage and RetrievalElectronic Health RecordsNatural Language ProcessingHumansGenerative Artificial Intelligence

Stichwörter der Autor:innen

annotation guidelineslarge language modelsnatural language processingsynthesized data

Autor:innen und Institutionen

  • McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX 77030, United States.
  • Enterprise Development and Integration, University of Texas MD Anderson Cancer Center, Houston, TX 77030, United States.
  • School of Dentistry, University of Texas Health Science Center at Houston, Houston, TX 77054, United States.
  • Department of Health Systems & Implementation Science, Virginia Tech Carilion School of Medicine, Roanoke, VA 24016, United States.

ORCID-Profile: Enshuo Hsu, Kirk Roberts

Förderung

  • National Institute of Allergy & Infectious Diseases
  • NLM NIH HHS
  • NIAID NIH HHS
  • National Library of Medicine

Zitieren

APA

Hsu, E., Ugbala, M., Kookal, K. K., Zouaidi, K., Rider, N. L., Walji, M. F., & Roberts, K. (2026). Synthesized annotation guidelines are knowledge-lite boosters for clinical information extraction. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag084

Vancouver

Hsu E, Ugbala M, Kookal KK, Zouaidi K, Rider NL, Walji MF, et al. Synthesized annotation guidelines are knowledge-lite boosters for clinical information extraction. Journal of the American Medical Informatics Association. 2026. doi:10.1093/jamia/ocag084

BibTeX
@article{Hsu2026,
  author = {Hsu, Enshuo and Ugbala, Martin and Kookal, Krishna Kumar and Zouaidi, Kawtar and Rider, Nicholas L and Walji, Muhammad F and Roberts, Kirk},
  title = {Synthesized annotation guidelines are knowledge-lite boosters for clinical information extraction},
  journal = {Journal of the American Medical Informatics Association},
  year = {2026},
  doi = {10.1093/jamia/ocag084}
}
Kostenlos starten →