EUVIMEDOtkrijte EUVIMED
VergleichsstudieOpen access

Assessing Eligibility for Anticancer Drug Health Insurance Reimbursement Using Large Language Models: Benchmark Development and Comparative Study

Junhyuk Seo, Taerim Kim, Ju-Hyun Kim

Journal of Medical Internet Research · 2026-06-15 · 28 · e95877-e95877

DOI: 10.2196/95877 ↗

At a glance

Vergleichsstudie in Journal of Medical Internet Research, published online on 2026-06-15. Written by Junhyuk Seo and 2 co-authors. Institution: Healthcare Research Institute, ETOILE Inc, Seoul, Republic of Korea. Major MEDLINE topics: Antineoplastic Agents, Benchmarking, Eligibility Determination, Insurance, Health, Reimbursement. Cited 1 times so far (Crossref); 31 references. The full text is freely available in PubMed Central.

SažetakSubjectsBibliografski podaciCiteRelated studies

Abstract (original language)

Background

Administrative costs in the health care system are driven in part by complex insurance eligibility determinations. Large language models (LLMs) are increasingly used for health insurance–related queries, yet their reliability for structured logical reasoning over coverage criteria has not been systematically evaluated.

Objective

This study aimed to develop a benchmark for anticancer drug reimbursement eligibility determination and evaluate whether LLMs can reliably perform eligibility verification.

Methods

We constructed a benchmark based on South Korea’s National Health Insurance reimbursement guidelines for 3 gynecologic cancers (cervical, uterine, and ovarian), using a tristate adjudication framework (eligible, ineligible, and undeterminable). Three gynecologic oncology experts and a utilization review nurse validated the benchmark. Six LLMs from 3 providers (Anthropic, Google, and OpenAI) were evaluated using the official guideline document as input. Each case was evaluated 3 times per model, with final predictions determined by majority vote, and performance was compared across the 3 outcome classes.

Results

The benchmark comprises 74 anticancer regimens with 222 cases. Overall verification accuracy ranged from 77.9% to 88.7% across the 6 models. Eligible and ineligible cases were classified with high recall (86.5%‐98.6%), but undeterminable cases showed a marked decline across all models (44.6%‐70.3%). Performance varied by cancer type, with uterine cancer showing the lowest undeterminable recall (16.7%), corresponding to the highest guideline complexity. Undeterminable cases were predominantly misclassified as eligible rather than ineligible. The tristate framework enabled logic-based error analysis of 235 incorrect predictions, revealing information gap-filling as the dominant failure pattern (n=196, 83.4%), followed by criterion misapplication (n=20, 8.5%) and false uncertainty (n=19, 8.1%). Subtype analysis indicated that information gap-filling errors were concentrated at hierarchical elements of the guideline. Sensitivity analyses showed that converting the guideline document to structured text degraded performance, while web search–enabled condition (0%‐3.2% tool invocation across models) and structure-guided prompting did not produce significant changes from baseline.

Conclusions

In this benchmark, LLMs classified clearly eligible and ineligible cases with relatively high recall but showed limited reliability on undeterminable cases. The dominant error pattern was information gap-filling, in which models inferred eligibility rather than withholding judgment. These findings indicate that LLMs, in their current form, should be deployed as supervised decision-support tools rather than as independent adjudicators in reimbursement review.

Subjects

MeSH terms (MEDLINE)

Antineoplastic AgentsBenchmarkingEligibility DeterminationInsurance, Health, ReimbursementLarge Language ModelsFemaleHumansRepublic of Korea

Author keywords

National Health Insuranceanticancer drugsbenchmarkclinical decision supporteligibility verificationgynecologic cancerhealth insurancelarge language modelsnatural language processingreimbursement

Substances

Antineoplastic Agents

Authors and institutions

  • Healthcare Research Institute, ETOILE Inc, Seoul, Republic of Korea.
  • Department of Digital Health, Samsung Advanced Institute of Health Sciences and Technology (SAIHST), Sungkyunkwan University, Seoul, Republic of Korea.
  • Department of Emergency Medicine, Sungkyunkwan University School of Medicine, Samsung Medical Center, Seoul, Republic of Korea.
  • Department of Obstetrics and Gynecology, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Republic of Korea, 82 10-9668-6227.

Cite

APA

Seo, J., Kim, T., & Kim, J. H. (2026). Assessing Eligibility for Anticancer Drug Health Insurance Reimbursement Using Large Language Models: Benchmark Development and Comparative Study. Journal of Medical Internet Research, 28, e95877-e95877. https://doi.org/10.2196/95877

Vancouver

Seo J, Kim T, Kim JH. Assessing Eligibility for Anticancer Drug Health Insurance Reimbursement Using Large Language Models: Benchmark Development and Comparative Study. Journal of Medical Internet Research. 2026;28:e95877-e95877. doi:10.2196/95877

BibTeX
@article{Seo2026,
  author = {Seo, Junhyuk and Kim, Taerim and Kim, Ju-Hyun},
  title = {Assessing Eligibility for Anticancer Drug Health Insurance Reimbursement Using Large Language Models: Benchmark Development and Comparative Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e95877--e95877},
  doi = {10.2196/95877}
}
Start for free →