EUVIMEDSkopri EUVIMED
Open access

Extractive Clinical Question-Answering With Multianswer and Multifocus Questions: Data Set Development and Evaluation Study

Sungrim Moon iD, Huan He iD, Heling Jia iD, Hongfang Liu iD, Jungwei Wilfred Fan iD

JMIR AI · 2023-06-20 · 2 · e41818

DOI: 10.2196/41818 ↗

At a glance

Journal article in JMIR AI, published online on 2023-06-20. Written by Sungrim Moon and 4 co-authors. Institution: Department of Artificial Intelligence & Informatics, Mayo Clinic, Rochester, MN, United States. Cited 9 times so far (Crossref); 23 references. The full text is freely available in PubMed Central.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

Extractive question-answering (EQA) is a useful natural language processing (NLP) application for answering patient-specific questions by locating answers in their clinical notes. Realistic clinical EQA can yield multiple answers to a single question and multiple focus points in 1 question, which are lacking in existing data sets for the development of artificial intelligence solutions.

Objective

This study aimed to create a data set for developing and evaluating clinical EQA systems that can handle natural multianswer and multifocus questions.

Methods

We leveraged the annotated relations from the 2018 National NLP Clinical Challenges corpus to generate an EQA data set. Specifically, the 1-to-N, M-to-1, and M-to-N drug-reason relations were included to form the multianswer and multifocus question-answering entries, which represent more complex and natural challenges in addition to the basic 1-drug-1-reason cases. A baseline solution was developed and tested on the data set.

Results

The derived RxWhyQA data set contains 96,939 QA entries. Among the answerable questions, 25% of them require multiple answers, and 2% of them ask about multiple drugs within 1 question. Frequent cues were observed around the answers in the text, and 90% of the drug and reason terms occurred within the same or an adjacent sentence. The baseline EQA solution achieved a best F1-score of 0.72 on the entire data set, and on specific subsets, it was 0.93 for the unanswerable questions, 0.48 for single-drug questions versus 0.60 for multidrug questions, and 0.54 for the single-answer questions versus 0.43 for multianswer questions.

Conclusions

The RxWhyQA data set can be used to train and evaluate systems that need to handle multianswer and multifocus questions. Specifically, multianswer EQA appears to be challenging and therefore warrants more investment in research. We created and shared a clinical EQA data set with multianswer and multifocus questions that would channel future research efforts toward more realistic scenarios.

Subjects

Author keywords

artificial intelligencedata setdatasetinformation extractionnatural language processingquestion-answering

Authors and institutions

  • Department of Artificial Intelligence & Informatics, Mayo Clinic, Rochester, MN, United States.

ORCID profiles: Sungrim Moon, Huan He, Heling Jia, Hongfang Liu, Jungwei Wilfred Fan

Cite

APA

Moon, S., He, H., Jia, H., Liu, H., & Fan, J. W. (2023). Extractive Clinical Question-Answering With Multianswer and Multifocus Questions: Data Set Development and Evaluation Study. JMIR AI, 2, e41818. https://doi.org/10.2196/41818

Vancouver

Moon S, He H, Jia H, Liu H, Fan JW. Extractive Clinical Question-Answering With Multianswer and Multifocus Questions: Data Set Development and Evaluation Study. JMIR AI. 2023;2:e41818. doi:10.2196/41818

BibTeX
@article{Moon2023,
  author = {Moon, Sungrim and He, Huan and Jia, Heling and Liu, Hongfang and Fan, Jungwei Wilfred},
  title = {Extractive Clinical Question-Answering With Multianswer and Multifocus Questions: Data Set Development and Evaluation Study},
  journal = {JMIR AI},
  year = {2023},
  volume = {2},
  pages = {e41818},
  doi = {10.2196/41818}
}
Start for free →