EUVIMEDEUVIMED entdecken
Crossref

A Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical Concept Extraction

A H Alamoodi, Omar Zughoul, Dianese David, Salem Garfan, Dragan Pamucar, O S Albahri, A S Albahri, Salman Yussof, Iman Mohamad Sharaf

Journal of Medical Systems · 31.08.2024 · 48 (1)

DOI: 10.1007/s10916-024-02090-y ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Systems, online erschienen am 31.08.2024. Verfasst von A H Alamoodi und 8 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Informatics and Computing in Energy, Universiti Tenaga Nasional, Kajang, Malaysia. alamoodi.abdullah91@gmail.com. Hauptthemen laut MEDLINE: Fuzzy Logic, Artificial Intelligence. Die Arbeit wurde bisher 23-mal zitiert (Crossref) und nennt 40 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Artificial intelligence (AI) has become a crucial element of modern technology, especially in the healthcare sector, which is apparent given the continuous development of large language models (LLMs), which are utilized in various domains, including medical beings. However, when it comes to using these LLMs for the medical domain, there's a need for an evaluation platform to determine their suitability and drive future development efforts. Towards that end, this study aims to address this concern by developing a comprehensive Multi-Criteria Decision Making (MCDM) approach that is specifically designed to evaluate medical LLMs. The success of AI, particularly LLMs, in the healthcare domain, depends on their efficacy, safety, and ethical compliance. Therefore, it is essential to have a robust evaluation framework for their integration into medical contexts. This study proposes using the Fuzzy-Weighted Zero-InConsistency (FWZIC) method extended to p, q-quasirung orthopair fuzzy set (p, q-QROFS) for weighing evaluation criteria. This extension enables the handling of uncertainties inherent in medical decision-making processes. The approach accommodates the imprecise and multifaceted nature of real-world medical data and criteria by incorporating fuzzy logic principles. The MultiAtributive Ideal-Real Comparative Analysis (MAIRCA) method is employed for the assessment of medical LLMs utilized in the case study of this research. The results of this research revealed that "Medical Relation Extraction" criteria with its sub-levels had more importance with (0.504) than "Clinical Concept Extraction" with (0.495). For the LLMs evaluated, out of 6 alternatives, ( A 4 ) "GatorTron S 10B" had the 1st rank as compared to ( A 1 ) "GatorTron 90B" had the 6th rank. The implications of this study extend beyond academic discourse, directly impacting healthcare practices and patient outcomes. The proposed framework can help healthcare professionals make more informed decisions regarding the adoption and utilization of LLMs in medical settings.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Fuzzy LogicArtificial IntelligenceHumansDecision Making

Stichwörter der Autor:innen

pq-QROFSClinical Concept ExtractionFWZICMAIRCAMedical Relation Extraction

Autor:innen und Institutionen

  • Institute of Informatics and Computing in Energy, Universiti Tenaga Nasional, Kajang, Malaysia. alamoodi.abdullah91@gmail.com.
  • Applied Science Research Center, Applied Science Private University, Amman, Jordan. alamoodi.abdullah91@gmail.com.
  • MEU Research Unit, Middle East University, Amman, Jordan. alamoodi.abdullah91@gmail.com.
  • Information Systems and Computer Science Department, Ahmed bin Mohammed Military College, Al-Shahaniya, Qatar.
  • Faculty of Computing and Meta-Technology (FKMT), Universiti Pendidikan Sultan Idris (UPSI), Perak, Malaysia.
  • Széchenyi István University, Győr, Hungary.

Zitieren

APA

Alamoodi, A. H., Zughoul, O., David, D., Garfan, S., Pamucar, D., Albahri, O. S., Albahri, A. S., Yussof, S., & Sharaf, I. M. (2024). A Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical Concept Extraction. Journal of Medical Systems, 48(1). https://doi.org/10.1007/s10916-024-02090-y

Vancouver

Alamoodi AH, Zughoul O, David D, Garfan S, Pamucar D, Albahri OS, et al. A Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical Concept Extraction. Journal of Medical Systems. 2024;48(1). doi:10.1007/s10916-024-02090-y

BibTeX
@article{Alamoodi2024,
  author = {Alamoodi, A H and Zughoul, Omar and David, Dianese and Garfan, Salem and Pamucar, Dragan and Albahri, O S and Albahri, A S and Yussof, Salman and Sharaf, Iman Mohamad},
  title = {A Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical Concept Extraction},
  journal = {Journal of Medical Systems},
  year = {2024},
  volume = {48},
  number = {1},
  doi = {10.1007/s10916-024-02090-y}
}
Kostenlos starten →