EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Predictive performance precision analysis in medicine: identification of low-confidence predictions at patient and profile levels (MED3pa I)

Olivier Lefebvre iD, Félix Camirand Lemyre, Jean-François Ethier, Lyna Hiba Chikouche, Ludmila Amriou, Dan Poenaru, Martin Vallières iD

Journal of the American Medical Informatics Association · 01.06.2026 · 33 (6) · 1100-1111

DOI: 10.1093/jamia/ocag034 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 31.03.2026. Verfasst von Olivier Lefebvre und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Université de Sherbrooke, Sherbrooke, QC J1K 2R1, Canada. Hauptthemen laut MEDLINE: Artificial Intelligence, Precision Medicine. Die Arbeit nennt 55 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Canada CIFAR AI Chair, Mila, Natural Sciences and Engineering Research Council of Canada (NSERC), Fonds de recherche du Québec-Nature et technologies, programme relève professorale.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Artificial Intelligence models are increasingly used in health care, yet global performance metrics can mask variations in reliability across individual patients or subgroups with shared attributes, called patient profiles. This study introduces predictive performance precision analysis in medicine (MED3pa), a method that identifies when models are less reliable, allowing clinicians to better assess model limitations. Materials and Methods We propose a framework that estimates predictive confidence using 3 combined approaches: individualized (IPC), aggregated (APC), and mixed predictive confidence (MPC). Individualized predictive confidence estimates confidence for each patient, APC assesses it across profiles, and MPC combines both. We evaluate our method on 4 datasets: 1 simulated, 2 public, and 1 private clinical dataset. Metrics by declaration rate curves show how performance changes when retaining only the most confident predictions, while interpretable decision trees reveal profiles with higher or lower model confidence.

Results

We demonstrate our method in internal, temporal, and external validation settings, as well as through a clinical example. In internal validation, limiting predictions to the 93% most confident cases improved sensitivity by 14.3% and the area under the receiver operating characteristic curve by 5.1%. In the clinical example, MED3pa identified a patient profile with high misclassification risk, demonstrating its potential for safer deployment.

Discussion

By identifying low-confidence predictions, our framework improves model reliability in clinical settings. It can be integrated into decision support systems to help clinicians make more informed decisions. Confidence thresholds help balance model performance with the proportion of patients for whom predictions are considered reliable.

Conclusion

Better leveraging confidence in model predictions could improve reliability and trustworthiness, supporting safer and more effective use in health care.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligencePrecision MedicineHumansPrediction AlgorithmsReproducibility of ResultsPredictive Learning Models

Stichwörter der Autor:innen

machine learningmodel performance evaluationmodel reliabilitypredictive confidence

Autor:innen und Institutionen

  • Department of Computer Science, Université de Sherbrooke, Sherbrooke, QC J1K 2R1, Canada.
  • Department of Mathematics, Université de Sherbrooke, Sherbrooke, QC J1K 2R1, Canada.
  • Department of Medicine, Université de Sherbrooke, Sherbrooke, QC J1H 5N4, Canada.
  • Department of Higher Studies (CS), École nationale Supérieure d'Informatique, Alger, 16309, Algeria.
  • Department of Pediatric Surgery, Montreal Children's Hospital, Montreal, QC H4A 3J1, Canada.
  • Centre for Outcomes Research and Evaluation, Research Institute of the McGill University Health Centre, Montreal, QC H4A 3J1, Canada.

ORCID-Profile: Olivier Lefebvre, Martin Vallières

Förderung

  • Canada CIFAR AI Chair, Mila
  • Natural Sciences and Engineering Research Council of Canada (NSERC)
  • Fonds de recherche du Québec-Nature et technologies, programme relève professorale
  • Canada CIFAR AI Chair
  • Fonds de recherche du Québec—Nature et technologies, programme relève professorale

Zitieren

APA

Lefebvre, O., Camirand Lemyre, F., Ethier, J. F., Chikouche, L. H., Amriou, L., Poenaru, D., & Vallières, M. (2026). Predictive performance precision analysis in medicine: identification of low-confidence predictions at patient and profile levels (MED3pa I). Journal of the American Medical Informatics Association, 33(6), 1100-1111. https://doi.org/10.1093/jamia/ocag034

Vancouver

Lefebvre O, Camirand Lemyre F, Ethier JF, Chikouche LH, Amriou L, Poenaru D, et al. Predictive performance precision analysis in medicine: identification of low-confidence predictions at patient and profile levels (MED3pa I). Journal of the American Medical Informatics Association. 2026;33(6):1100-1111. doi:10.1093/jamia/ocag034

BibTeX
@article{Lefebvre2026,
  author = {Lefebvre, Olivier and Camirand Lemyre, Félix and Ethier, Jean-François and Chikouche, Lyna Hiba and Amriou, Ludmila and Poenaru, Dan and Vallières, Martin},
  title = {Predictive performance precision analysis in medicine: identification of low-confidence predictions at patient and profile levels (MED3pa I)},
  journal = {Journal of the American Medical Informatics Association},
  year = {2026},
  volume = {33},
  number = {6},
  pages = {1100--1111},
  doi = {10.1093/jamia/ocag034}
}
Kostenlos starten →