EUVIMEDDescobrir EUVIMED
LeitlinieDelphi-StudieOpen access

From Divergent Guidelines to Delphi-Ready Items: Tutorial for Large Language Model–Assisted Guideline Harmonization Using Penile Cancer as a Worked Example

Von divergenten Leitlinien zu Delphi-fähigen Items: Tutorial zur LLM-gestützten Harmonisierung von Leitlinien am Beispiel Peniskarzinom

Maximilian Haas, Emily Rinderknecht, Matthias May, Maximilian Burger, Christian Gilfrich, Anne Herrmann, Sabine Brookman-May, Christoph Alexander Eckl, Udo Kruschwitz, Samy Ateia, Gregor Donabauer

Journal of Medical Internet Research · 2026-10-20 · 28 · e96886-e96886

DOI: 10.2196/96886 ↗

Auf einen Blick

Das Tutorial zeigt einen reproduzierbaren Workflow, um divergente Leitlinienempfehlungen und Literatur-Lücken in Delphi-fähige Kandidatenitems zu überführen. Im Beispiel Peniskarzinom entstanden 62 begutachtete Items.

Fragestellung

Wie lässt sich ein reproduzierbarer Workflow zur Harmonisierung divergenter klinischer Leitlinien und Literatur-Lücken mit LLMs entwickeln und demonstrieren?

Methode

Tutorial mit Worked Example Peniskarzinom; manuelle Leitlinienauswahl, PubMed-Suche, 3 LLMs, vordefinierte Prompts, multimodale Extraktion, semantische Harmonisierung, Stance-Klassifikation und externe Expertenbewertung.

Ergebnisse

Aus 5 Leitlinien wurden 157 Empfehlungen extrahiert und zu 68 harmonisierten Items verdichtet. Davon waren 15 einheitlich unterstützt; parallel entstanden 95 White-Area-Items, 9 davon echte Lücken. Ergebnis waren 62 Delphi-fähige Items.

Schlussfolgerung der Autor:innen

Die Methode bietet laut Autor:innen einen skalierbaren, transparenten und übertragbaren Ansatz zur Identifikation und Strukturierung von Leitlinieninkonsistenzen und Evidenzlücken.

Kernaussagen

  • 157 Empfehlungen aus 5 Leitlinien wurden zu 68 Items harmonisiert
  • 15 Items waren einheitlich unterstützt, 53 divergent/unklar
  • 62 human-reviewed Delphi-ready candidate items entstanden

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Leitlinie in Journal of Medical Internet Research, published online on 2026-10-20. Written by Maximilian Haas and 10 co-authors. Institution: Department for Urology, University Hospital Regensburg, Caritas Krankenhaus St. Josef, Landshuter Str. 65, Regensburg, Bavaria, 93053, Germany, 49 94…. Major MEDLINE topics: Delphi Technique, Penile Neoplasms, Practice Guidelines as Topic. 101 references. The full text is freely available in PubMed Central.

Auf einen BlickResumoSubjectsDados bibliográficosCiteRelated studies

Abstract (original language)

Background

Clinical practice guidelines are central instruments of evidence-based care; yet, in many domains, particularly rare or complex diseases, multiple guidelines coexist and frequently offer conflicting advice. Existing strategies for resolving such divergences are often labor-intensive, opaque, and insufficiently reproducible.

Objective

This tutorial aimed to develop and demonstrate a reproducible workflow for transforming divergent clinical guideline recommendations and literature-derived white areas into human-reviewed Delphi-ready candidate items.

Methods

Following a step-by-step tutorial structure, we demonstrate the workflow using penile cancer as a worked example. The process comprises manual selection of clinical guidelines, a systematic PubMed search for literature-derived candidate issues potentially unaddressed by the selected guidelines, selection of 3 complementary large language models (LLMs), and execution of predefined prompts, multimodal recommendation extraction, semantic harmonization, and item-level guideline stance classification. The evaluative process combines a 3-LLM ensemble consistency check with an independent pre-Delphi appraisal by 4 penile cancer experts using a predefined 5-point suitability scale and acceptance threshold.

Results

In the worked example in the field of penile cancer as a high-stakes use case, the workflow extracted 157 clinical recommendations from 5 guidelines and consolidated them into 68 harmonized guideline-derived items. Stance classification identified 15 uniformly supported items and 53 divergent, ambiguous, or incompletely addressed items. In parallel, literature processing generated 95 harmonized white-area items, of which 9 were classified as true white-area items not addressed by the included guidelines. The final methodological output was a set of 62 human-reviewed Delphi-ready candidate items for subsequent clinical consensus voting.

Conclusions

This methodology provides a scalable, transparent, and transferable approach for identifying and structuring interguideline inconsistencies and evidence gaps. By combining LLM-assisted analysis with independent external expert appraisal, it supports the reproducible transformation of heterogeneous guideline content into Delphi-ready candidate items.

Subjects

MeSH terms (MEDLINE)

Delphi TechniquePenile NeoplasmsPractice Guidelines as TopicMaleHumansLarge Language Models

Author keywords

Delphi methodartificial intelligenceclinical practice guidelinesconsensus methodologyevidence harmonizationlarge language modelsrare diseasesreproducibilitywhite area analysis

Authors and institutions

  • Department for Urology, University Hospital Regensburg, Caritas Krankenhaus St. Josef, Landshuter Str. 65, Regensburg, Bavaria, 93053, Germany, 49 941782 ext 3506.
  • Department of Urology, University Hospital Augsburg, Augsburg, Bavaria, Germany.
  • Department of Urology, Barmherzige Brüder Klinikum St. Elisabeth in Straubing, Straubing, Bavaria, Germany.
  • Department of Epidemiology and Preventive Medicine, Medical Sociology, University of Regensburg, Regensburg, Bavaria, Germany.
  • Department of Urology, Ludwig-Maximilians-Universität München, Munich, Bavaria, Germany.
  • Faculty of Information Science, University of Regensburg, Regensburg, Germany.

Cite

APA

Haas, M., Rinderknecht, E., May, M., Burger, M., Gilfrich, C., Herrmann, A., Brookman-May, S., Eckl, C. A., Kruschwitz, U., Ateia, S., & Donabauer, G. (2026). From Divergent Guidelines to Delphi-Ready Items: Tutorial for Large Language Model–Assisted Guideline Harmonization Using Penile Cancer as a Worked Example. Journal of Medical Internet Research, 28, e96886-e96886. https://doi.org/10.2196/96886

Vancouver

Haas M, Rinderknecht E, May M, Burger M, Gilfrich C, Herrmann A, et al. From Divergent Guidelines to Delphi-Ready Items: Tutorial for Large Language Model–Assisted Guideline Harmonization Using Penile Cancer as a Worked Example. Journal of Medical Internet Research. 2026;28:e96886-e96886. doi:10.2196/96886

BibTeX
@article{Haas2026,
  author = {Haas, Maximilian and Rinderknecht, Emily and May, Matthias and Burger, Maximilian and Gilfrich, Christian and Herrmann, Anne and Brookman-May, Sabine and Eckl, Christoph Alexander and Kruschwitz, Udo and Ateia, Samy and Donabauer, Gregor},
  title = {From Divergent Guidelines to Delphi-Ready Items: Tutorial for Large Language Model–Assisted Guideline Harmonization Using Penile Cancer as a Worked Example},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e96886--e96886},
  doi = {10.2196/96886}
}
Start for free →