EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Prompt architecture induces methodological artifacts in large language models

Melanie Brucks iD, Olivier Toubia

PLOS One · 28.04.2025 · 20 (4) · e0319159

DOI: 10.1371/journal.pone.0319159 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, online erschienen am 28.04.2025. Verfasst von Melanie Brucks und 1 weiteren Autor:innen. Beteiligte Einrichtung: Columbia Business School, New York, NY, United States of America. Hauptthemen laut MEDLINE: Artifacts, Models, Theoretical. Die Arbeit wurde bisher 45-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We examine how the seemingly arbitrary way a prompt is posed, which we term “prompt architecture,” influences responses provided by large language models (LLMs). Five large-scale, full-factorial experiments performing standard (zero-shot) similarity evaluation tasks using GPT-3, GPT-4, and Llama 3.1 document how several features of prompt architecture (order, label, framing, and justification) interact to produce methodological artifacts, a form of statistical bias. We find robust evidence that these four elements unduly affect responses across all models, and although we observe differences between GPT-3 and GPT-4, the changes are not necessarily for the better. Specifically, LLMs demonstrate both response-order bias and label bias, and framing and justification moderate these biases. We then test different strategies intended to reduce methodological artifacts. Specifying to the LLM that the order and labels of items have been randomized does not alleviate either response-order or label bias, and the use of uncommon labels reduces (but does not eliminate) label bias but exacerbates response-order bias in GPT-4 (and does not reduce either bias in Llama 3.1). By contrast, aggregating across prompts generated using a full factorial design eliminates response-order and label bias. Overall, these findings highlight the inherent fallibility of any individual prompt when using LLMs, as any prompt contains characteristics that may subtly interact with a multitude of hidden associations embedded in rich language data.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

ArtifactsModels, TheoreticalHumansLarge Language Models

Autor:innen und Institutionen

  • Columbia Business School, New York, NY, United States of America.

ORCID-Profile: Melanie Brucks

Zitieren

APA

Brucks, M., & Toubia, O. (2025). Prompt architecture induces methodological artifacts in large language models. PLOS One, 20(4), e0319159. https://doi.org/10.1371/journal.pone.0319159

Vancouver

Brucks M, Toubia O. Prompt architecture induces methodological artifacts in large language models. PLOS One. 2025;20(4):e0319159. doi:10.1371/journal.pone.0319159

BibTeX
@article{Brucks2025,
  author = {Brucks, Melanie and Toubia, Olivier},
  title = {Prompt architecture induces methodological artifacts in large language models},
  journal = {PLOS One},
  year = {2025},
  volume = {20},
  number = {4},
  pages = {e0319159},
  doi = {10.1371/journal.pone.0319159}
}
Kostenlos starten →