EUVIMEDEUVIMED entdecken
Crossref

Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations

Sina Salimian, Gias Uddin, Sumon Biswas, Henry Leung

ACM Transactions on Software Engineering and Methodology · 17.08.2026

DOI: 10.1145/3841176 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 17.08.2026. Verfasst von Sina Salimian und 3 weiteren Autor:innen. Die Arbeit nennt 66 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The widespread deployment of Large Language Models (LLMs) has intensified concerns about subtle social biases embedded in their outputs. Existing guardrails often fail when faced with indirect or contextually complex bias-inducing prompts. To address these limitations, we propose a unified framework for both systematic bias evaluation and targeted mitigation. Our approach introduces six novel Metamorphic Relations (MRs) that, based on metamorphic testing principles, transform direct bias-inducing inputs into semantically equivalent yet adversarially challenging variants. These transformations enable an automated method for exposing hidden model biases. When an LLM responds inconsistently or unfairly across MR-generated variants, the underlying bias becomes detectable. We further show that the same MRs can be used to generate diverse bias-inducing samples for fine-tuning, directly linking the testing process to mitigation. Using seven state-of-the-art LLMs, spanning open-source and proprietary models, and a representative subset of 385 questions from the 8,978-item BiasAsker benchmark covering seven protected groups, our MRs reduce bias resiliency by up to 27 percentage points relative to base prompts, exposing hidden biases that existing tools miss. Moreover, fine-tuning with both original and MR-mutated samples significantly enhances bias resiliency, increasing safe response rates to 87.6–88.9% across models (from base rates as low as 54.7%). These results highlight metamorphic relations as a practical mechanism for improving fairness in conversational AI.

Zitieren

APA

Salimian, S., Uddin, G., Biswas, S., & Leung, H. (2026). Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3841176

Vancouver

Salimian S, Uddin G, Biswas S, Leung H. Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations. ACM Transactions on Software Engineering and Methodology. 2026. doi:10.1145/3841176

BibTeX
@article{Salimian2026,
  author = {Salimian, Sina and Uddin, Gias and Biswas, Sumon and Leung, Henry},
  title = {Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2026},
  doi = {10.1145/3841176}
}
Kostenlos starten →