EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Metamorphic Testing for Semantic Invariance in Large Language Models

J. De Curtò, I. De Zarzà

IEEE Access · 2025 · 13 · 214772-214791

DOI: 10.1109/access.2025.3646270 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2025. Verfasst von J. De Curtò und 1 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Luxembourg Institute of Science and Technology (LIST) through the Projects “ADIALab-MAST” and “LLMs4E;”, LLMs4EU Project is co-funded by the European Union through the Digital Europe Program, Barcelona Supercomputing Center through the Project “TIFON;”.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) have demonstrated remarkable capabilities in mathematical and scientific reasoning tasks, yet standard benchmark evaluations fail to capture the stability of these reasoning processes under semantically equivalent input variations. We introduce a metamorphic testing framework for assessing semantic invariance in LLM reasoning, systematically evaluating model responses across eight metamorphic relations: identity, paraphrase, fact reordering, expansion, contraction, academic context, business context, and contrastive formulation. We conduct a comparative analysis of two recent foundation models—Hermes-4-70B and DeepSeek-R1-0528—across 79 reasoning problems spanning eight categories (Physics, Mathematics, Chemistry, Economics, Statistics, Biology, Calculus, and Optimization) at three difficulty levels. Both models exhibit similar aggregate invariance scores (~0.90), yet manifest distinct vulnerability profiles–DeepSeek shows notable sensitivity to paraphrase and fact-reordering transformations, particularly in formal domains (Calculus: -0.104, Statistics: -0.048). Contrastive transformations prove particularly challenging for Hermes, inducing mean score deltas of -0.147 (Hermes) and -0.005 (DeepSeek). Semantic similarity analysis further reveals that Hermes maintains more coherent reasoning traces (mean similarity: 0.87) compared to DeepSeek (0.78), with DeepSeek exhibiting near-complete reasoning breakdown in specific category-transformation combinations. These findings demonstrate that metamorphic testing uncovers robustness characteristics invisible to conventional accuracy metrics, providing essential insights for deploying LLMs in high-stakes reasoning applications. We release our evaluation framework and complete experimental data to facilitate reproducible robustness assessment of foundation models.

Förderung

  • Luxembourg Institute of Science and Technology (LIST) through the Projects “ADIALab-MAST” and “LLMs4E;”
  • LLMs4EU Project is co-funded by the European Union through the Digital Europe Program
  • Barcelona Supercomputing Center through the Project “TIFON;”
  • Centre for the Development of Industrial Technology (CDTI) with the Support of the Spanish Ministry of Science and Innovation
  • Agencia Estatal de Investigación

Zitieren

APA

Curtò, J. D., & Zarzà, I. D. (2025). Metamorphic Testing for Semantic Invariance in Large Language Models. IEEE Access, 13, 214772-214791. https://doi.org/10.1109/access.2025.3646270

Vancouver

Curtò JD, Zarzà ID. Metamorphic Testing for Semantic Invariance in Large Language Models. IEEE Access. 2025;13:214772-214791. doi:10.1109/access.2025.3646270

BibTeX
@article{Curt2025,
  author = {Curtò, J. De and Zarzà, I. De},
  title = {Metamorphic Testing for Semantic Invariance in Large Language Models},
  journal = {IEEE Access},
  year = {2025},
  volume = {13},
  pages = {214772--214791},
  doi = {10.1109/access.2025.3646270}
}
Kostenlos starten →