EUVIMEDEUVIMED entdecken
Crossref

AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems

Zirui Wang, Guangba Yu, Michael R. Lyu

ACM Transactions on Software Engineering and Methodology · 06.07.2026

DOI: 10.1145/3821576 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 06.07.2026. Verfasst von Zirui Wang und 2 weiteren Autor:innen. Die Arbeit nennt 77 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The transition from Cloud-Native to AI-Native architectures is fundamentally reshaping software engineering, replacing deterministic microservices with probabilistic agentic services. However, this shift renders traditional black-box evaluation paradigms insufficient: existing benchmarks measure raw model capabilities while remaining blind to system-level execution dynamics. To bridge this gap, we introduce AI-NativeBench, the first application-centric and white-box AI-Native benchmark suite grounded in Model Context Protocol (MCP) and Agent-to-Agent (A2A) standards. By treating agentic spans as first-class citizens within distributed traces, our methodology enables granular analysis of engineering characteristics beyond simple capabilities. Leveraging this benchmark across 21 system variants, we uncover critical engineering realities invisible to traditional metrics: a parameter paradox where lightweight models often surpass flagships in protocol adherence, a pervasive inference dominance that renders protocol overhead secondary, and an expensive failure pattern where self-healing mechanisms paradoxically act as cost multipliers on unviable workflows. This work provides the first systematic evidence to guide the transition from measuring model capability to engineering reliable AI-Native systems. To facilitate reproducibility and further research, we have open-sourced the benchmark and dataset.

Zitieren

APA

Wang, Z., Yu, G., & Lyu, M. R. (2026). AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3821576

Vancouver

Wang Z, Yu G, Lyu MR. AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems. ACM Transactions on Software Engineering and Methodology. 2026. doi:10.1145/3821576

BibTeX
@article{Wang2026,
  author = {Wang, Zirui and Yu, Guangba and Lyu, Michael R.},
  title = {AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2026},
  doi = {10.1145/3821576}
}
Kostenlos starten →