ProvLevel-Reason: An Oracle-Scored Benchmark for LLM Reasoning over Software Build-Provenance Attestations
International Journal of Information and Communication Technology Trends · 25.06.2026 · 6 (1) · 269-276
DOI: 10.66525/ijictt95 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Information and Communication Technology Trends, online erschienen am 25.06.2026. Verfasst von Haoyu Zhang und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0).
Abstract (Originalsprache)
A build-provenance attestation is the document a consumer reads to decide whether an artifact was produced by a trustworthy build: it records who built it, from what sources, on what platform, and whether the record is signed. Adjudicating it means applying a laddered rubric, the SLSA build levels, to a structured in-toto/DSSE document. As language models increasingly become the readers of this metadata, the question is whether they apply that rubric or merely pattern-match on its surface. We introduce ProvLevel-Reason, a controlled benchmark that gives a model one real-format attestation and asks it to (A) decide the SLSA build level achieved (L0–L3) and (B) given a target level, enumerate exactly which requirements are unmet and check three provenance-to-artifact consistency properties. Answers are scored against a deterministic oracle that encodes the SLSA v1.1 Build Track plus in-toto/DSSE structural checks, with zero human labels; a generator plants controlled failure modes and a round-trip self-test agrees with the oracle on samples. Across five API models on Task A ( items) and four on Task B ( items, stratified by number of unmet requirements) we find three effects. First, multi-failure attribution is the hard axis: when two requirements are unmet, direct-prompted models report only one of the two – of the time. Second, chain-of-thought cuts both ways, and which way depends on the model. It rescues multi-failure recall for most models, yet on gpt-4o-mini it collapses precision on already-satisfied targets, inventing gaps on of them (exact-match ). Third, a structural completeness check is the consistent weak point, and how stubbornly it persists depends on the provider. When judging level, both OpenAI models and gemini-2.5-flash fail to treat a missing buildType as disqualifying even under CoT (accuracy ), while Anthropic models recover it; asked instead to enumerate gaps, OpenAI recovers but gemini-2.5-flash still does not (incomplete-recall ), a blindness that survives both scale and task framing. With parse_fail and a total inference cost of $14, ProvLevel-Reason turns “is this build trustworthy, and if not, why” into an oracle-scored test.
Zitieren
APA
Zhang, H., & Wang, Z. (2026). ProvLevel-Reason: An Oracle-Scored Benchmark for LLM Reasoning over Software Build-Provenance Attestations. International Journal of Information and Communication Technology Trends, 6(1), 269-276. https://doi.org/10.66525/ijictt95
Vancouver
Zhang H, Wang Z. ProvLevel-Reason: An Oracle-Scored Benchmark for LLM Reasoning over Software Build-Provenance Attestations. International Journal of Information and Communication Technology Trends. 2026;6(1):269-276. doi:10.66525/ijictt95
BibTeX
@article{Zhang2026,
author = {Zhang, Haoyu and Wang, Zhuoxi},
title = {ProvLevel-Reason: An Oracle-Scored Benchmark for LLM Reasoning over Software Build-Provenance Attestations},
journal = {International Journal of Information and Communication Technology Trends},
year = {2026},
volume = {6},
number = {1},
pages = {269--276},
doi = {10.66525/ijictt95}
}
EUVIMED