EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-SA 4.0

ProvLevel-Reason: An Oracle-Scored Benchmark for LLM Reasoning over Software Build-Provenance Attestations

Haoyu Zhang, Zhuoxi Wang

International Journal of Information and Communication Technology Trends · 25.06.2026 · 6 (1) · 269-276

DOI: 10.66525/ijictt95 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Information and Communication Technology Trends, online erschienen am 25.06.2026. Verfasst von Haoyu Zhang und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

A build-provenance attestation is the document a consumer reads to decide whether an artifact was produced by a trustworthy build: it records who built it, from what sources, on what platform, and whether the record is signed. Adjudicating it means applying a laddered rubric, the SLSA build levels, to a structured in-toto/DSSE document. As language models increasingly become the readers of this metadata, the question is whether they apply that rubric or merely pattern-match on its surface. We introduce ProvLevel-Reason, a controlled benchmark that gives a model one real-format attestation and asks it to (A) decide the SLSA build level achieved (L0–L3) and (B) given a target level, enumerate exactly which requirements are unmet and check three provenance-to-artifact consistency properties. Answers are scored against a deterministic oracle that encodes the SLSA v1.1 Build Track plus in-toto/DSSE structural checks, with zero human labels; a generator plants controlled failure modes and a round-trip self-test agrees with the oracle on samples. Across five API models on Task A ( items) and four on Task B ( items, stratified by number of unmet requirements) we find three effects. First, multi-failure attribution is the hard axis: when two requirements are unmet, direct-prompted models report only one of the two – of the time. Second, chain-of-thought cuts both ways, and which way depends on the model. It rescues multi-failure recall for most models, yet on gpt-4o-mini it collapses precision on already-satisfied targets, inventing gaps on of them (exact-match ). Third, a structural completeness check is the consistent weak point, and how stubbornly it persists depends on the provider. When judging level, both OpenAI models and gemini-2.5-flash fail to treat a missing buildType as disqualifying even under CoT (accuracy ), while Anthropic models recover it; asked instead to enumerate gaps, OpenAI recovers but gemini-2.5-flash still does not (incomplete-recall ), a blindness that survives both scale and task framing. With parse_fail and a total inference cost of $14, ProvLevel-Reason turns “is this build trustworthy, and if not, why” into an oracle-scored test.

Zitieren

APA

Zhang, H., & Wang, Z. (2026). ProvLevel-Reason: An Oracle-Scored Benchmark for LLM Reasoning over Software Build-Provenance Attestations. International Journal of Information and Communication Technology Trends, 6(1), 269-276. https://doi.org/10.66525/ijictt95

Vancouver

Zhang H, Wang Z. ProvLevel-Reason: An Oracle-Scored Benchmark for LLM Reasoning over Software Build-Provenance Attestations. International Journal of Information and Communication Technology Trends. 2026;6(1):269-276. doi:10.66525/ijictt95

BibTeX
@article{Zhang2026,
  author = {Zhang, Haoyu and Wang, Zhuoxi},
  title = {ProvLevel-Reason: An Oracle-Scored Benchmark for LLM Reasoning over Software Build-Provenance Attestations},
  journal = {International Journal of Information and Communication Technology Trends},
  year = {2026},
  volume = {6},
  number = {1},
  pages = {269--276},
  doi = {10.66525/ijictt95}
}
Kostenlos starten →