Evaluating the Use of LLMs for Documentation to Code Traceability
ACM Transactions on Software Engineering and Methodology · 06.06.2026
DOI: 10.1145/3819820 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 06.06.2026. Verfasst von Ebube Alor und 2 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 100 Literaturangaben.
Abstract (Originalsprache)
Large Language Models (LLMs) offer new potential for automating documentation-to-code traceability, yet their capabilities and limitations remain underexplored. We evaluate three LLMs (Claude Sonnet, GPT-4o, and o3-mini) on documentation-to-code traceability across diverse software projects. We create two datasets from open-source systems (Unity Catalog and Crawl4AI), curated to minimize training-data overlap with the evaluated models, and derive a 27-project benchmark from CoDocBench and TransArc to study generalizability across established codebases. Our experiments assess three capabilities: (1) trace link identification, (2) relationship explanation quality, and (3) multi-step chain reconstruction, while comparing LLMs against TF-IDF, BM25, and CodeBERT baselines. On the curated datasets, the best LLM attains F1-scores of 79.4% and 80.4%, and on the Established Projects benchmark, LLMs reach average F1 up to 92.3%, substantially outperforming traditional baselines except in a single dense system. Fully correct explanations cover 42.9%–71.1% of links, while partial correctness exceeds 97%, and chain endpoints are almost always recovered, though intermediate steps are often imperfect. We also show that task framing, particularly one-to-many matching, is crucial for recall, whereas naive many-to-many prompts achieve around 5% F1 due to enumeration bias. Overall, LLMs are promising assistants for trace discovery when combined with human oversight and careful tool design.
Zitieren
APA
Alor, E., Khatoonabadi, S., & Shihab, E. (2026). Evaluating the Use of LLMs for Documentation to Code Traceability. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3819820
Vancouver
Alor E, Khatoonabadi S, Shihab E. Evaluating the Use of LLMs for Documentation to Code Traceability. ACM Transactions on Software Engineering and Methodology. 2026. doi:10.1145/3819820
BibTeX
@article{Alor2026,
author = {Alor, Ebube and Khatoonabadi, SayedHassan and Shihab, Emad},
title = {Evaluating the Use of LLMs for Documentation to Code Traceability},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2026},
doi = {10.1145/3819820}
}
EUVIMED