From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI
ACM SIGCOMM Computer Communication Review · 01.07.2026 · 56 (2) · 29-41
DOI: 10.1145/3831956.3831960 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM SIGCOMM Computer Communication Review, online erschienen am 17.08.2026. Verfasst von Felix Hoffmann und 3 weiteren Autor:innen. Die Arbeit nennt 32 Literaturangaben.
Abstract (Originalsprache)
We present a large-scale measurement of emerging Artificial Intelligence (AI)-oriented web permission and descriptor mechanisms across approximately 4M domains. Beyond traditional robots.txt files, we analyze the adoption, structure, and interactions of newer files, including the permission-oriented ai.txt and the descriptor-oriented llms.txt and llms-full.txt. While robots.txt remains the dominant mechanism for signaling crawler permissions, adoption of AI-specific permission files is emerging, especially among technology-focused domains. Directive usage shows a strong emphasis on restricting AI agents, with prevalent disallow rules and relatively few AI-specific permission controls. Crossfile comparisons reveal that conflicts between robots.txt and ai.txt are infrequent, while some llms.txt link targets point to content blocked by access-control files.
Zitieren
APA
Hoffmann, F., Goergens, T., Khosla, S., & Bajpai, V. (2026). From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI. ACM SIGCOMM Computer Communication Review, 56(2), 29-41. https://doi.org/10.1145/3831956.3831960
Vancouver
Hoffmann F, Goergens T, Khosla S, Bajpai V. From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI. ACM SIGCOMM Computer Communication Review. 2026;56(2):29-41. doi:10.1145/3831956.3831960
BibTeX
@article{Hoffmann2026,
author = {Hoffmann, Felix and Goergens, Tobias and Khosla, Sonal and Bajpai, Vaibhav},
title = {From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI},
journal = {ACM SIGCOMM Computer Communication Review},
year = {2026},
volume = {56},
number = {2},
pages = {29--41},
doi = {10.1145/3831956.3831960}
}
EUVIMED