EUVIMEDEUVIMED entdecken
Crossref

From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI

Felix Hoffmann, Tobias Goergens, Sonal Khosla, Vaibhav Bajpai

ACM SIGCOMM Computer Communication Review · 01.07.2026 · 56 (2) · 29-41

DOI: 10.1145/3831956.3831960 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM SIGCOMM Computer Communication Review, online erschienen am 17.08.2026. Verfasst von Felix Hoffmann und 3 weiteren Autor:innen. Die Arbeit nennt 32 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We present a large-scale measurement of emerging Artificial Intelligence (AI)-oriented web permission and descriptor mechanisms across approximately 4M domains. Beyond traditional robots.txt files, we analyze the adoption, structure, and interactions of newer files, including the permission-oriented ai.txt and the descriptor-oriented llms.txt and llms-full.txt. While robots.txt remains the dominant mechanism for signaling crawler permissions, adoption of AI-specific permission files is emerging, especially among technology-focused domains. Directive usage shows a strong emphasis on restricting AI agents, with prevalent disallow rules and relatively few AI-specific permission controls. Crossfile comparisons reveal that conflicts between robots.txt and ai.txt are infrequent, while some llms.txt link targets point to content blocked by access-control files.

Zitieren

APA

Hoffmann, F., Goergens, T., Khosla, S., & Bajpai, V. (2026). From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI. ACM SIGCOMM Computer Communication Review, 56(2), 29-41. https://doi.org/10.1145/3831956.3831960

Vancouver

Hoffmann F, Goergens T, Khosla S, Bajpai V. From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI. ACM SIGCOMM Computer Communication Review. 2026;56(2):29-41. doi:10.1145/3831956.3831960

BibTeX
@article{Hoffmann2026,
  author = {Hoffmann, Felix and Goergens, Tobias and Khosla, Sonal and Bajpai, Vaibhav},
  title = {From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI},
  journal = {ACM SIGCOMM Computer Communication Review},
  year = {2026},
  volume = {56},
  number = {2},
  pages = {29--41},
  doi = {10.1145/3831956.3831960}
}
Kostenlos starten →