Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies
ACM Transactions on the Web · 31.08.2025 · 19 (3) · 1-29
DOI: 10.1145/3732294 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on the Web, online erschienen am 22.08.2025. Verfasst von Chirag Shah und 14 weiteren Autor:innen. Die Arbeit wurde bisher 29-mal zitiert (Crossref) und nennt 78 Literaturangaben.
Abstract (Originalsprache)
Understanding user intents in information access scenarios can help us provide more relevant and personalized search results and recommendations. However, analyzing user intents is not easy, especially for emerging forms of Web search such as Artificial Intelligence (AI)-driven chat. To understand user intents from retrospective log data, we need a way to label them with meaningful categories that capture their diversity and dynamics. Existing methods rely on manual or Machine-Learned (ML) labeling, which is either expensive or inflexible for large and dynamic datasets. Large Language Models (LLMs) could generate rich and relevant concepts, descriptions, and examples for user intents using log data of user interactions. However, using LLMs to generate a user intent taxonomy and applying it for a given Information Retrieval (IR) application can be problematic for two main reasons: (1) such a taxonomy is not externally validated; and (2) there may be an undesirable feedback loop if an LLM does both these tasks without external validation. To address this, we propose a new methodology with human experts and assessors to verify the quality of the LLM-generated taxonomy. We also present an end-to-end pipeline that uses an LLM with Human-in-the-Loop (HITL) to produce, refine, and apply labels for user intent analysis in log data. We demonstrate its effectiveness by uncovering new insights into user intents from search and chat logs from the Microsoft Bing Web search engine. The novelty in this research stems from the method for generating purpose-driven user intent taxonomies with strong validation. Our approach not only helps remove methodological and practical bottlenecks from intent-focused research, but also provides a new framework for generating, validating, and applying other kinds of taxonomies in a scalable and adaptable way, with reasonable human effort.
Zitieren
APA
Shah, C., White, R., Andersen, R., Buscher, G., Counts, S., Das, S., Montazer, A., Manivannan, S., Neville, J., Rangan, N., Safavi, T., Suri, S., Wan, M., Wang, L., & Yang, L. (2025). Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies. ACM Transactions on the Web, 19(3), 1-29. https://doi.org/10.1145/3732294
Vancouver
Shah C, White R, Andersen R, Buscher G, Counts S, Das S, et al. Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies. ACM Transactions on the Web. 2025;19(3):1-29. doi:10.1145/3732294
BibTeX
@article{Shah2025,
author = {Shah, Chirag and White, Ryen and Andersen, Reid and Buscher, Georg and Counts, Scott and Das, Sarkar and Montazer, Ali and Manivannan, Sathish and Neville, Jennifer and Rangan, Nagu and Safavi, Tara and Suri, Siddharth and Wan, Mengting and Wang, Leijie and Yang, Longqi},
title = {Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies},
journal = {ACM Transactions on the Web},
year = {2025},
volume = {19},
number = {3},
pages = {1--29},
doi = {10.1145/3732294}
}
EUVIMED