When Retrieval Does Not Help: Benchmarking Large Language Models on Regret and Disappointment in Customer Reviews
Analytics · 2026-11-20 · 5 (4) · 55
DOI: 10.3390/analytics5040055 ↗At a glance
Journal article in Analytics, published online on 2026-11-20. Written by Sarthak Pattnaik and 2 co-authors. 46 references. The full text is freely available (open access, CC BY 4.0).
Abstract (original language)
Regret follows from a person’s own choice; disappointment from circumstances outside their control. We benchmark large language model (LLM) configurations on separating the two in 2019 low-rated Amazon product reviews. Two independent human annotators labeled 385 reviews (Cohen’s κ=0.80): 200 drawn at random and 185 drawn where systems label regret. Unweighted scores describe this enriched set; stratum weights give pool estimates. We compare GPT-4o with and without theory-based definitions and retrieval-augmented generation (RAG) across the regret literature, an adapted Speculative RAG pipeline, our original pipeline, a regret-cue lexicon, and a supervised TF-IDF classifier, which rarely detected regret (F1 = 0.12). On the 372 reviews with an agreed label, GPT-4o without retrieval agreed best with the annotators (κ=0.73 with label names, 0.72 with definitions; F1 for regret 0.75, or 0.73 weighted to the pool). Adding retrieval to the definitions lowered agreement to κ=0.45 (McNemar p=1.9×10−8), mainly by labeling product complaints as regret. Speculative RAG with a GPT-4o-mini drafter (κ=0.20) and the original pipeline (κ=0.16) did worst. Asked to choose between the two emotions, the annotators placed 2.0% of random reviews on the regret side; the LLM systems’ estimates on the same reviews ranged from 1.5% to 21.5%. For this distinction, a plain GPT-4o prompt was more accurate than retrieving the regret literature and used under a quarter of the input tokens.
Cite
APA
Pattnaik, S., Jain, C., & Pinsky, E. (2026). When Retrieval Does Not Help: Benchmarking Large Language Models on Regret and Disappointment in Customer Reviews. Analytics, 5(4), 55. https://doi.org/10.3390/analytics5040055
Vancouver
Pattnaik S, Jain C, Pinsky E. When Retrieval Does Not Help: Benchmarking Large Language Models on Regret and Disappointment in Customer Reviews. Analytics. 2026;5(4):55. doi:10.3390/analytics5040055
BibTeX
@article{Pattnaik2026,
author = {Pattnaik, Sarthak and Jain, Chhayank and Pinsky, Eugene},
title = {When Retrieval Does Not Help: Benchmarking Large Language Models on Regret and Disappointment in Customer Reviews},
journal = {Analytics},
year = {2026},
volume = {5},
number = {4},
pages = {55},
doi = {10.3390/analytics5040055}
}
EUVIMED