EUVIMEDOdkrijte EUVIMED
Open access · CC BY 4.0

When Retrieval Does Not Help: Benchmarking Large Language Models on Regret and Disappointment in Customer Reviews

Sarthak Pattnaik, Chhayank Jain, Eugene Pinsky

Analytics · 2026-11-20 · 5 (4) · 55

DOI: 10.3390/analytics5040055 ↗

At a glance

Journal article in Analytics, published online on 2026-11-20. Written by Sarthak Pattnaik and 2 co-authors. 46 references. The full text is freely available (open access, CC BY 4.0).

PovzetekBibliografski podatkiCiteRelated studies

Abstract (original language)

Regret follows from a person’s own choice; disappointment from circumstances outside their control. We benchmark large language model (LLM) configurations on separating the two in 2019 low-rated Amazon product reviews. Two independent human annotators labeled 385 reviews (Cohen’s κ=0.80): 200 drawn at random and 185 drawn where systems label regret. Unweighted scores describe this enriched set; stratum weights give pool estimates. We compare GPT-4o with and without theory-based definitions and retrieval-augmented generation (RAG) across the regret literature, an adapted Speculative RAG pipeline, our original pipeline, a regret-cue lexicon, and a supervised TF-IDF classifier, which rarely detected regret (F1 = 0.12). On the 372 reviews with an agreed label, GPT-4o without retrieval agreed best with the annotators (κ=0.73 with label names, 0.72 with definitions; F1 for regret 0.75, or 0.73 weighted to the pool). Adding retrieval to the definitions lowered agreement to κ=0.45 (McNemar p=1.9×10−8), mainly by labeling product complaints as regret. Speculative RAG with a GPT-4o-mini drafter (κ=0.20) and the original pipeline (κ=0.16) did worst. Asked to choose between the two emotions, the annotators placed 2.0% of random reviews on the regret side; the LLM systems’ estimates on the same reviews ranged from 1.5% to 21.5%. For this distinction, a plain GPT-4o prompt was more accurate than retrieving the regret literature and used under a quarter of the input tokens.

Cite

APA

Pattnaik, S., Jain, C., & Pinsky, E. (2026). When Retrieval Does Not Help: Benchmarking Large Language Models on Regret and Disappointment in Customer Reviews. Analytics, 5(4), 55. https://doi.org/10.3390/analytics5040055

Vancouver

Pattnaik S, Jain C, Pinsky E. When Retrieval Does Not Help: Benchmarking Large Language Models on Regret and Disappointment in Customer Reviews. Analytics. 2026;5(4):55. doi:10.3390/analytics5040055

BibTeX
@article{Pattnaik2026,
  author = {Pattnaik, Sarthak and Jain, Chhayank and Pinsky, Eugene},
  title = {When Retrieval Does Not Help: Benchmarking Large Language Models on Regret and Disappointment in Customer Reviews},
  journal = {Analytics},
  year = {2026},
  volume = {5},
  number = {4},
  pages = {55},
  doi = {10.3390/analytics5040055}
}
Start for free →