EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Self-Consistency Vote Share as a Selective-Classification Signal for Zero-Shot LLM Text Classifiers

Omar N. M. Taher, Hassan A. Jeiad, Firas A. Raheem, Aymen D. Salman, Amjad J. Humaidi

Information · 06.11.2026 · 17 (11) · 1115

DOI: 10.3390/info17111115 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Information, online erschienen am 06.11.2026. Verfasst von Omar N. M. Taher und 4 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Zero-shot large language model (LLM) classifiers are often deployed behind interfaces that expose no token probabilities, so practitioners sample the model several times and use the winning label’s vote share as confidence. We study the simplest selective rule built on this signal: answer only if one label receives more than half of k votes. The answered predictions are exactly those of self-consistency, so the object of study is the abstention decision. On identical items (AG News n = 7600, DBpedia n = 7000, GoEmotions n = 5427; three repeats), we evaluate DeepSeek-R1-Distill-Qwen-7B and Llama-3.2-3B-Instruct at k = 1, 3 and 5 against single-call probability and verbalized-confidence baselines, token-based cost, and risk-controlled and conformal selective prediction; 32B and 70B models are checked on 2000 items. Extra calls lowered calibration error (ECE 22.2% to 14.8% for the 7B model on AG News) and slightly raised accuracy on easy tasks (+0.9 to +2.6 points), but accuracy with abstentions counted as errors fell on GoEmotions (−2.0 points for the 7B model), and cost grows linearly in generated tokens. Vote share was not generally a better error detector than single-call token probabilities, and for a 70B model it was nearly uninformative (AUROC 0.51 to 0.59). At k = 5, voting at temperature 0.01 was at least as accurate as at 0.7 for the small Llama model. The fixed rule gives no risk guarantee; vote share is a fallback for interfaces without probabilities, not a replacement for them.

Zitieren

APA

Taher, O. N. M., Jeiad, H. A., Raheem, F. A., Salman, A. D., & Humaidi, A. J. (2026). Self-Consistency Vote Share as a Selective-Classification Signal for Zero-Shot LLM Text Classifiers. Information, 17(11), 1115. https://doi.org/10.3390/info17111115

Vancouver

Taher ONM, Jeiad HA, Raheem FA, Salman AD, Humaidi AJ. Self-Consistency Vote Share as a Selective-Classification Signal for Zero-Shot LLM Text Classifiers. Information. 2026;17(11):1115. doi:10.3390/info17111115

BibTeX
@article{Taher2026,
  author = {Taher, Omar N. M. and Jeiad, Hassan A. and Raheem, Firas A. and Salman, Aymen D. and Humaidi, Amjad J.},
  title = {Self-Consistency Vote Share as a Selective-Classification Signal for Zero-Shot LLM Text Classifiers},
  journal = {Information},
  year = {2026},
  volume = {17},
  number = {11},
  pages = {1115},
  doi = {10.3390/info17111115}
}
Kostenlos starten →