EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Negative Contrastive Chain-of-Thought Distillation for Transfer Reasoning Capability to Small Language Models

Jae-Wook Han, Hayoung Jo, Jung-Ho Hong, Seong-Whan Lee

IEEE Access · 2026 · 14 · 118580-118590

DOI: 10.1109/access.2026.3713532 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Jae-Wook Han und 3 weiteren Autor:innen. Die Arbeit nennt 60 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Institute of Information & Communications Technology Planning & Evaluation (IITP) funded by the Korea government.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) demonstrate strong reasoning capabilities through chain-of-thought prompting. Recently, numerous studies have focused on transferring reasoning capability from LLMs to small language models (SLMs) through chain-of-thought (CoT) distillation. However, these approaches face two limitations. First, simplistic multiple iterations to augment correct predictions are ineffective, due to the LLM tends to reproduce the same errors. Second, directly utilizing incorrect predictions during the training process exposes the model to learning incorrect predictions of the LLM during distillation. To address these problems, we propose a novel framework named negative contrastive CoT distillation (NCD), which efficiently augments correct predictions while training an SLM to exclude incorrect predictions. NCD comprises of two stages: corrective reviewer augmentation (CRA) and negative exclusive learning (NEL). CRA utilizes the self-correction capability of LLM to augment the correct predictions within a single iteration. NEL is a distillation strategy that strengthens the ability of SLM to mimic the correct predictions of LLM by leveraging the incorrect predictions of LLM. The experimental results show that NCD improves the reasoning capability of the SLM by a sizable gap in diverse reasoning tasks, including mathematical and commonsense reasoning tasks. We demonstrate both the efficiency of CRA and the effectiveness of NEL.

Förderung

  • Institute of Information & Communications Technology Planning & Evaluation (IITP) funded by the Korea government

Zitieren

APA

Han, J., Jo, H., Hong, J., & Lee, S. (2026). Negative Contrastive Chain-of-Thought Distillation for Transfer Reasoning Capability to Small Language Models. IEEE Access, 14, 118580-118590. https://doi.org/10.1109/access.2026.3713532

Vancouver

Han J, Jo H, Hong J, Lee S. Negative Contrastive Chain-of-Thought Distillation for Transfer Reasoning Capability to Small Language Models. IEEE Access. 2026;14:118580-118590. doi:10.1109/access.2026.3713532

BibTeX
@article{Han2026,
  author = {Han, Jae-Wook and Jo, Hayoung and Hong, Jung-Ho and Lee, Seong-Whan},
  title = {Negative Contrastive Chain-of-Thought Distillation for Transfer Reasoning Capability to Small Language Models},
  journal = {IEEE Access},
  year = {2026},
  volume = {14},
  pages = {118580--118590},
  doi = {10.1109/access.2026.3713532}
}
Kostenlos starten →