EUVIMEDEUVIMED entdecken
Crossref

Beyond Neuron-Level Sparsity: Achieving Faithful and Interpretable LLMs with Mixture of Decoders

Grigorios Chrysos

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (47) · 39811-39812

DOI: 10.1609/aaai.v40i47.41340 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Grigorios Chrysos.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

As large language models (LLMs) scale, ensuring interpretability and privacy becomes critical. This talk addresses these interconnected challenges with novel approaches to model specialization and safety. First, we tackle the dense, distributed nature of LLM representations by casting Mixture-of-Experts (MoE) as a tensor decomposition, enabling specialized experts in a factorized space. Second, we argue that current neuron-level sparsity methods create a severe accuracy-sparsity trade-off, and we propose a paradigm shift to layer-level sparsity with the Mixture of Decoders (MxD). We explain how MxD uses tensor factorization to expand dense layers into thousands of specialized, full-rank sublayers, demonstrating how it significantly outperforms alternatives in preserving model faithfulness and performance across LLMs up to 3B parameters. Finally, we address privacy in open-weight models by proposing a scalable and certifiable algorithm that induces maximal uncertainty on protected instances, proving tight bounds that characterize the resulting privacy-utility tradeoff.

Zitieren

APA

Chrysos, G. (2026). Beyond Neuron-Level Sparsity: Achieving Faithful and Interpretable LLMs with Mixture of Decoders. Proceedings of the AAAI Conference on Artificial Intelligence, 40(47), 39811-39812. https://doi.org/10.1609/aaai.v40i47.41340

Vancouver

Chrysos G. Beyond Neuron-Level Sparsity: Achieving Faithful and Interpretable LLMs with Mixture of Decoders. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(47):39811-39812. doi:10.1609/aaai.v40i47.41340

BibTeX
@article{Chrysos2026,
  author = {Chrysos, Grigorios},
  title = {Beyond Neuron-Level Sparsity: Achieving Faithful and Interpretable LLMs with Mixture of Decoders},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {47},
  pages = {39811--39812},
  doi = {10.1609/aaai.v40i47.41340}
}
Kostenlos starten →