EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

FISTA-Refined Dictionary Learning with Learned TopK Initialization for Sparse Decomposition of Transformer Residual Streams

Kyung-In Joung, Jang Pyo Bae

Algorithms · 20.08.2026 · 19 (8) · 697

DOI: 10.3390/a19080697 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Algorithms, online erschienen am 20.08.2026. Verfasst von Kyung-In Joung und 1 weiteren Autor:innen. Die Arbeit nennt 26 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Sparse autoencoders (SAEs) decompose transformer activations into sparse feature dictionaries, but one-shot support selection can leave many features unused and become difficult to optimize at very small active budgets. We study an SAE-like dictionary-learning procedure that combines a learned TopK encoder with a few unrolled Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) steps for coefficient refinement. On GPT-2 Small layer-11 residual streams, a 12,288-feature TopK256 model with five FISTA steps achieves ΔCE = 0.0745 and a 3.09% dead-feature ratio, compared with 0.0859 and 50.76% for a matched TopK SAE. Ablations show that initialization and learned TopK feature selection explain most of the reduction in dead features, while FISTA further improves fidelity. In a 32,768-feature regime, direct exact-TopK32 training is unstable. Training becomes feasible by retaining a TopK128 candidate support while reducing the final active budget from TopK128 to TopK64 and then TopK32. Five refinement steps reach ΔCE = 0.2442 and 17.84% dead features after 264 million token presentations. A residual-gain-ranked loss over candidates at ranks 33–48 improves near-boundary feature recruitment with a modest fidelity cost. These results show that lightweight iterative correction can improve fixed-budget sparse decomposition without replacing one-shot SAE inference with a full optimization solver.

Zitieren

APA

Joung, K., & Bae, J. P. (2026). FISTA-Refined Dictionary Learning with Learned TopK Initialization for Sparse Decomposition of Transformer Residual Streams. Algorithms, 19(8), 697. https://doi.org/10.3390/a19080697

Vancouver

Joung K, Bae JP. FISTA-Refined Dictionary Learning with Learned TopK Initialization for Sparse Decomposition of Transformer Residual Streams. Algorithms. 2026;19(8):697. doi:10.3390/a19080697

BibTeX
@article{Joung2026,
  author = {Joung, Kyung-In and Bae, Jang Pyo},
  title = {FISTA-Refined Dictionary Learning with Learned TopK Initialization for Sparse Decomposition of Transformer Residual Streams},
  journal = {Algorithms},
  year = {2026},
  volume = {19},
  number = {8},
  pages = {697},
  doi = {10.3390/a19080697}
}
Kostenlos starten →