Bigger Is Not Always Better: Controlled Evaluation of Embedding Compression for Cybersecurity Question Answering
IEEE Access · 2026 · 14 · 112939-112958
DOI: 10.1109/access.2026.3714239 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Andre Abreu Moreno und 1 weiteren Autor:innen. Die Arbeit nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Brazilian Funding Authority for Studies and Projects (Financiadora de Estudos e Projetos—Finep), under the Cyber Semantics Project, Brazilian Army Systems Development Center; and by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior—Brasil (CAPES)—Finance Code 001.
Abstract (Originalsprache)
The rapid growth of unstructured data has increased the memory demands of dense semantic retrieval systems, where high-dimensional embeddings often dominate storage cost. This scenario reinforces the need for storage-efficient retrieval architectures capable of preserving semantic effectiveness while reducing the memory footprint of embedding indexes. In this context, this work investigates how far embedding representations can be compressed while preserving retrieval effectiveness in cybersecurity question answering. We systematically evaluate dimensionality reduction with principal component analysis (PCA) and vector quantization strategies—scalar quantization (SQ), product quantization (PQ), and optimized product quantization (OPQ)—for exhaustive dense retrieval using Faiss on AttackQA and CybersecurityQAA. Four embedding models, all-MiniLM-L6-v2, e5-large-v2, Qwen3-Embedding-0.6B, and bge-large-en-v1.5, are compared under a coverage-controlled protocol using common-success query subsets. Degradation tolerances of 1% and 10% are computed relative to the best reference effectiveness obtained for each encoder, dataset, metric, and query subset. Retrieval effectiveness is measured using $\mathrm {Success}\text{@}K$ for $K \in \{1,10\}$ , alongside serialized index size and query latency. These findings indicate that coordinated dimensionality reduction and quantization are promising strategies for memory-constrained dense retrieval, supporting practical trade-offs between effectiveness, index size, and latency in cybersecurity question answering. In particular, the results show that PCA provides the main structural compression mechanism, reducing serialized index size by up to 94.5% with less than 1% degradation in $\mathrm {Success}\text{@}10$ , and by up to 97.7% under the 10% degradation tolerance. When PCA is combined with quantization, the memory footprint can be further reduced, with admissible configurations exceeding 99% index-size reduction even under the stricter degradation budget, and reaching up to approximately 99.8% reduction under more permissive settings. Finally, under stricter degradation budgets, scalar quantization provides stable additional compression after PCA, whereas product-based methods become more attractive in more permissive regimes, especially under $\mathrm {Success}\text{@}10$ .
Förderung
- Brazilian Funding Authority for Studies and Projects (Financiadora de Estudos e Projetos—Finep), under the Cyber Semantics Project
- Brazilian Army Systems Development Center; and by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior—Brasil (CAPES)—Finance Code 001
Zitieren
APA
Moreno, A. A., & Forster, C. H. Q. (2026). Bigger Is Not Always Better: Controlled Evaluation of Embedding Compression for Cybersecurity Question Answering. IEEE Access, 14, 112939-112958. https://doi.org/10.1109/access.2026.3714239
Vancouver
Moreno AA, Forster CHQ. Bigger Is Not Always Better: Controlled Evaluation of Embedding Compression for Cybersecurity Question Answering. IEEE Access. 2026;14:112939-112958. doi:10.1109/access.2026.3714239
BibTeX
@article{Moreno2026,
author = {Moreno, Andre Abreu and Forster, Carlos Henrique Quartucci},
title = {Bigger Is Not Always Better: Controlled Evaluation of Embedding Compression for Cybersecurity Question Answering},
journal = {IEEE Access},
year = {2026},
volume = {14},
pages = {112939--112958},
doi = {10.1109/access.2026.3714239}
}
EUVIMED