Better queries, cheaper attention: adapting transformers for efficient sparse reconstruction
Machine Learning: Science and Technology · 01.10.2026 · 7 (5) · 055052
DOI: 10.1088/2632-2153/aeb3cc ↗Kurz gefasst
Fachartikel in der Zeitschrift Machine Learning: Science and Technology, online erschienen am 29.10.2026. Verfasst von Philippa Duckett und 4 weiteren Autor:innen. Die Arbeit nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Department for Science, Innovation and Technology, Royal Society, UK Research and Innovation.
Abstract (Originalsprache)
Query-based transformer decoders are effective for object reconstruction from sparse scientific sensor measurements, but their scalability to high-multiplicity data is limited by fixed, input-independent query sets and costly decoder cross-attention. We introduce a geometry-aware dynamic-query (DQ) decoder that couples input-conditioned query construction with structured sparse cross-attention. Decoder queries are initialised from selected encoder-level measurement representations that serve as candidate trajectory seeds, making both query content and query multiplicity input-dependent. Local strided cross-attention (LSCA) exploits the induced geometric ordering by replacing learned mask-gated cross-attention with a geometry-defined local support that restricts attention to physically plausible query–hit interactions and exposes sparsity for block-sparse execution. We study this architecture for charged-particle trajectory reconstruction in a simplified High-Luminosity Large Hadron Collider detector, where thousands of trajectories must be reconstructed from tens of thousands of sparse measurements. In the nominal configuration, evaluated for particles with p T ⩾ 1 GeV , the DQ architecture increases trajectory reconstruction efficiency from 94.1 % to 98.1 % and reduces the fake rate by more than a factor of two relative to the fixed-query baseline. The DQ+LSCA model reduces end-to-end inference latency by more than 70 % and peak allocated inference memory by more than a factor of 10 relative to the fixed-query baseline.
Förderung
- Department for Science, Innovation and Technology
- Royal Society
- UK Research and Innovation
- Science and Technology Facilities Council
Zitieren
APA
Duckett, P., Stroud, S. V., Hart, M., Facini, G., & Scanlon, T. (2026). Better queries, cheaper attention: adapting transformers for efficient sparse reconstruction. Machine Learning: Science and Technology, 7(5), 055052. https://doi.org/10.1088/2632-2153/aeb3cc
Vancouver
Duckett P, Stroud SV, Hart M, Facini G, Scanlon T. Better queries, cheaper attention: adapting transformers for efficient sparse reconstruction. Machine Learning: Science and Technology. 2026;7(5):055052. doi:10.1088/2632-2153/aeb3cc
BibTeX
@article{Duckett2026,
author = {Duckett, Philippa and Stroud, Samuel Van and Hart, Max and Facini, Gabriel and Scanlon, Tim},
title = {Better queries, cheaper attention: adapting transformers for efficient sparse reconstruction},
journal = {Machine Learning: Science and Technology},
year = {2026},
volume = {7},
number = {5},
pages = {055052},
doi = {10.1088/2632-2153/aeb3cc}
}
EUVIMED