EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Automated Acoustic Side-Channel Attack on Keyboard Inputs via Combined Video–Audio Analysis

Dario Vranješ, Ivo Stančić, Marin Bugarić, Toni Perković

Electronics · 07.08.2026 · 15 (16) · 3509

DOI: 10.3390/electronics15163509 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 07.08.2026. Verfasst von Dario Vranješ und 3 weiteren Autor:innen. Die Arbeit nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Union.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Acoustic side-channel attacks (ASCAs) exploit unintended sound emitted by keyboards to infer typed input, but existing methods generally assume manually labelled training data and controlled environments, limiting their applicability to realistic scenarios such as online lectures. We develop a pipeline that automatically labels keystroke-sound samples captured from online coding tutorials: video frames are processed with optical character recognition (OCR) to extract the ground-truth character sequence, audio is segmented into clips centred on detected click events, and the two streams are aligned. A convolutional neural network (CNN) is trained on mel-spectrogram features, with transfer learning used to adapt the pretrained model to a target user with minimal samples. The classifier is trained on all 68 physical keys present in the recordings; of these, 50 produce a character or whitespace and the remaining 18 are control, navigation, and modifier keys. On a held-out test set, the CNN achieves 98.1% top-1, 99.4% top-2, and 100% top-3 accuracy. Transfer learning retains strong performance with as few as 13 samples per key. Pairing OCR-derived ground truth with acoustic CNN classification removes the labelling bottleneck that has limited previous ASCAs, and the transfer-learning stage makes the attack viable with minimal per-victim data. All code, trained models, and labelled datasets are released to support reproducible research.

Förderung

  • European Union

Zitieren

APA

Vranješ, D., Stančić, I., Bugarić, M., & Perković, T. (2026). Automated Acoustic Side-Channel Attack on Keyboard Inputs via Combined Video–Audio Analysis. Electronics, 15(16), 3509. https://doi.org/10.3390/electronics15163509

Vancouver

Vranješ D, Stančić I, Bugarić M, Perković T. Automated Acoustic Side-Channel Attack on Keyboard Inputs via Combined Video–Audio Analysis. Electronics. 2026;15(16):3509. doi:10.3390/electronics15163509

BibTeX
@article{Vranje2026,
  author = {Vranješ, Dario and Stančić, Ivo and Bugarić, Marin and Perković, Toni},
  title = {Automated Acoustic Side-Channel Attack on Keyboard Inputs via Combined Video–Audio Analysis},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {16},
  pages = {3509},
  doi = {10.3390/electronics15163509}
}
Kostenlos starten →