EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Multi-Scale Audio Spectrogram Transformer for Classroom Teaching Interaction Recognition

Fan Liu, Jiandong Fang

Future Internet · 02.02.2023 · 15 (2) · 65

DOI: 10.3390/fi15020065 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Future Internet, online erschienen am 02.02.2023. Verfasst von Fan Liu und 1 weiteren Autor:innen. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 56 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Inner Mongolia Natural Science Foundation Project, Basic Scientific Research Business Expense Project of Inner Mongolia Universities, Inner Mongolia Science and Technology Plan Project.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Classroom interactivity is one of the important metrics for assessing classrooms, and identifying classroom interactivity through classroom image data is limited by the interference of complex teaching scenarios. However, audio data within the classroom are characterized by significant student–teacher interaction. This study proposes a multi-scale audio spectrogram transformer (MAST) speech scene classification algorithm and constructs a classroom interactive audio dataset to achieve interactive teacher–student recognition in the classroom teaching process. First, the original speech signal is sampled and pre-processed to generate a multi-channel spectrogram, which enhances the representation of features compared with single-channel features; Second, in order to efficiently capture the long-range global context of the audio spectrogram, the audio features are globally modeled by the multi-head self-attention mechanism of MAST, and the feature resolution is reduced during feature extraction to continuously enrich the layer-level features while reducing the model complexity; Finally, a further combination with a time-frequency enrichment module maps the final output to a class feature map, enabling accurate audio category recognition. The experimental comparison of MAST is carried out on the public environment audio dataset and the self-built classroom audio interaction datasets. Compared with the previous state-of-the-art methods on public datasets AudioSet and ESC-50, its accuracy has been improved by 3% and 5%, respectively, and the accuracy of the self-built classroom audio interaction dataset has reached 92.1%. These results demonstrate the effectiveness of MAST in the field of general audio classification and the smart classroom domain.

Förderung

  • Inner Mongolia Natural Science Foundation Project
  • Basic Scientific Research Business Expense Project of Inner Mongolia Universities
  • Inner Mongolia Science and Technology Plan Project

Zitieren

APA

Liu, F., & Fang, J. (2023). Multi-Scale Audio Spectrogram Transformer for Classroom Teaching Interaction Recognition. Future Internet, 15(2), 65. https://doi.org/10.3390/fi15020065

Vancouver

Liu F, Fang J. Multi-Scale Audio Spectrogram Transformer for Classroom Teaching Interaction Recognition. Future Internet. 2023;15(2):65. doi:10.3390/fi15020065

BibTeX
@article{Liu2023,
  author = {Liu, Fan and Fang, Jiandong},
  title = {Multi-Scale Audio Spectrogram Transformer for Classroom Teaching Interaction Recognition},
  journal = {Future Internet},
  year = {2023},
  volume = {15},
  number = {2},
  pages = {65},
  doi = {10.3390/fi15020065}
}
Kostenlos starten →