EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

An Interactive Desktop Application for Human Gesture-Controlled Music Generation Using Deep Learning-Based Hand Tracking

Marius-Valentin Drăgoi, Răzvan-Andrei Timofte, Iuliana Marin, Cozmin Adrian Cristoiu

Technologies · 11.11.2026 · 14 (11) · 747

DOI: 10.3390/technologies14110747 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Technologies, online erschienen am 11.11.2026. Verfasst von Marius-Valentin Drăgoi und 3 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper presents an interactive desktop application for human gesture-controlled music generation. The system uses a standard webcam and deep learning-based hand tracking to detect both hands and convert hand features into musical control actions. The application was developed in Python and uses OpenCV, MediaPipe, FluidSynth, MIDI synthesis, and WAV stem playback. It provides three modes. Free Play lets the user control notes and dynamics with hand movement. Song Generation lets the user guide melody, bass, and chords inside a beat-based structure. Stem Control lets the user activate or mute separate audio stems with finger gestures. Deep learning is used for hand tracking, while musical output is produced through rule-based mappings, MIDI synthesis, and prerecorded WAV stem playback. The system was tested with 30 voluntary participants. The average frame rate was 21.81 fps, and the average MediaPipe inference time was 20.80 ms. The average calibration score was 7.93 out of 8 checks. In Stem Control, an average of 4.80 out of five software-defined task criteria were met. The software-defined beat-drop timing condition was met for 16 of the 30 participants. Questionnaire scores were positive, with 4.57 for ease of use, 3.97 for naturalness, and 4.47 for reuse intention. The results show that webcam-based hand tracking can support interactive music control without requiring specialized hardware.

Zitieren

APA

Drăgoi, M., Timofte, R., Marin, I., & Cristoiu, C. A. (2026). An Interactive Desktop Application for Human Gesture-Controlled Music Generation Using Deep Learning-Based Hand Tracking. Technologies, 14(11), 747. https://doi.org/10.3390/technologies14110747

Vancouver

Drăgoi M, Timofte R, Marin I, Cristoiu CA. An Interactive Desktop Application for Human Gesture-Controlled Music Generation Using Deep Learning-Based Hand Tracking. Technologies. 2026;14(11):747. doi:10.3390/technologies14110747

BibTeX
@article{Drgoi2026,
  author = {Drăgoi, Marius-Valentin and Timofte, Răzvan-Andrei and Marin, Iuliana and Cristoiu, Cozmin Adrian},
  title = {An Interactive Desktop Application for Human Gesture-Controlled Music Generation Using Deep Learning-Based Hand Tracking},
  journal = {Technologies},
  year = {2026},
  volume = {14},
  number = {11},
  pages = {747},
  doi = {10.3390/technologies14110747}
}
Kostenlos starten →