Multi‐Modal Instrument Performances (MMIP): A Musical Database
Computer Graphics Forum · 2025-05 · 44 (2)
DOI: 10.1111/cgf.70025 ↗Kurz gefasst
Fachartikel in der Zeitschrift Computer Graphics Forum, online erschienen am 10.04.2025. Verfasst von T. Kyriakou und 2 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 65 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Horizon 2020 Framework Programme, Deputy Ministry of Research, Innovation and Digital Policy.
Abstract (Originalsprache)
Musical instrument performances are multimodal creative art forms that integrate audiovisual elements, resulting from musicians' interactions with instruments through body movements, finger actions, and facial expressions. Digitizing such performances for archiving, streaming, analysis, or synthesis requires capturing every element that shapes the overall experience, which is crucial for preserving the performance's essence. In this work, following current trends in large‐scale dataset development for deep learning analysis and generative models, we introduce the Multi‐Modal Instrument Performances (MMIP) database ( https://mmip.cs.ucy.ac.cy ). This is the first dataset to incorporate synchronized high‐quality 3D motion capture data for the body, fingers, facial expressions, and instruments, along with audio, multi‐angle videos, and MIDI data. The database currently includes 3.5 hours of performances featuring three instruments: guitar, piano, and drums. Additionally, we discuss the challenges of acquiring these multi‐modal data, detailing our approach to data collection, signal synchronization, annotation, and metadata management. Our data formats align with industry standards for ease of use, and we have developed an open‐access online repository that offers a user‐friendly environment for data exploration, supporting data organization, search capabilities, and custom visualization tools. Notable features include a MIDI‐to‐instrument animation project for visualizing the instruments and a script for playing back FBX files with synchronized audio in a web environment.
Förderung
- Horizon 2020 Framework Programme
- Deputy Ministry of Research, Innovation and Digital Policy
Zitieren
APA
Kyriakou, T., Aristidou, A., & Charalambous, P. (2025). Multi‐Modal Instrument Performances (MMIP): A Musical Database. Computer Graphics Forum, 44(2). https://doi.org/10.1111/cgf.70025
Vancouver
Kyriakou T, Aristidou A, Charalambous P. Multi‐Modal Instrument Performances (MMIP): A Musical Database. Computer Graphics Forum. 2025;44(2). doi:10.1111/cgf.70025
BibTeX
@article{Kyriakou2025,
author = {Kyriakou, T. and Aristidou, A. and Charalambous, P.},
title = {Multi‐Modal Instrument Performances (MMIP): A Musical Database},
journal = {Computer Graphics Forum},
year = {2025},
volume = {44},
number = {2},
doi = {10.1111/cgf.70025}
}
EUVIMED