Harmonizer: A Universal Signal Tokenization Framework for Multimodal Large Language Models
Mathematics · 29.05.2025 · 13 (11) · 1819
DOI: 10.3390/math13111819 ↗Kurz gefasst
Fachartikel in der Zeitschrift Mathematics, online erschienen am 29.05.2025. Verfasst von Amin Amiri und 4 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 134 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIH AIM-AHEAD, NSF.
Abstract (Originalsprache)
This paper introduces Harmonizer, a universal framework designed for tokenizing heterogeneous input signals, including text, audio, and video, to enable seamless integration into multimodal large language models (LLMs). Harmonizer employs a unified approach to convert diverse, non-linguistic signals into discrete tokens via its FusionQuantizer architecture, built on FluxFormer, to efficiently capture essential signal features while minimizing complexity. We enhance features through STFT-based spectral decomposition, Hilbert transform analytic signal extraction, and SCLAHE spectrogram contrast optimization, and train using a composite loss function to produce reliable embeddings and construct a robust vector vocabulary. Experimental validation on music datasets such as E-GMD v1.0.0, Maestro v3.0.0, and GTZAN demonstrates high fidelity across 288 s of vocal signals (MSE = 0.0037, CC = 0.9282, Cosine Sim. = 0.9278, DTW = 12.12, MFCC Sim. = 0.9997, Spectral Conv. = 0.2485). Preliminary tests on text reconstruction and UCF-101 video clips further confirm Harmonizer’s applicability across discrete and spatiotemporal modalities. Rooted in the universality of wave phenomena and Fourier theory, Harmonizer offers a physics-inspired, modality-agnostic fusion mechanism via wave superposition and interference principles. In summary, Harmonizer integrates natural language processing and signal processing into a coherent tokenization paradigm for efficient, interpretable multimodal learning.
Förderung
- NIH AIM-AHEAD
- NSF
Zitieren
APA
Amiri, A., Ghaffarnia, A., Nia, N. G., Wu, D., & Liang, Y. (2025). Harmonizer: A Universal Signal Tokenization Framework for Multimodal Large Language Models. Mathematics, 13(11), 1819. https://doi.org/10.3390/math13111819
Vancouver
Amiri A, Ghaffarnia A, Nia NG, Wu D, Liang Y. Harmonizer: A Universal Signal Tokenization Framework for Multimodal Large Language Models. Mathematics. 2025;13(11):1819. doi:10.3390/math13111819
BibTeX
@article{Amiri2025,
author = {Amiri, Amin and Ghaffarnia, Alireza and Nia, Nafiseh Ghaffar and Wu, Dalei and Liang, Yu},
title = {Harmonizer: A Universal Signal Tokenization Framework for Multimodal Large Language Models},
journal = {Mathematics},
year = {2025},
volume = {13},
number = {11},
pages = {1819},
doi = {10.3390/math13111819}
}
EUVIMED