EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Harmonizer: A Universal Signal Tokenization Framework for Multimodal Large Language Models

Amin Amiri, Alireza Ghaffarnia, Nafiseh Ghaffar Nia, Dalei Wu, Yu Liang

Mathematics · 29.05.2025 · 13 (11) · 1819

DOI: 10.3390/math13111819 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Mathematics, online erschienen am 29.05.2025. Verfasst von Amin Amiri und 4 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 134 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIH AIM-AHEAD, NSF.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper introduces Harmonizer, a universal framework designed for tokenizing heterogeneous input signals, including text, audio, and video, to enable seamless integration into multimodal large language models (LLMs). Harmonizer employs a unified approach to convert diverse, non-linguistic signals into discrete tokens via its FusionQuantizer architecture, built on FluxFormer, to efficiently capture essential signal features while minimizing complexity. We enhance features through STFT-based spectral decomposition, Hilbert transform analytic signal extraction, and SCLAHE spectrogram contrast optimization, and train using a composite loss function to produce reliable embeddings and construct a robust vector vocabulary. Experimental validation on music datasets such as E-GMD v1.0.0, Maestro v3.0.0, and GTZAN demonstrates high fidelity across 288 s of vocal signals (MSE = 0.0037, CC = 0.9282, Cosine Sim. = 0.9278, DTW = 12.12, MFCC Sim. = 0.9997, Spectral Conv. = 0.2485). Preliminary tests on text reconstruction and UCF-101 video clips further confirm Harmonizer’s applicability across discrete and spatiotemporal modalities. Rooted in the universality of wave phenomena and Fourier theory, Harmonizer offers a physics-inspired, modality-agnostic fusion mechanism via wave superposition and interference principles. In summary, Harmonizer integrates natural language processing and signal processing into a coherent tokenization paradigm for efficient, interpretable multimodal learning.

Förderung

  • NIH AIM-AHEAD
  • NSF

Zitieren

APA

Amiri, A., Ghaffarnia, A., Nia, N. G., Wu, D., & Liang, Y. (2025). Harmonizer: A Universal Signal Tokenization Framework for Multimodal Large Language Models. Mathematics, 13(11), 1819. https://doi.org/10.3390/math13111819

Vancouver

Amiri A, Ghaffarnia A, Nia NG, Wu D, Liang Y. Harmonizer: A Universal Signal Tokenization Framework for Multimodal Large Language Models. Mathematics. 2025;13(11):1819. doi:10.3390/math13111819

BibTeX
@article{Amiri2025,
  author = {Amiri, Amin and Ghaffarnia, Alireza and Nia, Nafiseh Ghaffar and Wu, Dalei and Liang, Yu},
  title = {Harmonizer: A Universal Signal Tokenization Framework for Multimodal Large Language Models},
  journal = {Mathematics},
  year = {2025},
  volume = {13},
  number = {11},
  pages = {1819},
  doi = {10.3390/math13111819}
}
Kostenlos starten →