EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Optimize Multimodal Data Mixture for Pre-Training with Loss Regression

Linjiang Shang, Huanyu Cheng, Bo Zhou, Yin Zhang

Algorithms · 15.06.2026 · 19 (6) · 482

DOI: 10.3390/a19060482 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Algorithms, online erschienen am 15.06.2026. Verfasst von Linjiang Shang und 3 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Science and Technology Projects of State Grid Jiangsu Electric Power Company Ltd..

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Different mixtures of multimodal training data significantly impact the performance of multimodal large language models, and manually tuning data mixtures is inefficient, computationally expensive, and frequently suboptimal because of complex, nonlinear inter-modal interactions. How to determine data-mixture hyperparameters in an efficient and principled manner becomes the bottleneck for progress in the field. This study establishes a scalable, learnable framework, DMPredictor, that treats multimodal data-mixture design as a regression-based hyperparameter-optimization problem and automates the selection of effective training data mixtures. DMPredictor is trained on data mixture samples derived from hundreds of small proxy models (2M parameters), each of which is trained on 1B tokens sampled using different data mixtures. The framework incorporates alignment-aware smoothing and quality-reweighting, enabling diverse exploration of the multimodal data mixture space while avoiding distribution collapse. DMPredictor produces accurate performance forecasts and identifies nearly optimal data mixtures. The predicted optimal mixture surpasses human-designed baselines on diverse benchmarks, achieving +2.7% on MMMU, +6.4% on TextVQA, and +195.2 on MME. Moreover, the mixture optimization complexity is largely reduced by small proxies and a small number of tokens. The proposed approach offers a robust, computationally efficient pathway for optimizing mixtures of multimodal training data, addressing the critical challenge of training data heterogeneity.

Förderung

  • Science and Technology Projects of State Grid Jiangsu Electric Power Company Ltd.

Zitieren

APA

Shang, L., Cheng, H., Zhou, B., & Zhang, Y. (2026). Optimize Multimodal Data Mixture for Pre-Training with Loss Regression. Algorithms, 19(6), 482. https://doi.org/10.3390/a19060482

Vancouver

Shang L, Cheng H, Zhou B, Zhang Y. Optimize Multimodal Data Mixture for Pre-Training with Loss Regression. Algorithms. 2026;19(6):482. doi:10.3390/a19060482

BibTeX
@article{Shang2026,
  author = {Shang, Linjiang and Cheng, Huanyu and Zhou, Bo and Zhang, Yin},
  title = {Optimize Multimodal Data Mixture for Pre-Training with Loss Regression},
  journal = {Algorithms},
  year = {2026},
  volume = {19},
  number = {6},
  pages = {482},
  doi = {10.3390/a19060482}
}
Kostenlos starten →