EUVIMEDEUVIMED entdecken
Crossref

Magnus: A Holistic Approach to Data Management for Large-Scale Machine Learning Workloads

Jun Song, Jingyi Ding, Irshad Kandy, Yanghao Lin, Zhongjia Wei, Zilong Zhou, Zhiwei Peng, Jixi Shan, Hongyue Mao, Xiuqi Huang, Xun Song, Cheng Chen und 8 weitere

Proceedings of the VLDB Endowment · 2025-08 · 18 (12) · 4964-4977

DOI: 10.14778/3750601.3750620 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 16.09.2025. Verfasst von Jun Song und 19 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 51 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Machine learning (ML) has become a cornerstone of key applications at ByteDance. As model complexity and data volumes surge, data management for large-scale ML workloads faces substantial challenges, particularly with recent advances in large recommendation models (LRMs) and large multimodal models (LMMs). Traditional approaches exhibit limitations in storage efficiency, metadata scalability, update mechanisms, and integration with ML frameworks. To address these challenges, we propose Magnus, a holistic data management system built upon Apache Iceberg. Magnus integrates innovative optimizations across resource-efficient storage formats optimized for large wide tables and multimodal data, built-in support for vector and inverted indexes to accelerate data retrieval, scalable metadata planning with Git-like branching and tagging capabilities, and high-performance update/upsert based on lightweight merge-on-read (MOR) strategies. Additionally, Magnus provides native support and specialized enhancement for LRM and LMM training workloads. Experimental results demonstrate significant performance gains in real-world ML scenarios. Magnus has been deployed at ByteDance for over five years, enabling robust and efficient data infrastructure for large-scale ML workloads.

Zitieren

APA

Song, J., Ding, J., Kandy, I., Lin, Y., Wei, Z., Zhou, Z., Peng, Z., Shan, J., Mao, H., Huang, X., Song, X., Chen, C., Li, Y., Yang, T., Jia, W., Dong, X., Lei, K., Shi, R., Zhao, P., & Chen, W. (2025). Magnus: A Holistic Approach to Data Management for Large-Scale Machine Learning Workloads. Proceedings of the VLDB Endowment, 18(12), 4964-4977. https://doi.org/10.14778/3750601.3750620

Vancouver

Song J, Ding J, Kandy I, Lin Y, Wei Z, Zhou Z, et al. Magnus: A Holistic Approach to Data Management for Large-Scale Machine Learning Workloads. Proceedings of the VLDB Endowment. 2025;18(12):4964-4977. doi:10.14778/3750601.3750620

BibTeX
@article{Song2025,
  author = {Song, Jun and Ding, Jingyi and Kandy, Irshad and Lin, Yanghao and Wei, Zhongjia and Zhou, Zilong and Peng, Zhiwei and Shan, Jixi and Mao, Hongyue and Huang, Xiuqi and Song, Xun and Chen, Cheng and Li, Yanjia and Yang, Tianhao and Jia, Wei and Dong, Xiaohong and Lei, Kang and Shi, Rui and Zhao, Pengwei and Chen, Wei},
  title = {Magnus: A Holistic Approach to Data Management for Large-Scale Machine Learning Workloads},
  journal = {Proceedings of the VLDB Endowment},
  year = {2025},
  volume = {18},
  number = {12},
  pages = {4964--4977},
  doi = {10.14778/3750601.3750620}
}
Kostenlos starten →