EUVIMEDEUVIMED entdecken
Crossref

Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine

Xiaoshuang Huang, Lingdong Shen, Jia Liu, Fangxin Shang, Hongxiang Li, Haifeng Huang, Yehui Yang

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (4) · 3779-3787

DOI: 10.1609/aaai.v39i4.32394 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Xiaoshuang Huang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 17-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In recent years, Multimodal Large Language Models (MLLM) have achieved notable advancements, demonstrating the feasibility of developing an intelligent biomedical assistant. However, current biomedical MLLMs predominantly focus on image-level understanding and restrict interactions to textual commands, thus limiting their capability boundaries and the flexibility of usage. In this paper, we introduce a novel end-to-end multimodal large language model for the biomedical domain, named MedPLIB, which possesses pixel-level understanding. Excitingly, it supports visual question answering (VQA), arbitrary pixel-level prompts (points, bounding boxes, and free-form shapes), and pixel-level grounding. We propose a novel Mixture-of-Experts (MoE) multi-stage training strategy, which divides MoE into separate training phases for a visual-language expert model and a pixel-grounding expert model, followed by fine-tuning using MoE. This strategy effectively coordinates multitask learning while maintaining the computational cost at inference equivalent to that of a single expert model. To advance the research of biomedical MLLMs, we introduce the Medical Complex Vision Question Answering Dataset (MeCoVQA), which comprises an array of 8 modalities for complex medical imaging question answering and image region understanding. Experimental results indicate that MedPLIB has achieved state-of-the-art outcomes across multiple medical visual language tasks. More importantly, in zero-shot evaluations for the pixel grounding task, MedPLIB leads the best small and large models by margins of 19.7 and 15.6 respectively on the mDice metric.

Zitieren

APA

Huang, X., Shen, L., Liu, J., Shang, F., Li, H., Huang, H., & Yang, Y. (2025). Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine. Proceedings of the AAAI Conference on Artificial Intelligence, 39(4), 3779-3787. https://doi.org/10.1609/aaai.v39i4.32394

Vancouver

Huang X, Shen L, Liu J, Shang F, Li H, Huang H, et al. Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(4):3779-3787. doi:10.1609/aaai.v39i4.32394

BibTeX
@article{Huang2025,
  author = {Huang, Xiaoshuang and Shen, Lingdong and Liu, Jia and Shang, Fangxin and Li, Hongxiang and Huang, Haifeng and Yang, Yehui},
  title = {Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {4},
  pages = {3779--3787},
  doi = {10.1609/aaai.v39i4.32394}
}
Kostenlos starten →