GeoPix: A multimodal large language model for pixel-level image understanding in remote sensing
IEEE Geoscience and Remote Sensing Magazine · 2025-09 · 13 (3) · 324-337
DOI: 10.1109/mgrs.2025.3560293 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Geoscience and Remote Sensing Magazine, 2025-09. Verfasst von Ruizhe Ou und 4 weiteren Autor:innen. Die Arbeit wurde bisher 35-mal zitiert (Crossref) und nennt 57 Literaturangaben. Gefördert von: Yunnan Provincial Science and Technology, National Natural Science Foundation of China, China Postdoctoral Science Foundation.
Abstract (Originalsprache)
Multimodal (MM) large language models (MLLMs) have achieved remarkable success in image- and region-level remote sensing (RS) image understanding tasks, such as image captioning (IC), visual question answering (VQA), and visual grounding. However, existing RS MLLMs lack pixel-level dialogue capability, which involves responding to user instructions with segmentation masks for specific instances. In this article, we propose GeoPix, an RS MLLM that extends image understanding capabilities to the pixel level. This is achieved by equipping the MLLM with a mask predictor, which transforms visual features from the vision encoder into masks conditioned on the LLM’s segmentation token embeddings. To facilitate the segmentation of multiscale objects in RS imagery, a class-wise learnable memory (CLM) module is integrated into the mask predictor to capture and store class-wise geo-context at the instance level across the entire dataset. In addition, to address the absence of large-scale datasets for training pixel-level RS MLLMs, we construct the GeoPixInstruct dataset, comprising 65,463 images and 140,412 instances, with each instance annotated with text descriptions, bounding boxes, and masks. Furthermore, we develop a two-stage training strategy to balance the distinct requirements of text generation and mask prediction in MM multitask optimization. Extensive experiments verify the effectiveness and superiority of GeoPix in pixel-level segmentation tasks while also maintaining competitive performance in image- and region-level benchmarks. The models, dataset, and code are publicly available at https://github.com/Norman-Ou/GeoPix.
Förderung
- Yunnan Provincial Science and Technology
- National Natural Science Foundation of China
- China Postdoctoral Science Foundation
- Peking University
Zitieren
APA
Ou, R., Hu, Y., Zhang, F., Chen, J., & Liu, Y. (2025). GeoPix: A multimodal large language model for pixel-level image understanding in remote sensing. IEEE Geoscience and Remote Sensing Magazine, 13(3), 324-337. https://doi.org/10.1109/mgrs.2025.3560293
Vancouver
Ou R, Hu Y, Zhang F, Chen J, Liu Y. GeoPix: A multimodal large language model for pixel-level image understanding in remote sensing. IEEE Geoscience and Remote Sensing Magazine. 2025;13(3):324-337. doi:10.1109/mgrs.2025.3560293
BibTeX
@article{Ou2025,
author = {Ou, Ruizhe and Hu, Yuan and Zhang, Fan and Chen, Jiaxin and Liu, Yu},
title = {GeoPix: A multimodal large language model for pixel-level image understanding in remote sensing},
journal = {IEEE Geoscience and Remote Sensing Magazine},
year = {2025},
volume = {13},
number = {3},
pages = {324--337},
doi = {10.1109/mgrs.2025.3560293}
}
EUVIMED