EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

PointLLM-V2: Empowering Large Language Models to Better Understand Point Clouds

Runsen Xu, Shuai Yang, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, Dahua Lin

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025 · 1-15

DOI: 10.1109/tpami.2025.3590784 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. Verfasst von Runsen Xu und 6 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The unprecedented advancements in Large Language Models (LLMs) have shown a profound impact on natural language processing but are yet to fully embrace the realm of 3D understanding. This paper introduces PointLLM, a preliminary effort to fill this gap, empowering LLMs to understand point clouds and offering a new avenue beyond 2D data. PointLLM understands colored object point clouds with human instructions, including coordinate-based part specifications, and generates contextually appropriate responses, illustrating its grasp of point clouds and common sense. Specifically, it leverages a point cloud encoder with a powerful LLM to effectively fuse geometric, appearance, and linguistic information. To overcome the scarcity of point-text instruction following data, we developed an automated data generation pipeline, collecting a large-scale dataset of about 1.8M samples with 1M different 3D objects, which facilitates the adoption of the two-stage training strategy prevalent in MLLM development. Additionally, we address the absence of appropriate benchmarks and the limitations of current evaluation metrics by proposing two novel benchmarks: Generative 3D Object Classification and 3D Object Captioning, which are supported by new, comprehensive evaluation metrics derived from human and GPT analyses. Through exploring various training strategies, we develop PointLLM, significantly outperforming 2D and 3D baselines and achieving SOTA performance, with a notable achievement in object captioning tasks where it surpasses human annotators in over 50% of the samples. Codes, datasets, and benchmarks will be available at https://github.com/OpenRobotLab/PointLLM.

Zitieren

APA

Xu, R., Yang, S., Wang, X., Wang, T., Chen, Y., Pang, J., & Lin, D. (2025). PointLLM-V2: Empowering Large Language Models to Better Understand Point Clouds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1-15. https://doi.org/10.1109/tpami.2025.3590784

Vancouver

Xu R, Yang S, Wang X, Wang T, Chen Y, Pang J, et al. PointLLM-V2: Empowering Large Language Models to Better Understand Point Clouds. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025:1-15. doi:10.1109/tpami.2025.3590784

BibTeX
@article{Xu2025,
  author = {Xu, Runsen and Yang, Shuai and Wang, Xiaolong and Wang, Tai and Chen, Yilun and Pang, Jiangmiao and Lin, Dahua},
  title = {PointLLM-V2: Empowering Large Language Models to Better Understand Point Clouds},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  pages = {1--15},
  doi = {10.1109/tpami.2025.3590784}
}
Kostenlos starten →