EUVIMEDEUVIMED entdecken
Crossref

Woodpecker: hallucination correction for multimodal large language models

Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, Enhong Chen

Science China Information Sciences · 2024-12 · 67 (12)

DOI: 10.1007/s11432-024-4251-x ↗

Kurz gefasst

Fachartikel in der Zeitschrift Science China Information Sciences, online erschienen am 13.12.2024. Verfasst von Shukang Yin und 9 weiteren Autor:innen. Die Arbeit wurde bisher 118-mal zitiert (Crossref) und nennt 53 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Hallucinations is a big shadow hanging over the rapidly evolving multimodal large language models (MLLMs), referring to that the generated text is inconsistent with the image content. To mitigate hallucinations, existing studies mainly resort to an instruction-tuning manner that requires retraining the models with specific data. In this paper, we pave a different way, introducing a training-free method named Woodpecker. Like woodpeckers heal trees, it picks out and corrects hallucinations from the generated text. Concretely, Woodpecker consists of five stages: key concept extraction, question formulation, visual knowledge validation, visual claim generation, and hallucination correction. Implemented in a post-remedy manner, Woodpecker can easily serve different MLLMs, while being interpretable by accessing intermediate outputs of the five stages. We evaluate Woodpecker both quantitatively and qualitatively and show the huge potential of this new paradigm. On the POPE benchmark, our method obtains a 30.66%/24.33% improvement in accuracy over the baseline MiniGPT-4/mPLUG-Owl. The source code is released at https://github.com/BradyFU/Woodpecker.

Zitieren

APA

Yin, S., Fu, C., Zhao, S., Xu, T., Wang, H., Sui, D., Shen, Y., Li, K., Sun, X., & Chen, E. (2024). Woodpecker: hallucination correction for multimodal large language models. Science China Information Sciences, 67(12). https://doi.org/10.1007/s11432-024-4251-x

Vancouver

Yin S, Fu C, Zhao S, Xu T, Wang H, Sui D, et al. Woodpecker: hallucination correction for multimodal large language models. Science China Information Sciences. 2024;67(12). doi:10.1007/s11432-024-4251-x

BibTeX
@article{Yin2024,
  author = {Yin, Shukang and Fu, Chaoyou and Zhao, Sirui and Xu, Tong and Wang, Hao and Sui, Dianbo and Shen, Yunhang and Li, Ke and Sun, Xing and Chen, Enhong},
  title = {Woodpecker: hallucination correction for multimodal large language models},
  journal = {Science China Information Sciences},
  year = {2024},
  volume = {67},
  number = {12},
  doi = {10.1007/s11432-024-4251-x}
}
Kostenlos starten →