EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation

Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie, Ashmal Vayani, Ahmed Y. Radwan, Mukund S. Chettiar, Amandeep Singh, Mubarak Shah, Deval Pandya

ACM Transactions on Intelligent Systems and Technology · 28.02.2027 · 18 (1) · 1-26

DOI: 10.1145/3845999 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 29.10.2026. Verfasst von Shaina Raza und 8 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Union’s Horizon Europe research and innovation programme.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Although recent large multimodal models (LMMs) show impressive progress on vision–language tasks, their alignment with human-centered (HC) principles such as fairness, ethics, inclusivity, empathy, and robustness is often overlooked. Existing LMM benchmarks are largely accuracy-agnostic. We present HumaniBench, a unified framework for characterizing HC alignment across realistic, socially grounded visual contexts. It contains 32,000 expert-verified image–question pairs from real-world news imagery, each mapped to one or more HC principles through explicit metrics. Comparing 15 state-of-the-art LMMs reveals consistent tradeoffs: proprietary systems lead on ethics, reasoning, and empathy, while open source models show superior visual grounding and resilience. All models show persistent gaps in fairness and multilingual inclusivity. Chain-of-thought prompting and test-time scaling yield 8–12% gains on several HC dimensions. HumaniBench enables fine-grained analysis of alignment tradeoffs not captured by conventional multimodal benchmarks. Project: https://vectorinstitute.github.io/humanibench/ Data: https://huggingface.co/vector-institute/HumaniBench Code: https://github.com/VectorInstitute/HumaniBench

Förderung

  • European Union’s Horizon Europe research and innovation programme

Zitieren

APA

Raza, S., Narayanan, A., Khazaie, V. R., Vayani, A., Radwan, A. Y., Chettiar, M. S., Singh, A., Shah, M., & Pandya, D. (2027). HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation. ACM Transactions on Intelligent Systems and Technology, 18(1), 1-26. https://doi.org/10.1145/3845999

Vancouver

Raza S, Narayanan A, Khazaie VR, Vayani A, Radwan AY, Chettiar MS, et al. HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation. ACM Transactions on Intelligent Systems and Technology. 2027;18(1):1-26. doi:10.1145/3845999

BibTeX
@article{Raza2027,
  author = {Raza, Shaina and Narayanan, Aravind and Khazaie, Vahid Reza and Vayani, Ashmal and Radwan, Ahmed Y. and Chettiar, Mukund S. and Singh, Amandeep and Shah, Mubarak and Pandya, Deval},
  title = {HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation},
  journal = {ACM Transactions on Intelligent Systems and Technology},
  year = {2027},
  volume = {18},
  number = {1},
  pages = {1--26},
  doi = {10.1145/3845999}
}
Kostenlos starten →