HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
ACM Transactions on Intelligent Systems and Technology · 28.02.2027 · 18 (1) · 1-26
DOI: 10.1145/3845999 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 29.10.2026. Verfasst von Shaina Raza und 8 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Union’s Horizon Europe research and innovation programme.
Abstract (Originalsprache)
Although recent large multimodal models (LMMs) show impressive progress on vision–language tasks, their alignment with human-centered (HC) principles such as fairness, ethics, inclusivity, empathy, and robustness is often overlooked. Existing LMM benchmarks are largely accuracy-agnostic. We present HumaniBench, a unified framework for characterizing HC alignment across realistic, socially grounded visual contexts. It contains 32,000 expert-verified image–question pairs from real-world news imagery, each mapped to one or more HC principles through explicit metrics. Comparing 15 state-of-the-art LMMs reveals consistent tradeoffs: proprietary systems lead on ethics, reasoning, and empathy, while open source models show superior visual grounding and resilience. All models show persistent gaps in fairness and multilingual inclusivity. Chain-of-thought prompting and test-time scaling yield 8–12% gains on several HC dimensions. HumaniBench enables fine-grained analysis of alignment tradeoffs not captured by conventional multimodal benchmarks. Project: https://vectorinstitute.github.io/humanibench/ Data: https://huggingface.co/vector-institute/HumaniBench Code: https://github.com/VectorInstitute/HumaniBench
Förderung
- European Union’s Horizon Europe research and innovation programme
Zitieren
APA
Raza, S., Narayanan, A., Khazaie, V. R., Vayani, A., Radwan, A. Y., Chettiar, M. S., Singh, A., Shah, M., & Pandya, D. (2027). HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation. ACM Transactions on Intelligent Systems and Technology, 18(1), 1-26. https://doi.org/10.1145/3845999
Vancouver
Raza S, Narayanan A, Khazaie VR, Vayani A, Radwan AY, Chettiar MS, et al. HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation. ACM Transactions on Intelligent Systems and Technology. 2027;18(1):1-26. doi:10.1145/3845999
BibTeX
@article{Raza2027,
author = {Raza, Shaina and Narayanan, Aravind and Khazaie, Vahid Reza and Vayani, Ashmal and Radwan, Ahmed Y. and Chettiar, Mukund S. and Singh, Amandeep and Shah, Mubarak and Pandya, Deval},
title = {HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation},
journal = {ACM Transactions on Intelligent Systems and Technology},
year = {2027},
volume = {18},
number = {1},
pages = {1--26},
doi = {10.1145/3845999}
}
EUVIMED