EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating the cultural alignment of multilingual LLMs in typical Japanese workplace scenarios

Zhiwei Gao iD, Nobuyuki Shimizu, Sumio Fujita iD, Shaowen Peng, Shoko Wakamiya iD, Eiji Aramaki iD

PLOS One · 27.07.2026 · 21 (7) · e0338524

DOI: 10.1371/journal.pone.0338524 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, online erschienen am 27.07.2026. Verfasst von Zhiwei Gao und 5 weiteren Autor:innen. Beteiligte Einrichtung: Nara Institute of Science and Technology, Ikoma, Japan. Hauptthemen laut MEDLINE: Large Language Models, Multilingualism, Workplace. Die Arbeit nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Cross-ministerial Strategic Innovation Promotion Program.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

While current evaluations of LLM cultural alignment predominantly rely on static benchmarks in Western contexts, their ability to navigate generative, high-context socio-pragmatic demands in non-Western environments remains critically underexplored. This study investigates how multilingual LLMs adapt to the Japanese workplace—a stringent stress-test environment characterized by strong high-context communication norms and rigid honorific conventions—using Hofstede’s six cultural dimensions as a heuristic framework. We evaluated five state-of-the-art LLMs (LLM-jp, Phi, Llama, Qwen, and GLM) through a large-scale crowdsourced human evaluation. Based on 1,718 valid evaluator sessions, native Japanese raters assessed model outputs to generate a holistic Japanese Workplace Cultural Alignment Score (JWCAS). To dissect the underlying communicative strategies, we paired this with a three-layer diagnostic sub-score analysis (Linguistic Form, Socio-Cultural Values, and Social Action). Our results reveal that leading multilingual models (Phi and GLM) achieved overall JWCAS scores comparable to, or significantly higher than, the native Japanese model (LLM-jp). Crucially, our sub-score analysis demonstrates that holistic evaluation metrics can obscure deep pragmatic deficits: while LLM-jp overfits to surface-level linguistic politeness (Layer 1), it shows critical weaknesses in socio-cultural values (Layer 2) and context-aware social strategies (Layer 3). In contrast, leading multilingual models demonstrate balanced competence across all layers. These findings suggest that true cultural competence requires moving beyond native linguistic mastery, highlighting the necessity of multi-dimensional diagnostic frameworks for cross-cultural AI alignment.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsMultilingualismWorkplaceJapan

Autor:innen und Institutionen

  • Nara Institute of Science and Technology, Ikoma, Japan.
  • LY Corporation, Tokyo, Japan.

ORCID-Profile: Zhiwei Gao, Sumio Fujita, Shoko Wakamiya, Eiji Aramaki

Förderung

  • Cross-ministerial Strategic Innovation Promotion Program

Zitieren

APA

Gao, Z., Shimizu, N., Fujita, S., Peng, S., Wakamiya, S., & Aramaki, E. (2026). Evaluating the cultural alignment of multilingual LLMs in typical Japanese workplace scenarios. PLOS One, 21(7), e0338524. https://doi.org/10.1371/journal.pone.0338524

Vancouver

Gao Z, Shimizu N, Fujita S, Peng S, Wakamiya S, Aramaki E. Evaluating the cultural alignment of multilingual LLMs in typical Japanese workplace scenarios. PLOS One. 2026;21(7):e0338524. doi:10.1371/journal.pone.0338524

BibTeX
@article{Gao2026,
  author = {Gao, Zhiwei and Shimizu, Nobuyuki and Fujita, Sumio and Peng, Shaowen and Wakamiya, Shoko and Aramaki, Eiji},
  title = {Evaluating the cultural alignment of multilingual LLMs in typical Japanese workplace scenarios},
  journal = {PLOS One},
  year = {2026},
  volume = {21},
  number = {7},
  pages = {e0338524},
  doi = {10.1371/journal.pone.0338524}
}
Kostenlos starten →