EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Library-Oriented Large Language Model Approach to Cross-Lingual and Cross-Modal Document Retrieval

Wang Yi, Xiahuan Cai, Hongtao Ma, Zhengjie Fu, Yan Zhan

Electronics · 07.08.2025 · 14 (15) · 3145

DOI: 10.3390/electronics14153145 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 07.08.2025. Verfasst von Wang Yi und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Under the growing demand for processing multimodal and cross-lingual information, traditional retrieval systems have encountered substantial limitations when handling heterogeneous inputs such as images, textual layouts, and multilingual language expressions. To address these challenges, a unified retrieval framework has been proposed, which integrates visual features from images, layout-aware optical character recognition (OCR) text, and bilingual semantic representations in Chinese and English. This framework aims to construct a shared semantic embedding space that mitigates semantic discrepancies across modalities and resolves inconsistencies in cross-lingual mappings. The architecture incorporates three main components: a visual encoder, a structure-aware OCR module, and a multilingual Transformer. Furthermore, a joint contrastive learning loss has been introduced to enhance alignment across both modalities and languages. The proposed method has been evaluated on three core tasks: a single-modality retrieval task from image → OCR, a cross-lingual retrieval task between Chinese and English, and a joint multimodal retrieval task involving image, OCR, and language inputs. Experimental results demonstrate that, in the joint multimodal setting, the proposed model achieved a Precision@10 of 0.693, Recall@10 of 0.684, nDCG@10 of 0.672, and F1@10 of 0.685, substantially outperforming established baselines such as CLIP, LayoutLMv3, and UNITER. Ablation studies revealed that removing either the structure-aware OCR module or the cross-lingual alignment mechanism resulted in a decrease in mean reciprocal rank (MRR) to 0.561, thereby confirming the critical role of these components in reinforcing semantic consistency across modalities. This study highlights the powerful potential of large language models in multimodal semantic fusion and retrieval tasks, providing robust solutions for large-scale semantic understanding and application scenarios in multilingual and multimodal contexts.

Zitieren

APA

Yi, W., Cai, X., Ma, H., Fu, Z., & Zhan, Y. (2025). A Library-Oriented Large Language Model Approach to Cross-Lingual and Cross-Modal Document Retrieval. Electronics, 14(15), 3145. https://doi.org/10.3390/electronics14153145

Vancouver

Yi W, Cai X, Ma H, Fu Z, Zhan Y. A Library-Oriented Large Language Model Approach to Cross-Lingual and Cross-Modal Document Retrieval. Electronics. 2025;14(15):3145. doi:10.3390/electronics14153145

BibTeX
@article{Yi2025,
  author = {Yi, Wang and Cai, Xiahuan and Ma, Hongtao and Fu, Zhengjie and Zhan, Yan},
  title = {A Library-Oriented Large Language Model Approach to Cross-Lingual and Cross-Modal Document Retrieval},
  journal = {Electronics},
  year = {2025},
  volume = {14},
  number = {15},
  pages = {3145},
  doi = {10.3390/electronics14153145}
}
Kostenlos starten →