Multimodal Knowledge Graphs Based on Rule-Based Linguistic Analysis and Computer Vision
Journal of Visualized Experiments · 2026
DOI: 10.3791/69803 ↗At a glance
Video/Audio in Journal of Visualized Experiments, 2026. Written by Preeti Vats and 3 co-authors. Institution: Indira Gandhi Delhi Technical University for Women; preeti017phdit22@igdtuw.ac.in. Major MEDLINE topics: Linguistics, Natural Language Processing. 26 references.
Abstract (original language)
Visual-Semantic Hindi-Aligned Multimodal Knowledge Graph (VISHAM-KG) is a framework designed to construct consistent multimodal knowledge graphs (KGs) from Hindi visual documents by systematically aligning visual-text entities. The aim of this study is to integrate rule-based linguistic analysis with computer vision-based object detection that supports the structured semantic representation and grounded reasoning in low-resource Indic languages. The proposed algorithm begins with the preparation of Natural Language Processing (NLP) Hindi visual documents, followed by optical character recognition (OCR) for Devanagari script extraction and linguistic preprocessing, which includes various processes such as tokenization, lemmatization, part-of-speech tagging, and dependency parsing. In parallel, visual entities are extracted from images using object detection and filtered using confidence thresholds. Textual and visual entities are embedded into a shared semantic space using the multilingual transformer model XLM-R, along with CLIP-ViT, and aligned using cosine similarity-based thresholds. These aligned entities are combined with rule-based dependency relations to generate multimodal triplets. The protocol produces a structured multimodal knowledge graph encoded as subject-relation-object triplets with explicit visual grounding based on the Indian knowledge base. This resulting output will support cross-modal querying, entity alignment, and knowledge graph reasoning for Hindi visual documents and provide a replicable framework for multimodal knowledge construction in low-resource linguistic settings.
Subjects
MeSH terms (MEDLINE)
Authors and institutions
- Indira Gandhi Delhi Technical University for Women; preeti017phdit22@igdtuw.ac.in.
- Indira Gandhi Delhi Technical University for Women.
Cite
APA
Vats, P., Sharma, N., Sharma, D. K., & Wary, A. (2026). Multimodal Knowledge Graphs Based on Rule-Based Linguistic Analysis and Computer Vision. Journal of Visualized Experiments. https://doi.org/10.3791/69803
Vancouver
Vats P, Sharma N, Sharma DK, Wary A. Multimodal Knowledge Graphs Based on Rule-Based Linguistic Analysis and Computer Vision. Journal of Visualized Experiments. 2026. doi:10.3791/69803
BibTeX
@article{Vats2026,
author = {Vats, Preeti and Sharma, Nonita and Sharma, Deepak Kumar and Wary, Alongbar},
title = {Multimodal Knowledge Graphs Based on Rule-Based Linguistic Analysis and Computer Vision},
journal = {Journal of Visualized Experiments},
year = {2026},
doi = {10.3791/69803}
}
EUVIMED