Application of a Large Visual Language Model on Tongue Image Description Generation and Physical Constitution Reasoning in Traditional Chinese Medicine (TongueVLM): Model Development and Validation Study
JMIR Medical Informatics · 12.03.2026 · 14 · e87237
DOI: 10.2196/87237 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift JMIR Medical Informatics, online erschienen am 12.03.2026. Verfasst von Chengdong Peng und 5 weiteren Autor:innen. Beteiligte Einrichtung: School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui, China. Hauptthemen laut MEDLINE: Medicine, Chinese Traditional, Tongue, Body Constitution. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
In the field of traditional Chinese medicine (TCM), diagnostic work based on tongue images to recognize the physical constitution is a process of collecting clinical information, reasoning, and combining the patient’s tongue image features with questioning. It is necessary to simulate the recognition of pathological information of tongue images by TCM practitioners and professional dialogue based on tongue image features, which helps to develop an intelligent interactive system for TCM diagnosis.
Objective
This study aimed to develop and validate a vertical model of the TCM domain with TCM’s understanding and reasoning capability for tongue images.
Methods
A TongueVLM multimodal large model is designed, which includes a visual encoder module, a modal fusion module, and a language decoder module. First, the visual encoder based on the CLIP-ViT (Contrastive Language-Image Pre-Training With Vision Transformer) pretrained model is used for image patch, dimensionality reduction, and migration learning, which maps the high-dimensional tongue features into low-dimensional language encoding vectors. Further, a modal fusion module with a residual architecture is applied to map visual features to a natural language word embedding space, realizing the conceptual alignment between visual encoding and TCM terminology. Finally, fine-tuning of visual instructions is performed based on the LLaMA (large language model meta artificial intelligence), and a TCM-domain large language model with 7B parameters is trained.
Results
The constructed multimodal dataset has 3 test datasets, and experiments are conducted using 3000 samples from each test dataset, respectively. Experimental results indicate that the TongueVLM model outperforms general-purpose large models on all 3 tasks. On the multimodal test dataset, the TongueVLM model achieved accuracy rates of 79.8%, 78.6%, and 60.7% in evaluation tasks respectively, it achieves 9.1%, 8.4%, and 1.1% in greater accuracy than LLaVA-OneVision, and is 7.5%, 7%, and 5.9% more accurate than Qwen2.5-VL-7B, with the text generation time being around 24 tokens per second.
Conclusions
The TongueVLM model, which achieves tongue image description generation and physical constitution reasoning in TCM, is suitable for the application of a Chinese medicine intelligent diagnosis system.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui, China.
- Artificial Intelligence Laboratory, Hefei Yunzhen Information Technology Co, Ltd, Hefei, Anhui, China.
- School of Information and Artificial Intelligence, Anhui Agricultural University, Hefei, Anhui, China.
- The Second Clinical Medical School, Anhui University of Traditional Chinese Medicine, Hefei, Anhui, China.
ORCID-Profile: Chengdong Peng, Jun Gao, Nuo Yang, Yong Wang, Renming Chen, Changwu Dong
Zitieren
APA
Peng, C., Gao, J., Yang, N., Wang, Y., Chen, R., & Dong, C. (2026). Application of a Large Visual Language Model on Tongue Image Description Generation and Physical Constitution Reasoning in Traditional Chinese Medicine (TongueVLM): Model Development and Validation Study. JMIR Medical Informatics, 14, e87237. https://doi.org/10.2196/87237
Vancouver
Peng C, Gao J, Yang N, Wang Y, Chen R, Dong C. Application of a Large Visual Language Model on Tongue Image Description Generation and Physical Constitution Reasoning in Traditional Chinese Medicine (TongueVLM): Model Development and Validation Study. JMIR Medical Informatics. 2026;14:e87237. doi:10.2196/87237
BibTeX
@article{Peng2026,
author = {Peng, Chengdong and Gao, Jun and Yang, Nuo and Wang, Yong and Chen, Renming and Dong, Changwu},
title = {Application of a Large Visual Language Model on Tongue Image Description Generation and Physical Constitution Reasoning in Traditional Chinese Medicine (TongueVLM): Model Development and Validation Study},
journal = {JMIR Medical Informatics},
year = {2026},
volume = {14},
pages = {e87237},
doi = {10.2196/87237}
}
EUVIMED