Evaluating the Reliability and Accuracy of an AI-Powered Search Engine in Providing Responses on Dietary Supplements: Quantitative and Qualitative Evaluation
JMIR AI · 2025-10-29 · 4 · e78436-e78436
DOI: 10.2196/78436 ↗At a glance
Journal article in JMIR AI, published online on 2025-10-29. Written by Mingxin Liu and 6 co-authors. Institution: Department of Health Communication, Graduate School of Medicine, The University of Tokyo, Bunkyo, Hongo 7-3-1, Tokyo, 113-8655, Japan, 81 03-5800-654…. Cited 5 times so far (Crossref); 45 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
The widespread adoption of artificial intelligence (AI)–powered search engines has transformed how people access health information. Microsoft Copilot, formerly Bing Chat, offers real-time web-sourced responses to user queries, raising concerns about the reliability of its health content. This is particularly critical in the domain of dietary supplements, where scientific consensus is limited and online misinformation is prevalent. Despite the popularity of supplements in Japan, little is known about the accuracy of AI-generated advice on their effectiveness for common diseases.
Objective
We aimed to evaluate the reliability and accuracy of Microsoft Copilot, an AI search engine, in responding to health-related queries about dietary supplements. Our findings can help consumers use large language models more safely and effectively when seeking information on dietary supplements and support developers in improving large language models’ performance in this field.
Methods
We simulated typical consumer behavior by posing 180 questions (6 per supplement × 30 supplements) to Copilot’s 3 response modes (creative, balanced, and precise) in Japanese. These questions addressed the effectiveness of supplements in treating 6 common conditions (cancer, diabetes, obesity, constipation, joint pain, and hypertension). We classified the AI search engine’s answers as “effective,” “uncertain,” or “ineffective” and evaluated for accuracy against evidence-based assessments conducted by licensed physicians. We conducted a qualitative content analysis of the response texts and systematically examined the types of sources cited in all responses.
Results
The proportion of Copilot responses claiming supplement effectiveness was 29.4% (53/180), 47.8% (86/180), and 45% (81/180) for the creative, balanced, and precise modes, respectively, whereas overall accuracy of the responses was low across all modes: 36.1% (65/180), 31.7% (57/180), and 31.7% (57/180) for creative, balanced, and precise, respectively. No significant difference was observed among the 3 modes ( P= .59). Notably, 72.7% (2240/3081) of the citations came from unverified sources such as blogs, sales websites, and social media. Of the 540 responses analyzed, 54 (10%) contained at least 1 citation in which the cited source did not include or support the claim made by Copilot, indicating hallucinated content. Only 48.5% (262/540) of the responses included a recommendation to consult health care professionals. Among disease categories, the highest accuracy was found for cancer-related questions, likely due to lower misinformation prevalence.
Conclusions
This is the first study to assess Copilot’s performance on dietary supplement information. Despite its authoritative appearance, Copilot frequently cited noncredible sources and provided ambiguous or inaccurate information. Its tendency to avoid definitive stances and align with perceived user expectations poses potential risks for health misinformation. These findings highlight the need for integrating health communication principles—such as transparency, audience empowerment, and informed choice—into the development and regulation of AI search engines to ensure safe public use.
Subjects
Author keywords
Authors and institutions
- Department of Health Communication, Graduate School of Medicine, The University of Tokyo, Bunkyo, Hongo 7-3-1, Tokyo, 113-8655, Japan, 81 03-5800-6549.
- Department of Health Communication, School of Public Health, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.
- University Hospital Medical Information Network (UMIN) Center, University of Tokyo Hospital, Tokyo, Japan.
- Graduate School of Human Sciences, Waseda University, Tokorozawa, Japan.
Cite
APA
Liu, M., Okuhara, T., Shirabe, R., Nishiie, Y., Xu, Y., Okada, H., & Kiuchi, T. (2025). Evaluating the Reliability and Accuracy of an AI-Powered Search Engine in Providing Responses on Dietary Supplements: Quantitative and Qualitative Evaluation. JMIR AI, 4, e78436-e78436. https://doi.org/10.2196/78436
Vancouver
Liu M, Okuhara T, Shirabe R, Nishiie Y, Xu Y, Okada H, et al. Evaluating the Reliability and Accuracy of an AI-Powered Search Engine in Providing Responses on Dietary Supplements: Quantitative and Qualitative Evaluation. JMIR AI. 2025;4:e78436-e78436. doi:10.2196/78436
BibTeX
@article{Liu2025,
author = {Liu, Mingxin and Okuhara, Tsuyoshi and Shirabe, Ritsuko and Nishiie, Yuriko and Xu, Yinghan and Okada, Hiroko and Kiuchi, Takahiro},
title = {Evaluating the Reliability and Accuracy of an AI-Powered Search Engine in Providing Responses on Dietary Supplements: Quantitative and Qualitative Evaluation},
journal = {JMIR AI},
year = {2025},
volume = {4},
pages = {e78436--e78436},
doi = {10.2196/78436}
}
EUVIMED