EUVIMEDObjevte EUVIMED
Open access

Evaluating the Reliability and Accuracy of an AI-Powered Search Engine in Providing Responses on Dietary Supplements: Quantitative and Qualitative Evaluation

Mingxin Liu, Tsuyoshi Okuhara, Ritsuko Shirabe, Yuriko Nishiie, Yinghan Xu, Hiroko Okada, Takahiro Kiuchi

JMIR AI · 2025-10-29 · 4 · e78436-e78436

DOI: 10.2196/78436 ↗

At a glance

Journal article in JMIR AI, published online on 2025-10-29. Written by Mingxin Liu and 6 co-authors. Institution: Department of Health Communication, Graduate School of Medicine, The University of Tokyo, Bunkyo, Hongo 7-3-1, Tokyo, 113-8655, Japan, 81 03-5800-654…. Cited 5 times so far (Crossref); 45 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

The widespread adoption of artificial intelligence (AI)–powered search engines has transformed how people access health information. Microsoft Copilot, formerly Bing Chat, offers real-time web-sourced responses to user queries, raising concerns about the reliability of its health content. This is particularly critical in the domain of dietary supplements, where scientific consensus is limited and online misinformation is prevalent. Despite the popularity of supplements in Japan, little is known about the accuracy of AI-generated advice on their effectiveness for common diseases.

Objective

We aimed to evaluate the reliability and accuracy of Microsoft Copilot, an AI search engine, in responding to health-related queries about dietary supplements. Our findings can help consumers use large language models more safely and effectively when seeking information on dietary supplements and support developers in improving large language models’ performance in this field.

Methods

We simulated typical consumer behavior by posing 180 questions (6 per supplement × 30 supplements) to Copilot’s 3 response modes (creative, balanced, and precise) in Japanese. These questions addressed the effectiveness of supplements in treating 6 common conditions (cancer, diabetes, obesity, constipation, joint pain, and hypertension). We classified the AI search engine’s answers as “effective,” “uncertain,” or “ineffective” and evaluated for accuracy against evidence-based assessments conducted by licensed physicians. We conducted a qualitative content analysis of the response texts and systematically examined the types of sources cited in all responses.

Results

The proportion of Copilot responses claiming supplement effectiveness was 29.4% (53/180), 47.8% (86/180), and 45% (81/180) for the creative, balanced, and precise modes, respectively, whereas overall accuracy of the responses was low across all modes: 36.1% (65/180), 31.7% (57/180), and 31.7% (57/180) for creative, balanced, and precise, respectively. No significant difference was observed among the 3 modes ( P= .59). Notably, 72.7% (2240/3081) of the citations came from unverified sources such as blogs, sales websites, and social media. Of the 540 responses analyzed, 54 (10%) contained at least 1 citation in which the cited source did not include or support the claim made by Copilot, indicating hallucinated content. Only 48.5% (262/540) of the responses included a recommendation to consult health care professionals. Among disease categories, the highest accuracy was found for cancer-related questions, likely due to lower misinformation prevalence.

Conclusions

This is the first study to assess Copilot’s performance on dietary supplement information. Despite its authoritative appearance, Copilot frequently cited noncredible sources and provided ambiguous or inaccurate information. Its tendency to avoid definitive stances and align with perceived user expectations poses potential risks for health misinformation. These findings highlight the need for integrating health communication principles—such as transparency, audience empowerment, and informed choice—into the development and regulation of AI search engines to ensure safe public use.

Subjects

Author keywords

AIAI search engineCopilotartificial intelligenceartificial intelligence search enginedietary supplementshealth communicationhealth educationlarge language model

Authors and institutions

  • Department of Health Communication, Graduate School of Medicine, The University of Tokyo, Bunkyo, Hongo 7-3-1, Tokyo, 113-8655, Japan, 81 03-5800-6549.
  • Department of Health Communication, School of Public Health, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.
  • University Hospital Medical Information Network (UMIN) Center, University of Tokyo Hospital, Tokyo, Japan.
  • Graduate School of Human Sciences, Waseda University, Tokorozawa, Japan.

Cite

APA

Liu, M., Okuhara, T., Shirabe, R., Nishiie, Y., Xu, Y., Okada, H., & Kiuchi, T. (2025). Evaluating the Reliability and Accuracy of an AI-Powered Search Engine in Providing Responses on Dietary Supplements: Quantitative and Qualitative Evaluation. JMIR AI, 4, e78436-e78436. https://doi.org/10.2196/78436

Vancouver

Liu M, Okuhara T, Shirabe R, Nishiie Y, Xu Y, Okada H, et al. Evaluating the Reliability and Accuracy of an AI-Powered Search Engine in Providing Responses on Dietary Supplements: Quantitative and Qualitative Evaluation. JMIR AI. 2025;4:e78436-e78436. doi:10.2196/78436

BibTeX
@article{Liu2025,
  author = {Liu, Mingxin and Okuhara, Tsuyoshi and Shirabe, Ritsuko and Nishiie, Yuriko and Xu, Yinghan and Okada, Hiroko and Kiuchi, Takahiro},
  title = {Evaluating the Reliability and Accuracy of an AI-Powered Search Engine in Providing Responses on Dietary Supplements: Quantitative and Qualitative Evaluation},
  journal = {JMIR AI},
  year = {2025},
  volume = {4},
  pages = {e78436--e78436},
  doi = {10.2196/78436}
}
Start for free →