EUVIMEDObjevte EUVIMED
Open access

Use of ChatGPT to Explore Gender and Geographic Disparities in Scientific Peer Review

Paul Sebo iD

Journal of Medical Internet Research · 2024

DOI: 10.2196/57667 ↗

At a glance

Journal article in Journal of Medical Internet Research, 2024. Written by Paul Sebo. Institution: University of Geneva, Geneva, Switzerland. Major MEDLINE topics: Peer Review. Cited 6 times so far (Crossref); 28 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

In the realm of scientific research, peer review serves as a cornerstone for ensuring the quality and integrity of scholarly papers. Recent trends in promoting transparency and accountability has led some journals to publish peer-review reports alongside papers.

Objective

ChatGPT-4 (OpenAI) was used to quantitatively assess sentiment and politeness in peer-review reports from high-impact medical journals. The objective was to explore gender and geographical disparities to enhance inclusivity within the peer-review process.

Methods

All 9 general medical journals with an impact factor >2 that publish peer-review reports were identified. A total of 12 research papers per journal were randomly selected, all published in 2023. The names of the first and last authors along with the first author’s country of affiliation were collected, and the gender of both the first and last authors was determined. For each review, ChatGPT-4 was asked to evaluate the “sentiment score,” ranging from –100 (negative) to 0 (neutral) to +100 (positive), and the “politeness score,” ranging from –100 (rude) to 0 (neutral) to +100 (polite). The measurements were repeated 5 times and the minimum and maximum values were removed. The mean sentiment and politeness scores for each review were computed and then summarized using the median and interquartile range. Statistical analyses included Wilcoxon rank-sum tests, Kruskal-Wallis rank tests, and negative binomial regressions.

Results

Analysis of 291 peer-review reports corresponding to 108 papers unveiled notable regional disparities. Papers from the Middle East, Latin America, or Africa exhibited lower sentiment and politeness scores compared to those from North America, Europe, or Pacific and Asia (sentiment scores: 27 vs 60 and 62 respectively; politeness scores: 43.5 vs 67 and 65 respectively, adjusted P=.02). No significant differences based on authors’ gender were observed (all P>.05).

Conclusions

Notable regional disparities were found, with papers from the Middle East, Latin America, and Africa demonstrating significantly lower scores, while no discernible differences were observed based on authors’ gender. The absence of gender-based differences suggests that gender biases may not manifest as prominently as other forms of bias within the context of peer review. The study underscores the need for targeted interventions to address regional disparities in peer review and advocates for ongoing efforts to promote equity and inclusivity in scholarly communication.

Subjects

MeSH terms (MEDLINE)

Peer ReviewHumansMaleFemaleJournal Impact FactorPeriodicals as TopicSex FactorsPeer Review, Research

Author keywords

AfricaChatGPTartificial intelligenceassessmentcommunicationconsultationdiscriminationdisparitygendergender biasgeographicglobal southinequalitypeer reviewresearcher

Authors and institutions

  • University of Geneva, Geneva, Switzerland.

ORCID profiles: Paul Sebo

Cite

APA

Sebo, P. (2024). Use of ChatGPT to Explore Gender and Geographic Disparities in Scientific Peer Review. Journal of Medical Internet Research. https://doi.org/10.2196/57667

Vancouver

Sebo P. Use of ChatGPT to Explore Gender and Geographic Disparities in Scientific Peer Review. Journal of Medical Internet Research. 2024. doi:10.2196/57667

BibTeX
@article{Sebo2024,
  author = {Sebo, Paul},
  title = {Use of ChatGPT to Explore Gender and Geographic Disparities in Scientific Peer Review},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/57667}
}
Start for free →