EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Comparative analysis of automatic gender detection from names: evaluating the stability and performance of ChatGPT versus Namsor, and Gender-API

Adrián Domínguez-Díaz, Manuel Goyanes, Luis de-Marcos, Víctor Pablo Prado-Sánchez iD

PeerJ Computer Science · 17.10.2024 · 10 · e2378

DOI: 10.7717/peerj-cs.2378 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PeerJ Computer Science, online erschienen am 17.10.2024. Verfasst von Adrián Domínguez-Díaz und 3 weiteren Autor:innen. Beteiligte Einrichtung: Ciencias de la Computación, Universidad de Alcalá, Alcalá de Henares, Spain. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The gender classification from names is crucial for uncovering a myriad of gender-related research questions. Traditionally, this has been automatically computed by gender detection tools (GDTs), which now face new industry players in the form of conversational bots like ChatGPT. This paper statistically tests the stability and performance of ChatGPT 3.5 Turbo and ChatGPT 4o for gender detection. It also compares two of the most used GDTs (Namsor and Gender-API) with ChatGPT using a dataset of 5,779 records compiled from previous studies for the most challenging variant, which is the gender inference from full name without providing any additional information. Results statistically show that ChatGPT is very stable presenting low standard deviation and tight confidence intervals for the same input, while it presents small differences in performance when prompt changes. ChatGPT slightly outperforms the other tools with an overall accuracy over 96%, although the difference is around 3% with both GDTs. When the probability returned by GDTs is factored in, differences get narrower and comparable in terms of inter-coder reliability and error coded. ChatGPT stands out in the reduced number of non-classifications (0% in most tests), which in combination with the other metrics analyzed, results in a solid alternative for gender inference. This paper contributes to current literature on gender detection classification from names by testing the stability and performance of the most used state-of-the-art AI tool, suggesting that the generative language model of ChatGPT provides a robust alternative to traditional gender application programming interfaces (APIs), yet GDTs (especially Namsor) should be considered for research-oriented purposes.

Schlagwörter

Stichwörter der Autor:innen

Automatic gender detectionChatGPTConversational botGender detectionGender detection toolGender inferenceName-to-gender

Autor:innen und Institutionen

  • Ciencias de la Computación, Universidad de Alcalá, Alcalá de Henares, Spain.
  • Comunicación, Universidad Carlos III de Madrid, Getafe, Spain.

ORCID-Profile: Víctor Pablo Prado-Sánchez

Zitieren

APA

Domínguez-Díaz, A., Goyanes, M., de-Marcos, L., & Prado-Sánchez, V. P. (2024). Comparative analysis of automatic gender detection from names: evaluating the stability and performance of ChatGPT versus Namsor, and Gender-API. PeerJ Computer Science, 10, e2378. https://doi.org/10.7717/peerj-cs.2378

Vancouver

Domínguez-Díaz A, Goyanes M, de-Marcos L, Prado-Sánchez VP. Comparative analysis of automatic gender detection from names: evaluating the stability and performance of ChatGPT versus Namsor, and Gender-API. PeerJ Computer Science. 2024;10:e2378. doi:10.7717/peerj-cs.2378

BibTeX
@article{DomnguezDaz2024,
  author = {Domínguez-Díaz, Adrián and Goyanes, Manuel and de-Marcos, Luis and Prado-Sánchez, Víctor Pablo},
  title = {Comparative analysis of automatic gender detection from names: evaluating the stability and performance of ChatGPT versus Namsor, and Gender-API},
  journal = {PeerJ Computer Science},
  year = {2024},
  volume = {10},
  pages = {e2378},
  doi = {10.7717/peerj-cs.2378}
}
Kostenlos starten →