Comparative analysis of automatic gender detection from names: evaluating the stability and performance of ChatGPT versus Namsor, and Gender-API
PeerJ Computer Science · 17.10.2024 · 10 · e2378
DOI: 10.7717/peerj-cs.2378 ↗Kurz gefasst
Fachartikel in der Zeitschrift PeerJ Computer Science, online erschienen am 17.10.2024. Verfasst von Adrián Domínguez-Díaz und 3 weiteren Autor:innen. Beteiligte Einrichtung: Ciencias de la Computación, Universidad de Alcalá, Alcalá de Henares, Spain. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The gender classification from names is crucial for uncovering a myriad of gender-related research questions. Traditionally, this has been automatically computed by gender detection tools (GDTs), which now face new industry players in the form of conversational bots like ChatGPT. This paper statistically tests the stability and performance of ChatGPT 3.5 Turbo and ChatGPT 4o for gender detection. It also compares two of the most used GDTs (Namsor and Gender-API) with ChatGPT using a dataset of 5,779 records compiled from previous studies for the most challenging variant, which is the gender inference from full name without providing any additional information. Results statistically show that ChatGPT is very stable presenting low standard deviation and tight confidence intervals for the same input, while it presents small differences in performance when prompt changes. ChatGPT slightly outperforms the other tools with an overall accuracy over 96%, although the difference is around 3% with both GDTs. When the probability returned by GDTs is factored in, differences get narrower and comparable in terms of inter-coder reliability and error coded. ChatGPT stands out in the reduced number of non-classifications (0% in most tests), which in combination with the other metrics analyzed, results in a solid alternative for gender inference. This paper contributes to current literature on gender detection classification from names by testing the stability and performance of the most used state-of-the-art AI tool, suggesting that the generative language model of ChatGPT provides a robust alternative to traditional gender application programming interfaces (APIs), yet GDTs (especially Namsor) should be considered for research-oriented purposes.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Ciencias de la Computación, Universidad de Alcalá, Alcalá de Henares, Spain.
- Comunicación, Universidad Carlos III de Madrid, Getafe, Spain.
ORCID-Profile: Víctor Pablo Prado-Sánchez
Zitieren
APA
Domínguez-Díaz, A., Goyanes, M., de-Marcos, L., & Prado-Sánchez, V. P. (2024). Comparative analysis of automatic gender detection from names: evaluating the stability and performance of ChatGPT versus Namsor, and Gender-API. PeerJ Computer Science, 10, e2378. https://doi.org/10.7717/peerj-cs.2378
Vancouver
Domínguez-Díaz A, Goyanes M, de-Marcos L, Prado-Sánchez VP. Comparative analysis of automatic gender detection from names: evaluating the stability and performance of ChatGPT versus Namsor, and Gender-API. PeerJ Computer Science. 2024;10:e2378. doi:10.7717/peerj-cs.2378
BibTeX
@article{DomnguezDaz2024,
author = {Domínguez-Díaz, Adrián and Goyanes, Manuel and de-Marcos, Luis and Prado-Sánchez, Víctor Pablo},
title = {Comparative analysis of automatic gender detection from names: evaluating the stability and performance of ChatGPT versus Namsor, and Gender-API},
journal = {PeerJ Computer Science},
year = {2024},
volume = {10},
pages = {e2378},
doi = {10.7717/peerj-cs.2378}
}
EUVIMED