EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Scaling up search engine audits: Practical insights for algorithm auditing

Roberto Ulloa iD, Mykola Makhortykh, Aleksandra Urman

Journal of Information Science · 2024-04 · 50 (2) · 404-419

DOI: 10.1177/01655515221093029 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Information Science, online erschienen am 02.05.2022. Verfasst von Roberto Ulloa und 2 weiteren Autor:innen. Beteiligte Einrichtung: Computational Social Science, GESIS - Leibniz-Institute for the Social Sciences, Germany. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 58 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung, Friends of the Institute of Communication and Media Science at the University of Bern, Deutsche Forschungsgemeinschaft.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Algorithm audits have increased in recent years due to a growing need to independently assess the performance of automatically curated services that process, filter and rank the large and dynamic amount of information available on the Internet. Among several methodologies to perform such audits, virtual agents stand out because they offer the ability to perform systematic experiments, simulating human behaviour without the associated costs of recruiting participants. Motivated by the importance of research transparency and replicability of results, this article focuses on the challenges of such an approach. It provides methodological details, recommendations, lessons learned and limitations based on our experience of setting up experiments for eight search engines (including main, news, image and video sections) with hundreds of virtual agents placed in different regions. We demonstrate the successful performance of our research infrastructure across multiple data collections, with diverse experimental designs, and point to different changes and strategies that improve the quality of the method. We conclude that virtual agents are a promising venue for monitoring the performance of algorithms across long periods of time, and we hope that this article can serve as a basis for further research in this area.

Schlagwörter

Stichwörter der Autor:innen

Algorithm auditingdata collectionsearch engine auditsuser modelling

Autor:innen und Institutionen

  • Computational Social Science, GESIS - Leibniz-Institute for the Social Sciences, Germany.
  • Institute of Communication and Media Studies, University of Bern, Switzerland.
  • Social Computing Group, University of Zurich, Switzerland.

ORCID-Profile: Roberto Ulloa

Förderung

  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
  • Friends of the Institute of Communication and Media Science at the University of Bern
  • Deutsche Forschungsgemeinschaft

Zitieren

APA

Ulloa, R., Makhortykh, M., & Urman, A. (2024). Scaling up search engine audits: Practical insights for algorithm auditing. Journal of Information Science, 50(2), 404-419. https://doi.org/10.1177/01655515221093029

Vancouver

Ulloa R, Makhortykh M, Urman A. Scaling up search engine audits: Practical insights for algorithm auditing. Journal of Information Science. 2024;50(2):404-419. doi:10.1177/01655515221093029

BibTeX
@article{Ulloa2024,
  author = {Ulloa, Roberto and Makhortykh, Mykola and Urman, Aleksandra},
  title = {Scaling up search engine audits: Practical insights for algorithm auditing},
  journal = {Journal of Information Science},
  year = {2024},
  volume = {50},
  number = {2},
  pages = {404--419},
  doi = {10.1177/01655515221093029}
}
Kostenlos starten →