EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

PyRadar: Towards Automatically Retrieving and Validating Source Code Repository Information for PyPI Packages

Kai Gao, Weiwei Xu, Wenhao Yang, Minghui Zhou

Proceedings of the ACM on Software Engineering · 12.07.2024 · 1 (FSE) · 2608-2631

DOI: 10.1145/3660822 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 12.07.2024. Verfasst von Kai Gao und 3 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 74 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

A package’s source code repository records the package’s development history, which is critical for the use and risk monitoring of the package. However, a package release often misses its source code repository due to the separation of the package’s development platform from its distribution platform. To establish the link, existing tools retrieve the release’s repository information from its metadata, which suffers from two limitations: the metadata may not contain or contain wrong information. Our analysis shows that existing tools can only retrieve repository information for up to 70.5% of PyPI releases. To address the limitations, this paper proposes PyRadar, a novel framework that utilizes the metadata and source distribution to retrieve and validate the repository information for PyPI releases. We start with an empirical study to compare four existing tools on 4,227,425 PyPI releases and analyze phantom files (files appearing in the release’s distribution but not in the release’s repository) in 14,375 correct and 2,064 incorrect package-repository links. Based on the findings, we design PyRadar with three components, i.e., Metadata-based Retriever, Source Code Repository Validator, and Source Code-based Retriever, that progressively retrieves correct source code repository information for PyPI releases. In particular, the Metadata-based Retriever combines best practices of existing tools and successfully retrieves repository information from the metadata for 72.1% of PyPI releases. The Source Code Repository Validator applies common machine learning algorithms on six crafted features and achieves an AUC of up to 0.995. The Source Code-based Retriever queries World of Code with the SHA-1 hashes of all Python files in the release’s source distribution and retrieves repository information for 90.2% of packages in our dataset with an accuracy of 0.970. Both practitioners and researchers can employ the PyRadar to better use PyPI packages.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Gao, K., Xu, W., Yang, W., & Zhou, M. (2024). PyRadar: Towards Automatically Retrieving and Validating Source Code Repository Information for PyPI Packages. Proceedings of the ACM on Software Engineering, 1(FSE), 2608-2631. https://doi.org/10.1145/3660822

Vancouver

Gao K, Xu W, Yang W, Zhou M. PyRadar: Towards Automatically Retrieving and Validating Source Code Repository Information for PyPI Packages. Proceedings of the ACM on Software Engineering. 2024;1(FSE):2608-2631. doi:10.1145/3660822

BibTeX
@article{Gao2024,
  author = {Gao, Kai and Xu, Weiwei and Yang, Wenhao and Zhou, Minghui},
  title = {PyRadar: Towards Automatically Retrieving and Validating Source Code Repository Information for PyPI Packages},
  journal = {Proceedings of the ACM on Software Engineering},
  year = {2024},
  volume = {1},
  number = {FSE},
  pages = {2608--2631},
  doi = {10.1145/3660822}
}
Kostenlos starten →