Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri
Journal of Data Mining & Digital Humanities · 2024
DOI: 10.46298/jdmdh.10297 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Data Mining & Digital Humanities, 2024. Verfasst von Graham West und 6 weiteren Autor:innen.
Abstract (Originalsprache)
Performing classification on noisy, crowdsourced image datasets can prove challenging even for the best neural networks. Two issues which complicate the problem on such datasets are class imbalance and ground-truth uncertainty in labeling. The AL-ALL and AL-PUB datasets - consisting of tightly cropped, individual characters from images of ancient Greek papyri - are strongly affected by both issues. The application of ensemble modeling to such datasets can help identify images where the ground-truth is questionable and quantify the trustworthiness of those samples. As such, we apply stacked generalization consisting of nearly identical ResNets with different loss functions: one utilizing sparse cross-entropy (CXE) and the other Kullback-Liebler Divergence (KLD). Both networks use labels drawn from a crowd-sourced consensus. This consensus is derived from a Normalized Distribution of Annotations (NDA) based on all annotations for a given character in the dataset. For the second network, the KLD is calculated with respect to the NDA. For our ensemble model, we apply a k-nearest neighbors model to the outputs of the CXE and KLD networks. Individually, the ResNet models have approximately 93% accuracy, while the ensemble model achieves an accuracy of > 95%, increasing the classification trustworthiness. We also perform an analysis of the Shannon entropy of the various models' output distributions to measure classification uncertainty. Our results suggest that entropy is useful for predicting model misclassifications.
Zitieren
APA
West, G., Swindall, M. I., Keener, B., Player, T., Williams, A. C., Brusuelas, J. H., & Wallin, J. F. (2024). Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri. Journal of Data Mining & Digital Humanities. https://doi.org/10.46298/jdmdh.10297
Vancouver
West G, Swindall MI, Keener B, Player T, Williams AC, Brusuelas JH, et al. Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri. Journal of Data Mining & Digital Humanities. 2024. doi:10.46298/jdmdh.10297
BibTeX
@article{West2024,
author = {West, Graham and Swindall, Matthew I. and Keener, Ben and Player, Timothy and Williams, Alex C. and Brusuelas, James H. and Wallin, John F.},
title = {Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri},
journal = {Journal of Data Mining & Digital Humanities},
year = {2024},
doi = {10.46298/jdmdh.10297}
}
EUVIMED