EUVIMEDEUVIMED entdecken
Open Access

Testing high-dimensional multinomials with applications to text analysis

T Tony Cai, Zheng T Ke iD, Paxton Turner iD

Journal of the Royal Statistical Society Series B: Statistical Methodology · 13.09.2024 · 86 (4) · 922-942

DOI: 10.1093/jrsssb/qkae003 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the Royal Statistical Society Series B: Statistical Methodology, online erschienen am 28.02.2024. Verfasst von T Tony Cai und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Statistics and Data Science, University of Pennsylvania, Philadelphia, PA, USA. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 28 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIGMS NIH HHS, NSF, NIH.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Motivated by applications in text mining and discrete distribution inference, we test for equality of probability mass functions of K groups of high-dimensional multinomial distributions. Special cases of this problem include global testing for topic models, two-sample testing in authorship attribution, and closeness testing for discrete distributions. A test statistic, which is shown to have an asymptotic standard normal distribution under the null hypothesis, is proposed. This parameter-free limiting null distribution holds true without requiring identical multinomial parameters within each group or equal group sizes. The optimal detection boundary for this testing problem is established, and the proposed test is shown to achieve this optimal detection boundary across the entire parameter space of interest. The proposed method is demonstrated in simulation studies and applied to analyse two real-world datasets to examine, respectively, variation among customer reviews of Amazon movies and the diversity of statistical paper abstracts.

Schlagwörter

Stichwörter der Autor:innen

authorship attributioncloseness testingcustomer reviewsmartingale central limit theoremminimax optimalitytopic model

Autor:innen und Institutionen

  • Department of Statistics and Data Science, University of Pennsylvania, Philadelphia, PA, USA.
  • Department of Statistics, Harvard University, Cambridge, MA, USA.

ORCID-Profile: Zheng T Ke, Paxton Turner

Förderung

  • NIGMS NIH HHS
  • NSF
  • NIH

Zitieren

APA

Cai, T. T., Ke, Z. T., & Turner, P. (2024). Testing high-dimensional multinomials with applications to text analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(4), 922-942. https://doi.org/10.1093/jrsssb/qkae003

Vancouver

Cai TT, Ke ZT, Turner P. Testing high-dimensional multinomials with applications to text analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology. 2024;86(4):922-942. doi:10.1093/jrsssb/qkae003

BibTeX
@article{Cai2024,
  author = {Cai, T Tony and Ke, Zheng T and Turner, Paxton},
  title = {Testing high-dimensional multinomials with applications to text analysis},
  journal = {Journal of the Royal Statistical Society Series B: Statistical Methodology},
  year = {2024},
  volume = {86},
  number = {4},
  pages = {922--942},
  doi = {10.1093/jrsssb/qkae003}
}
Kostenlos starten →