EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A New Estimator of Kullback–Leibler Divergence via Shannon Entropy

Mehmet Sıddık Çadırcı iD, Martin Singull iD

Entropy · 24.06.2026 · 28 (7) · 720

DOI: 10.3390/e28070720 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Entropy, online erschienen am 24.06.2026. Verfasst von Mehmet Sıddık Çadırcı und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Statistics, Faculty of Science, Cumhuriyet University, 58140 Sivas, Türkiye. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We examine the estimation of the Kullback–Leibler (KL) divergence and the use of the goodness-of-fit test for multivariate normality. Our starting point is the maximum entropy principle for Shannon entropy: among all distributions with a fixed mean vector and covariance matrix, the multivariate Gaussian distributions uniquely maximize entropy. As a result, the KL divergence from a moment-matched Gaussian distribution to an unknown density can then be written as the entropy difference, which is a suitable information-theoretic measure of divergence from the Gaussian distribution. To estimate, we use k-nearest neighbor (kNN) estimators based on Shannon entropy and KL divergence derived from the Kozachenko–Leonenko approach and subsequent improvements, along with the consistency and L2-convergence results established for these estimators. Motivated by previous entropy-based goodness-of-fit ideas developed for Rényi-type functionals for generalized Gaussian and Student-type models, we describe a KL-based test statistic as being the difference between the entropy of a Gaussian model fitted to the sample mean and covariance and the KL divergence between the unknown entropy and the kNN estimate. The statistic converges to zero for multivariate normality and converges to a strictly positive bound with non-Gaussian alternatives. The results of Monte Carlo simulations conducted across various dimensions and sample sizes indicate that the proposed method provides accurate Type I error control among the alternatives considered and demonstrates promising empirical power.

Schlagwörter

Stichwörter der Autor:innen

Kullback–Leibler divergenceShannon entropygoodness-of-fit testkNN estimationmultivariate normality

Autor:innen und Institutionen

  • Department of Statistics, Faculty of Science, Cumhuriyet University, 58140 Sivas, Türkiye.
  • Department of Mathematics, Linköping University, 581 83 Linköping, Sweden.

ORCID-Profile: Mehmet Sıddık Çadırcı, Martin Singull

Zitieren

APA

Çadırcı, M. S., & Singull, M. (2026). A New Estimator of Kullback–Leibler Divergence via Shannon Entropy. Entropy, 28(7), 720. https://doi.org/10.3390/e28070720

Vancouver

Çadırcı MS, Singull M. A New Estimator of Kullback–Leibler Divergence via Shannon Entropy. Entropy. 2026;28(7):720. doi:10.3390/e28070720

BibTeX
@article{adrc2026,
  author = {Çadırcı, Mehmet Sıddık and Singull, Martin},
  title = {A New Estimator of Kullback–Leibler Divergence via Shannon Entropy},
  journal = {Entropy},
  year = {2026},
  volume = {28},
  number = {7},
  pages = {720},
  doi = {10.3390/e28070720}
}
Kostenlos starten →