EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

An Invariant Measure for Differential Entropy: From Kullback–Leibler Divergence to Scale-Invariant Information Theory

Félix Truong iD, Alexandre Giuliani iD

Entropy · 07.03.2026 · 28 (3) · 301

DOI: 10.3390/e28030301 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Entropy, online erschienen am 07.03.2026. Verfasst von Félix Truong und 1 weiteren Autor:innen. Beteiligte Einrichtung: Synchrotron SOLEIL, L'Orme des Merisiers, 91190 Saint-Aubin, France. Die Arbeit nennt 16 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Synchrotron SOLEIL and INRAE.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Shannon’s differential entropy for continuous variables suffers from a fundamental limitation: it is not invariant under scale transformations. This makes entropy values dependent on the choice of measurement units rather than reflecting intrinsic properties of distributions. While Jaynes proposed the limiting density of discrete points (LDDP) as a theoretical solution, a concrete method for computing the required invariant measure has been lacking. This paper establishes a rigorous connection between Kullback–Leibler divergence and the invariant measure, providing theoretical proofs of invariance under affine transformations and a practical computational method. We prove that entropy normalized by the median of k-nearest neighbor distances is invariant under affine transformations (Theorems 1 and 2). The non-negativity of the resulting entropy has been validated empirically across all tested distribution families, though a complete theoretical proof remains an open question. This approach extends naturally to multivariate settings, enabling scale-invariant mutual information estimation. We provide open-source implementations in Julia (EntropyInvariant.jl) and Python (entropy_invariant) and demonstrate their advantages over traditional approaches, particularly for variables with disparate scales.

Schlagwörter

Stichwörter der Autor:innen

Kullback–Leibler divergencedifferential entropyinvariant measurek-nearest neighborlimiting density of discrete pointsmutual informationscale invariance

Autor:innen und Institutionen

  • Synchrotron SOLEIL, L'Orme des Merisiers, 91190 Saint-Aubin, France.
  • UAR1008, Transform Department, INRAE, 44316 Nantes, France.

ORCID-Profile: Félix Truong, Alexandre Giuliani

Förderung

  • Synchrotron SOLEIL and INRAE

Zitieren

APA

Truong, F., & Giuliani, A. (2026). An Invariant Measure for Differential Entropy: From Kullback–Leibler Divergence to Scale-Invariant Information Theory. Entropy, 28(3), 301. https://doi.org/10.3390/e28030301

Vancouver

Truong F, Giuliani A. An Invariant Measure for Differential Entropy: From Kullback–Leibler Divergence to Scale-Invariant Information Theory. Entropy. 2026;28(3):301. doi:10.3390/e28030301

BibTeX
@article{Truong2026,
  author = {Truong, Félix and Giuliani, Alexandre},
  title = {An Invariant Measure for Differential Entropy: From Kullback–Leibler Divergence to Scale-Invariant Information Theory},
  journal = {Entropy},
  year = {2026},
  volume = {28},
  number = {3},
  pages = {301},
  doi = {10.3390/e28030301}
}
Kostenlos starten →