EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

High‐Dimensional Covariance Estimation From a Small Number of Samples

David Vishny, Matthias Morzfeld, Kyle Gwirtz, Eviatar Bach, Oliver R. A. Dunbar, Daniel Hodyss

Journal of Advances in Modeling Earth Systems · 2024-09 · 16 (9)

DOI: 10.1029/2024ms004417 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Advances in Modeling Earth Systems, online erschienen am 30.08.2024. Verfasst von David Vishny und 5 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 97 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Science Foundation, Heising-Simons Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We synthesize knowledge from numerical weather prediction, inverse theory, and statistics to address the problem of estimating a high‐dimensional covariance matrix from a small number of samples. This problem is fundamental in statistics, machine learning/artificial intelligence, and in modern Earth science. We create several new adaptive methods for high‐dimensional covariance estimation, but one method, which we call Noise‐Informed Covariance Estimation (NICE), stands out because it has three important properties: (a) NICE is conceptually simple and computationally efficient; (b) NICE guarantees symmetric positive semi‐definite covariance estimates; and (c) NICE is largely tuning‐free. We illustrate the use of NICE on a large set of Earth science–inspired numerical examples, including cycling data assimilation, inversion of geophysical field data, and training of feed‐forward neural networks with time‐averaged data from a chaotic dynamical system. Our theory, heuristics and numerical tests suggest that NICE may indeed be a viable option for high‐dimensional covariance estimation in many Earth science problems.

Förderung

  • National Science Foundation
  • Heising-Simons Foundation

Zitieren

APA

Vishny, D., Morzfeld, M., Gwirtz, K., Bach, E., Dunbar, O. R. A., & Hodyss, D. (2024). High‐Dimensional Covariance Estimation From a Small Number of Samples. Journal of Advances in Modeling Earth Systems, 16(9). https://doi.org/10.1029/2024ms004417

Vancouver

Vishny D, Morzfeld M, Gwirtz K, Bach E, Dunbar ORA, Hodyss D. High‐Dimensional Covariance Estimation From a Small Number of Samples. Journal of Advances in Modeling Earth Systems. 2024;16(9). doi:10.1029/2024ms004417

BibTeX
@article{Vishny2024,
  author = {Vishny, David and Morzfeld, Matthias and Gwirtz, Kyle and Bach, Eviatar and Dunbar, Oliver R. A. and Hodyss, Daniel},
  title = {High‐Dimensional Covariance Estimation From a Small Number of Samples},
  journal = {Journal of Advances in Modeling Earth Systems},
  year = {2024},
  volume = {16},
  number = {9},
  doi = {10.1029/2024ms004417}
}
Kostenlos starten →