Statistical Divergences for Learning and Inference: Limit Laws and Non-Asymptotic Bounds
Bibliographic record
Abstract
Statistical divergences have been widely used in statistics and artificial intelligence to measure the dissimilarity between probability distributions. The applications range from generative modeling to statistical inference. Early works in statistics have focused on discrete and low-dimensional probability distributions. We choose to tackle problems emerging in modern applications of statistics and artificial intelligence in which the sample space is either discrete with a large alphabet (e.g., natural language processing) or continuous of high dimension (e.g., computer vision). This dissertation revisits statistical divergences in modern applications and addresses challenges arising from the complex nature of the data. Chapter 2 studies the minimum Kullback-Leibler divergence estimation which is equivalent to the widely used maximum likelihood estimation. While the classical asymptotic theory is well established in a rather general setting, high-dimensional problems reveal several of its limitations. We develop finite-sample bounds characterizing the asymptotic behavior in a non-asymptotic fashion, allowing the dimension to grow with the sample size. Unlike previous work that relies heavily on the strong convexity of the objective function, we only assume the Hessian is lower bounded at optimum and allow it to gradually become degenerate. This is enabled by the notion of self-concordance originating from convex optimization. Chapter 3 investigates the framework of divergence frontiers, a notion of trade-off curves built upon statistical divergences, for comparing generative models. These trade-off curves are analogous to operating characteristic curves in statistical decision theory. Due to the complex and high-dimensional nature of the input space, an effective approach used by practitioners to estimate divergence frontiers involves a quantization step followed by an estimation step. We establish non-asymptotic bounds on the sample complexity of this estimator. We also show how smoothed distribution estimators such as Good-Turing or Krichevsky-Trofimov can overcome the missing mass problem and lead to faster rate of convergence. Chapters 4 and 5 explore the Schrödinger bridge problem---an information projection problem which projects a reference measure onto a linear subspace of probability distributions in terms of the Kullback-Leibler divergence. This problem is equivalent to the entropy-regularized optimal transport problem that recently attracted a huge attention from the statistics and machine learning communities. We develop limit laws and non-asymptotic bounds for its empirical estimators. Unlike the unregularized optimal transport, our results enjoy a parametric rate of convergence that does not suffer from the curse of dimensionality. We also propose statistical tests for testing homogeneity and independence based on the Schrödinger bridge problem.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.016 | 0.115 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.002 |
| Bibliometrics | 0.004 | 0.004 |
| Science and technology studies | 0.001 | 0.006 |
| Scholarly communication | 0.005 | 0.012 |
| Open science | 0.003 | 0.005 |
| Research integrity | 0.003 | 0.011 |
| Insufficient payload (model declined to judge) | 0.005 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".