Ruodu Wang's contribution to the Discussion of ‘Estimating means of bounded random variables by betting’ by Waudby-Smith and Ramdas
Bibliographic record
Abstract
I congratulate Ian Waudby-Smith and Aaditya Ramdas for this excellent contribution to the theory of e-testing (or testing by betting). Over the past several years, this theory has been developed rapidly in multiple exciting directions. The contribution of Waudby-Smith and Ramdas is an important piece in this active stream of literature. I have a few comments on the approach in the current paper and its generalisations. The main approach taken in the paper (see its Theorem 1), which is standard in the field of game-theoretical statistics (Shafer & Vovk, 2019), relies on constructing an e-process (Mtθ)t∈T for each parameter value θ∈Θ (which is the mean m in the paper). The e-process (Mtθ)t∈T is often (but not always) constructed by combining several sequential e-variables Eθ=(Etθ)t∈T from the data (i.e. satisfying Eθ[Etθ|Ft−1]≤1 for each t∈T and θ∈Θ), via a method of martingale: Mtθ=∏s=1t(1−λsθ(Esθ−1)),t∈T, where λ=(λtθ)t∈T is a predictable process. The abstract problem of combining sequential e-variables is studied in Vovk and Wang (2022), where it is shown that the above martingale method is the only admissible way to combine sequential e-variables into one e-variable. Therefore, anytime validity (i.e. validity under optional sampling) is obtained automatically if the goal is to make a decision based on a combined e-value. Although the above method of e-testing is by now standard and its validity is easy to show, the highly non-trivial tasks are to build suitable Eθ and to find powerful λθ. The validity is guaranteed even when the data-generating procedure varies arbitrarily over time, as long as the parameter of interest (mean m in this paper) in the null hypothesis is specified. Nevertheless, the power and optimality of λθ depend crucially on how data are generated. Most methods (such as GRAPA and aGRAPA introduced in the current paper) use sample mean, sample variance, or the empirical distribution to decide λθ. This requires some ‘stationarity’, ‘predictability’, or ‘temporal structure’ of the data. In some applications involving dynamic decision making (data depend on previous decisions), such as backtesting financial risk prediction, such stationarity cannot be assumed. In this context, some options of powerful betting strategies are studied by Wang et al. (2022). The e-testing approach can be applied to many other quantities in a model-free fashion, similar to the mean with bounded supported treated in this paper. Wang et al. (2022) developed one-sided e-tests for other quantities, including mean (with one-side bounded support), variance, quantile, and the risk measure Expected Shortfall. A main advantage of such methods is that they do not assume any knowledge of the data-generating probability or its temporal structure.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.015 | 0.080 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.003 |
| Bibliometrics | 0.003 | 0.005 |
| Science and technology studies | 0.002 | 0.006 |
| Scholarly communication | 0.004 | 0.007 |
| Open science | 0.005 | 0.005 |
| Research integrity | 0.009 | 0.011 |
| Insufficient payload (model declined to judge) | 0.004 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".