Analysis of the rate of convergence of two regression estimates defined by neural features which are easy to implement
Bibliographic record
Abstract
Recent results in nonparametric regression have shown that neural network regression estimates with many hidden layers are able to achieve good rates of convergence even in case of high-dimensional predictor variables, provided suitable assumptions on the structure of the regression function are imposed. In those recent results, the estimates were defined by minimizing the empirical L2 risk over a class of neural networks. In practice, however, it is not clear how this can be done exactly. In this article, motivated by some recent approximation results for neural networks, we introduce two new regression estimates defined by neural features where most of the neural network weights are chosen via random initialization and no training, thus sparing the costly data-dependent optimization. For the first estimate, which is defined by these neural features and an extra layer whose weights are set via least squares, we derive rates of convergence results in case the regression function is smooth. We then combine this estimate with the projection pursuit, where we choose the directions randomly, and we show that for sufficiently many repetitions we get a second regression estimate which achieves the one-dimensional rate of convergence (up to some logarithmic factor) in case that the regression function satisfies the assumptions of projection pursuit. Because the neural features are obtained by random initialization but not training of the weights, the two estimators thus defined are easy to implement.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.026 | 0.160 |
| Meta-epidemiology (narrow) | 0.003 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.001 | 0.004 |
| Scholarly communication | 0.002 | 0.005 |
| Open science | 0.002 | 0.004 |
| Research integrity | 0.003 | 0.005 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".