MDA for random forests: inconsistency, and a practical solution via the\n Sobol-MDA
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Variable importance measures are the main tools to analyze the black-box\nmechanisms of random forests. Although the mean decrease accuracy (MDA) is\nwidely accepted as the most efficient variable importance measure for random\nforests, little is known about its statistical properties. In fact, the\ndefinition of MDA varies across the main random forest software. In this\narticle, our objective is to rigorously analyze the behavior of the main MDA\nimplementations. Consequently, we mathematically formalize the various\nimplemented MDA algorithms, and then establish their limits when the sample\nsize increases. This asymptotic analysis reveals that these MDA versions differ\nas importance measures, since they converge towards different quantities. More\nimportantly, we break down these limits into three components: the first two\nterms are related to Sobol indices, which are well-defined measures of a\ncovariate contribution to the response variance, widely used in the sensitivity\nanalysis field, as opposed to the third term, whose value increases with\ndependence within covariates. Thus, we theoretically demonstrate that the MDA\ndoes not target the right quantity to detect influential covariates in a\ndependent setting, a fact that has already been noticed experimentally. To\naddress this issue, we define a new importance measure for random forests, the\nSobol-MDA, which fixes the flaws of the original MDA, and consistently\nestimates the accuracy decrease of the forest retrained without a given\ncovariate, but with an efficient computational cost. The Sobol-MDA empirically\noutperforms its competitors on both simulated and real data for variable\nselection. An open source implementation in R and C++ is available online.\n
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.004 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it