Statistical inference on representational geometries
Bibliographic record
Abstract
Models and brain measurements of visual processing have substantially increased in complexity in recent years. Summarizing and comparing their high-dimensional representations to each other requires specialized statistical methods. Here, we introduce new inference methods to evaluate models based on their predicted representational geometries, i.e. based on how well they match the distances or dissimilarities between the representations. Our inference methods are based on cross-validation and bootstrapping. We introduce a novel 2-factor bootstrap technique wrapped around a cross-validation procedure with analytically derived adjustments for the biases induced by 2-factor inflation of measurement noise and the choice of cross-validation folds. We validate our new inference methods using extensive simulations. We first simulate fMRI-like data based on local averages of deep neural network activations for images sampled from ecoset. In these simulations, we have full access to the true data generating process and can thus test a wide range of experiments. Additionally, we performed simulations based on subsampling data from large scale calcium imaging and fMRI experiments. These simulations are less flexible, but we are more confident that the patterns and their variability are representative of true experimental data. In all simulations, our new methods yield good estimates of the variance of model evaluations and thus valid statistical tests. In contrast, uncorrected bootstrap methods substantially overestimate variance and thus yield overly conservative tests. Also, ignoring the desired generalization to new stimuli leads to underestimated variance and thus to overly liberal tests. Similar statistical problems occur whenever other bootstrap methods aim to generalize to new stimuli and new subjects and/or are combined with cross-validation. Our new methods are available as part of the open source rsatoolbox in python at https://github.com/rsagroup/rsatoolbox.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.016 | 0.094 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.003 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.001 | 0.005 |
| Scholarly communication | 0.003 | 0.006 |
| Open science | 0.003 | 0.004 |
| Research integrity | 0.002 | 0.005 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".