Excess significance and power miscalculations in neurofeedback research
Bibliographic record
Abstract
Recent systematic reviews of neurofeedback with functional magnetic resonance imaging (fMRI-nf) (Tursic et al., 2020) and neurofeedback with functional near infrared spectroscopy (fNIRS-nf) (Kohl et al., 2020) miscalculate the statistical power and statistical sensitivity of several studies they review.The fMRI-nf review overestimates the mean and median statistical power of included studies by about 3 times and the statistical sensitivity by about 2 times (see Table 1 for recalculated values and comparisons).The fNIRS-nf review, on which I (RTT) was a coauthor, overestimates power by about 2 times and sensitivity by about 1.5 times (see Table 2).The miscalculations arise from an easy-to-miss default option for repeated measures (mixed) 1 ANOVAs in the statistical software program GPower (Faul et al., 2007), which both reviews used (see Figure 1 for a depiction) 2 .The default option defines a variable in the effect size calculation η 2 p in such a way that the common usage of small, medium, and large effects sizes for the interaction of repeated measures (mixed) ANOVAs (f ) doesn't hold true.If unaware of the default option, the power calculations will account for the correlations between repeated measures a second time, and in turn substantially-but erroneously-increase power.The GPower software itself highlights that Cohen (1988) recommended another option (as viewable in Figure 1).While Lakens (2013) explained this issue almost 10 years ago, it remains likely that researchers continue to use GPower without awareness of this default option and its implications 3 .Fortunately, the authors of both reviews published their data as supplementary material, making reanalysis possible.We recalculated the statistical power and sensitivity of the studies from Tursic et al. ( 2020) and Kohl et al. 1 We use the term "repeated measures (mixed) ANOVA" to describe a study design with one independent measure and one repeated measure (e.g., two independent groups measured at two time points).GPower uses the term "ANOVA: Repeated measures" for this type of design.2 Kohl and coauthors (including me-RTT) were unaware of this default option in GPower.I was only led to become aware of this issue when reading the fMRI-nf review and noticing a surprising amount of statistical power.3 Kieslich (2020) provides a detailed explanation of how GPower calculates f for the different options for repeated measures (mixed) ANOVAs.The GPower team informed us that Cohen proposed Cohen's f for between-subjects designs and that this is how GPower defines f when using the default option, regardless of the ANOVA design the user selects.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.002 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.002 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".