Of Parachutes and Participant Protection: Moving Beyond Quality to Advance Effective Research Ethics Oversight
Bibliographic record
Abstract
There are several reasons to believe that Institutional Review Boards (IRBs) and Human Research Protection Programs (HRPPs) contribute to ethical research and the protection of research participants, but there are also important reasons to interrogate this belief. Determining whether IRBs and HRPPs "work" requires empirical evaluation of whether and how well they actually achieve what they were designed to do. In other words, it is critical to examine their outcomes and not only their procedures and structures. In this response to Tsan, we argue that the concept of IRB and HRPP quality entails three dimensions: (1) effectiveness, (2) procedures and structures likely to promote effectiveness, and (3) features unrelated to effectiveness but nonetheless essential, such as efficiency, fairness, and proportionality. Because not all types of quality necessarily guarantee or entail effectiveness, we suggest that broad quality assessments, including such features as regulatory compliance and other procedural measures suggested by Tsan, are unhelpful as the first step in evaluating IRBs and HRPPs. Instead, we must start with outcomes relevant to effectiveness. To do this, we launched the Consortium to Advance Effective Research Ethics Oversight (AEREO), with a mission to define and specify ways to measure relevant outcomes for research ethics oversight, empirically evaluate whether those outcomes are achieved, test new approaches to achieving them, and ultimately, develop and implement empirically-based policy and practice to advance IRB and HRPP effectiveness. We describe several anticipated AEREO projects and call for collaboration between various stakeholders to more meaningfully evaluate IRB and HRPPs.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Direct model labels (unvalidated)
Per-model category and study-design labels from the labeling rounds. They are machine output, unvalidated, and the disagreement between models ships as data. No study design here is MEDLINE-validated yet.
| Model arm | Categories | Study design | Confidence |
|---|---|---|---|
| gemma | Metaresearch Domain: Evaluation · Genre: Empirical About the Canadian research system: no · About a Canadian topic: no | Qualitative | low |
| gpt | Metaresearch Domain: Methods · Genre: Other About the Canadian research system: no · About a Canadian topic: no | Theoretical or conceptual | low |
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.671 | 0.666 |
| Meta-epidemiology (narrow) | 0.001 | 0.002 |
| Meta-epidemiology (broad) | 0.003 | 0.003 |
| Bibliometrics | 0.006 | 0.006 |
| Science and technology studies | 0.012 | 0.090 |
| Scholarly communication | 0.034 | 0.062 |
| Open science | 0.011 | 0.034 |
| Research integrity | 0.024 | 0.038 |
| Insufficient payload (model declined to judge) | 0.009 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedLabeled directly by 2 models reading the full record.
The models disagree on parts of this classification; every voice is preserved in the section at the end of the page.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".