Computational reproducibility: A simplified framework for data curators
Bibliographic record
Abstract
Phrases like the 'data deluge' and the 'reproducibility crisis' may serve to further the impression that data curation is hard and that research data management is "basically fighting against chaos" (Briney, 2019). If trying to manage research data is chaotic, then the management of computationally-derived data presents an even bigger challenge due to the multiplicity of operating systems, coding languages, dependencies, and file types. This is exacerbated by the reality that most researchers and librarians are not formally trained as programmers. As data curators and managers, we may need to reconsider if the complete replication of research, especially computational research, is a realistic goal in all instances. Can curated data still be useful if it is only 'a little bit' reproducible or 'just about' reproducible? This purpose of this presentation is to propose an approach of 'just enough' data curation by arguing that partial reproducibility is better than nothing at all (Broman, n.d.). By focusing on incremental progress rather than prescriptive rules, researchers and curators can build their knowledge and skills as the need arises. A computational reproducibility framework, developed for the Canadian Data Curation Forum, will serve as the model for this approach, which combines learning about reproducibility with improving reproducibility. Computational reproducibility leads to better and more transparent research, but fear of a crisis and focus on perfection shouldn't prevent curation that may be 'good enough.' This presentation will discuss concrete and actionable steps to help researchers, data curators, and data managers improve their understanding and practice of computational reproducibility. Briney, K. (2019). Data management is hard and everyone is bad at it. This includes data managers more often that we care to admit. Data management is basically fighting against chaos.https://twitter.com/mykola/status/1198719315589160960 … [Tweet]. Broman, K. (n.d.). Initial steps toward reproducible research. https://kbroman.org/steps2rr/.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.325 | 0.442 |
| Meta-epidemiology (narrow) | 0.003 | 0.005 |
| Meta-epidemiology (broad) | 0.005 | 0.008 |
| Bibliometrics | 0.019 | 0.015 |
| Science and technology studies | 0.016 | 0.038 |
| Scholarly communication | 0.033 | 0.049 |
| Open science | 0.019 | 0.034 |
| Research integrity | 0.013 | 0.018 |
| Insufficient payload (model declined to judge) | 0.011 | 0.009 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".