About MétaCan

The frame flip

The usual way to map a field is to retrieve what looks like the field — a keyword list, a topic classifier, a journal set — and then ask which of the results are Canadian. This makes the field's boundary a property of your query. And it has a fatal property for anyone who wants to check your work: you cannot measure the recall of a lexicon against the works the lexicon never showed you. The misses are invisible by construction, so the map cannot report its own error, so it cannot be audited.

So this project inverts the frame. It starts from all Canadian research — an external, checkable criterion, enumerable from a pinned OpenAlex snapshot — and makes field membership a classification over a known universe rather than a retrieval over the literature. Once the universe is enumerable, recall becomes measurable, a sample has known selection probabilities, and a disagreement between screeners becomes a finding instead of an embarrassment.

The cost is that “Canadian” must itself be defined, and it is: by four routes — affiliation, funder, venue, and subject — each recorded on every work. A frame that forgets how it found something cannot be audited either, so every row on this site carries its provenance. Browse the works no affiliation-only frame would ever have seen.

Why the deliverable is a disagreement

Three frontier models screened the same 1,000 works against the same locked rubric. They did not agree. Of the works any model called metaresearch, only about a third were called metaresearch by all three, and nearly half rest on a single model's opinion.

The tempting move is to average this away and publish a base rate. That would be false precision, and worse, it would be hiding the only interesting thing the experiment found. Two screeners can agree on a rate while finding almost entirely different works: at a ~1% base rate, the settled rejects buy 98% agreement for free. So the deliverable is not a number. It is the disagreement dossier: the works that mark the field's empirical boundary, and against which inclusion criteria actually have to be written.

What the data cannot say

Three limits are measured rather than hedged, because a limit you have measured is a finding and a limit you have merely acknowledged is an excuse.

• The abstract gap is structural

  The screen's largest measured bias is the abstract gap: 23.3% of the frame (1,003,117 works) has NO ABSTRACT, and finding 11 showed the screen finds HALF as much metaresearch there. Cascading PubMed, Europe PMC and Crossref recovers 37.8% of a 500-work sample, cutting title-only exposure to ~14.5% of the frame. But I BUILT THE CASCADE AROUND CROSSREF as the discipline-agnostic rescue, and it recovered 2 abstracts against PubMed's 180: publishers do not deposit them, so THAT RESCUE DOES NOT EXIST (D15). The gap is therefore not a metadata failure a better index fixes; it is STRUCTURAL. Recovery is 91.2% for reviews against 6.2% for book chapters, 38.8% English against 15.4% French. So the tempting shortcut, 'just screen the works that have abstracts', is a SELECTION ON A COVARIATE THAT PREDICTS THE OUTCOME which would delete 61.6% of book chapters against 22.1% of articles, AND the works it deletes are exactly the works no cascade can rescue. Defensible only as a DECLARED exclusion with a measured cost, and the audit keeps a sampling floor in it.

• Retraction is not a boolean

  Joined to the Canadian frame by DOI, Retraction Watch records 143 works that OpenAlex does NOT flag as retracted, including 49 outright retractions. But the undercount is the smaller problem. 52 of these carry an EXPRESSION OF CONCERN, and OpenAlex HAS NO FIELD FOR ONE: `is_retracted` is a boolean over a state space with at least four values (retraction, expression of concern, correction, reinstatement), so it can express one and silently reports the rest as FALSE, which reads as 'fine'. Nor can a boolean carry WHY. This is finding 1's disease in a second schema: the canonical database cannot express the distinction the field turns on.

• Swap the screener, move the answer

  Swap which model is called 'the screener' and the base rate moves from 1.06% to 2.37%: a 2.2x spread, from 37,032 to 83,022 works in the frame. The two screeners agree on in/out for 98.4% of the frame (design-weighted), but that figure is dominated by the settled rejects: agreement falls to 95% inside the contested boundary. THE SCREENER-SWAP RANGE, NOT THE BINOMIAL CI ON EITHER MODEL ALONE, IS THE HONEST UNCERTAINTY ON THE FIELD'S SIZE.

All 22 findings, rendered from the pipeline’s own output →

The errors

This project records its own mistakes. DEVIATIONS.md lists them — deviations from the protocol, defects in the screening harness, an estimator that had to be cut rather than dressed up as a lower bound, a rescue path built around a source that turned out not to deposit the data at all. They were written down as they happened and before submission, not reconstructed afterwards.

This is not humility for its own sake. The whole argument of MétaCan is that a map which cannot report its own error is not a map you can trust. A project making that argument while quietly polishing its own record would refute itself in the act of publishing. So: the capture-recapture estimator implied Canada produces 59% of the world's metaresearch, which is absurd, and it was cut, not softened. The abstract cascade was built around Crossref as the discipline-agnostic rescue, and Crossref recovered 2 abstracts against PubMed's 180 — so that rescue does not exist, and the gap is structural. GPT-5.6 violated the locked output schema on 18 of 1,000 records; the manifest validator caught it, and it is recorded rather than silently repaired.

Method, in short

Frame

Every Canadian work in a pinned OpenAlex snapshot (all 482 partitions), each work exactly once, admitted by one or more of four routes: Canadian affiliation, Canadian funder, Canadian venue, or subject-about-Canada.

Screen

1,000 works drawn with known selection probabilities, stratified (French oversampled), screened by Claude Opus 4.8, GPT-5.6 (high) and Grok 4.5 against one locked rubric on its full eight-field payload. Chunks were randomized and manifest-logged before any model ran, and the harness writes the label files — never the model.

Weights

The sample is stratified, so every rate is design-weighted. A rate computed from the raw sample without the weight is wrong, and the weight ships with every screened record in the API.

Abstracts

Not stored. The inverted indexes are 8.6 GB of the frame's 9.3 GB of text and the host has 13 GB free, so the detail page fetches an abstract live from OpenAlex. Whether a work has one is stored, because that is itself a finding.

Retraction

Joined to Retraction Watch by DOI, and kept in its own table with four states, because OpenAlex's is_retracted is a boolean over a state space that has at least four values.

Reproducibility

Every number on this site is produced by a script in the repository and read from findings.json. Nothing is typed by hand, so the site and the analysis cannot drift apart.

Sources, licence, contact

Data: OpenAlex (CC0), Retraction Watch (CC-BY), ClinicalTrials.gov. Code MIT, data CC-BY-4.0. Built by Ahmad Sofi-Mahmudi for the Canadian Metaresearch Data Challenge (Canadian Reproducibility Network).

The repository · DEVIATIONS.md · the public API