{"meta":{"count":32,"source":"pilot/results/findings.json","note":"Written by the pilot scripts. The site renders this file; it does not restate it."},"findings":{"abstract_cascade":{"headline":"The screen's largest measured bias is the abstract gap: 23.3% of the frame (1,003,117 works) has NO ABSTRACT, and finding 11 showed the screen finds HALF as much metaresearch there. Cascading PubMed, Europe PMC and Crossref recovers 37.8% of a 500-work sample, cutting title-only exposure to ~14.5% of the frame. But I BUILT THE CASCADE AROUND CROSSREF as the discipline-agnostic rescue, and it recovered 2 abstracts against PubMed's 180: publishers do not deposit them, so THAT RESCUE DOES NOT EXIST (D15). The gap is therefore not a metadata failure a better index fixes; it is STRUCTURAL. Recovery is 91.2% for reviews against 6.2% for book chapters, 38.8% English against 15.4% French. So the tempting shortcut, 'just screen the works that have abstracts', is a SELECTION ON A COVARIATE THAT PREDICTS THE OUTCOME which would delete 61.6% of book chapters against 22.1% of articles, AND the works it deletes are exactly the works no cascade can rescue. Defensible only as a DECLARED exclusion with a measured cost, and the audit keeps a sampling floor in it.","values":{"frame_works":4299418,"frame_works_no_abstract":1003117,"pct_frame_no_abstract":23.3,"pct_dropped_by_type":{"book-chapter":61.6,"letter":52.1,"editorial":43.3,"review":29.5,"article":22.1,"book":21.6,"other":21.1,"report":18.6,"preprint":14,"dataset":8.2,"dissertation":4.3},"pct_dropped_by_language":{"fr":21.6,"en":23.7},"abstracts_only_is_a_selection_on_the_outcome":true,"sampled":500,"sources":"PubMed (Entrez) -> Europe PMC (REST) -> Crossref (REST)","recovered":189,"pct_gap_recovered":37.8,"recovered_by_source":{"pubmed":180,"europepmc":7,"crossref":2},"hypothesis_crossref_would_be_load_bearing":false,"crossref_recovered":2,"europepmc_recovered":7,"pubmed_recovered":180,"no_discipline_agnostic_rescue_exists":true,"the_gap_is_structural_not_a_metadata_failure":true,"recovery_pct_by_type":{"review":91.2,"article":40.5,"preprint":38.5,"book-chapter":6.2,"letter":0},"recovery_pct_english":38.8,"recovery_pct_french":15.4,"residual_pct_frame_title_only":14.5,"caveat":"Run on a 500-work hash-ordered sample of the no-abstract stratum, not the frame: the cascade is rate-limited and a million lookups is days. The recovery rate is an estimate with sampling error, and it is an estimate of a CEILING (an abstract that EXISTS is recoverable; it does not follow the screen then classifies the work correctly). Only works with a DOI can be looked up, so the DOI-less part of the stratum is untouched and its size bounds what any cascade can do. PubMed and Europe PMC are biomedical; Crossref is not, and it is in the chain for exactly that reason: a cascade of biomedical indexes would close the gap unevenly and make the residual bias MORE discipline-shaped while appearing to improve coverage. That reasoning was right and the remedy is not available: Crossref recovered 2 of 189 and Europe PMC 7, because publishers largely do not deposit abstracts to Crossref, so no discipline-agnostic rescue exists (D15). Restricting screening to abstract-bearing works remains a DECLARED EXCLUSION with a measured cost, not a scoping convenience, and the audit keeps a sampling floor in the excluded stratum so that cost stays estimable."},"computed_at_utc":"2026-07-13T11:55:17Z"},"active_learning":{"headline":"THE BATCH-OF-100 LOOP WORKS, AND THE 20 RANDOM ANCHORS IN IT DO NOT. Against a random-batch control at identical budget, active acquisition (50 max-teacher-disagreement + 30 max-uncertainty + 20 random) takes held-out AP from 0.0143 to 0.1391 while the control reaches 0.0502: 2.8x, winning 18 of 20 rounds, with held-out positive-set churn falling to 0.1111. At a ~1% base rate a random batch of 100 holds one positive, and the control curve is what that buys. BUT the anchors, which were supposed to keep an unbiased evaluation stream alive, hold 3 positives after 400 draws: prevalence 0.0057 with a 95% interval 0.01594 wide, WIDER THAN THE QUANTITY ITSELF. They are demoted to drift sentinels. THE LOOP LEARNS; THE PREREGISTERED HUMAN AUDIT MEASURES. And AP here is agreement with the majority TEACHER, so the whole curve is imitation, not accuracy. The first version of this script said the loop DEGRADED, because it measured AP on the shrinking pool the algorithm was itself editing; the churn sat at exactly 1.000 for twenty rounds, which is a constant, not a measurement (D33).","values":{"batch":{"batch_size":100,"max_disagreement":50,"max_uncertainty":30,"random_anchor":20},"rounds":20,"ap_active_start":0.0143,"ap_active_end":0.1391,"ap_random_start":0.0238,"ap_random_end":0.0502,"active_over_random_x":2.8,"active_beats_random_in_rounds":18,"holdout_churn_final":0.1111,"pi_directive_vindicated":true,"what_ap_measures_here":"Agreement with the MAJORITY TEACHER, not accuracy. A rising curve is better imitation of three LLMs that overlap each other at Jaccard ~0.5. No amount of imitation crosses that gap, and the audit is the only instrument that can.","anchors_n":400,"anchors_positives":3,"anchors_prevalence":0.0057,"anchors_ci_width":0.01594,"anchors_are_a_usable_evaluation_stream":false,"anchors_demoted_to":"drift sentinels, rescored under the frozen final model, never used to pick a batch","simulation_caveat":"Every query reveals a label that already exists: 16,800 labels over 5,600 works. This is in-sample recycling. It demonstrates the loop's MECHANICS and cannot show that the loop matures on 4.3M unlabeled works, nor validate the v3.1 instrument, under which no screening has run.","superseded_result":"The first version of this experiment measured AP and churn on the POOL of unrevealed works. Active learning REMOVES contested works from the pool by construction, so the pool gets easier every round and the evaluation target moves under the metric. It reported AP FALLING from 0.019 to 0.011 with churn pinned at exactly 1.000 for twenty straight rounds, and the conclusion was going to be 'the loop degrades'. The constant churn is what gave it away. A metric computed on a set the algorithm is actively editing is not a metric. See DEVIATIONS.md D33."},"computed_at_utc":"2026-07-15T22:52:19Z"},"adjudication":{"headline":"AN INDEPENDENT BLINDED JUDGE SAYS THE RUBRIC IS SILENT ON 89% OF THE WORKS THE MODELS FOUGHT OVER. Fable 5, which is not one of the three screened arms, adjudicated all 179 contested works seeing three screener opinions as A/B/C in random order with no model names, so it was asked which reading of the RUBRIC is right rather than which model to trust. It shows no arm preference (spread 1.16x, chi-square p = 0.493), so it is not a fourth vote for one of the screeners, and it overruled ALL THREE on 6 works. Its verdict on the instrument: the rubric is SILENT on 159 of 179 contested works, and only 17 splits are a screener misapplying a rule that actually exists. THE FIELD'S BOUNDARY IS BEING SET BY THE SCREENER, NOT BY THE INSTRUMENT. The seams that cost the most agreement are methods_dev_vs_study (39 works), lis_sts_asymmetry (26), and history_of_science (19). This is what converts adjudication from a tiebreak into a measurement: not 179 answers, but a frequency-weighted census of which sentences the rubric is missing and exactly how many works each one costs. The judge's TIERS are used for nothing and produce no estimate: a model cannot certify a model, and that argument applies to the judge too.","values":{"judge":"Fable 5 (xhigh), which is NOT one of the three screened arms","blinded":true,"blinding":"three screener opinions presented as A/B/C, order randomized per work, no model names in the prompt","contested_works":179,"judge_agrees_with":[94,95,109],"judge_agrees_with_none":6,"arm_preference_spread":1.16,"arm_preference_chisq_p":0.493,"judge_is_captured":false,"rubric_is_silent":159,"pct_rubric_is_silent":89,"splits_from_screener_error":17,"seam_census":[39,26,19,19,17,12,10,8,8,7,6,5,3],"adjudicated_tiers":[90,23,36,30],"adjudicated_in_scope":59,"caveat":"The judge is a MODEL, not a human, and this is NOT a reference standard. It is an independent, blinded, documented fourth reading whose value is the SEAM CENSUS, not the tiers. Its tiers are used for nothing downstream and produce no estimate. The arm-preference test shows the judge is not CAPTURED; it says nothing about whether it is CORRECT, and correctness is exactly what the human audit exists to establish. A model cannot certify a model, which is the whole argument of this project and it applies to the judge too."},"computed_at_utc":"2026-07-13T11:55:28Z"},"affiliation_gap":{"headline":"508,744 of 793,883 works (64%) in the metaresearch topic space have no raw affiliation strings in OpenAlex.","values":{"topic_space_total":793883,"with_raw_affiliation":285139,"without_raw_affiliation":508744,"pct_without":64.1},"computed_at_utc":"2026-07-13T11:54:42Z"},"agent_variance":{"headline":"Haiku, the model finding 13 budgets the entire screen on, lands near Sonnet's base rate (1.27% vs 1.06%) and agrees with it on 98.1% of the frame, but their in-scope SETS overlap 16% unweighted and 10% design-weighted (Sonnet-GPT: 54%/37%); of Sonnet's 58 positives Haiku agrees on 12. RATE AGREEMENT IS NOT SET AGREEMENT. Worse: agents of the SAME model on the SAME prompt disagree beyond chance in BOTH arms after conditioning on stratum (CMH p = 0.0056 and 0.015), with raw spreads 3.1x and 5.2x against 2.2x between models, and the agents' ordering FLIPS between arms. The noise inside one model is at least the size of the difference between models, and the pilot's own 40-agent screen is too underpowered to rule the same instability out (5 of 37 chunks found zero metaresearch; p = 0.113).","values":{"n_works":1290,"base_rate_sonnet_pct":1.06,"base_rate_gpt_pct":2.37,"base_rate_haiku_pct":1.27,"agreement_haiku_sonnet_pct":98.1,"jaccard_sonnet_gpt_pct":54,"jaccard_sonnet_haiku_pct":16,"jaccard_gpt_haiku_pct":12,"wjaccard_sonnet_gpt_pct":37,"wjaccard_sonnet_haiku_pct":10,"wjaccard_gpt_haiku_pct":6,"sonnet_positives":58,"haiku_agrees_on":12,"agent_rates_raw_pct":{"agent-1":1.6,"agent-2":1.25,"agent-3":3.85},"stratum_mix_differs_by_agent_p":1.34e-17,"arm1_cmh_p":0.00564,"arm1_permutation_p":0.0275,"arm1_raw_spread_x":3.1,"arm2_cmh_p":0.0152,"arm2_permutation_p":0.0075,"arm2_raw_spread_x":5.2,"agent_order_replicates":false,"spread_weighted_x_leverage_sensitive":13.2,"between_model_spread_x":2.2,"within_at_least_matches_between":true,"pilot_chunks":37,"pilot_agents_finding_zero":5,"pilot_between_agent_p":0.113,"pilot_underpowered_not_homogeneous":true,"caveat":"The first draft's between-agent test was CONFOUNDED: the stratum mix differs by agent (p = 1.3e-17), and the draft asserted a verification that did not exist (DEVIATIONS.md D13). The tests above condition on stratum, and the heterogeneity survives in both arms. The 13.2x design-weighted spread the draft led with rests on five high-weight events and is demoted to a recorded, leverage-sensitive descriptive. THE INFERENCE IS SCOPED: there are three agents per arm, assigned consecutive chunk blocks without randomization or a run-time manifest, so these p-values license 'these runs are not exchangeable', not a population claim about agents in general; that is exactly enough to break a budget that assumed exchangeability, and the full study assigns agents randomized, manifest-logged, fixed-size chunks with a duplicate-agent reliability arm. The pilot's own agents give p = 0.113 on ~2 expected events per chunk: underpowered, so the pilot is uninformative on agent homogeneity, not exonerated. Haiku was tested on the six-field payload (the pilot's own deviation, D1) and on a guided eight-field arm, so the defensible conclusion is 'not shown to be an interchangeable measurer, and unstable in the arms tested', not 'cannot screen'. An eight-field neutral-prompt arm is not used at all: one agent claimed six label files it never wrote (D11), so no payload effect is reported from any arm. The guided arm is used ONLY for the between-agent contrast, which its shared prompt leaves internally valid."},"computed_at_utc":"2026-07-15T22:52:18Z"},"agreement":{"headline":"Swap which model is called 'the screener' and the base rate moves from 1.06% to 2.37%: a 2.2x spread, from 45,397 to 101,776 works in the frame. The two screeners agree on in/out for 98.4% of the frame (design-weighted), but that figure is dominated by the settled rejects: agreement falls to 95% inside the contested boundary. THE SCREENER-SWAP RANGE, NOT THE BINOMIAL CI ON EITHER MODEL ALONE, IS THE HONEST UNCERTAINTY ON THE FIELD'S SIZE.","values":{"n_double_screened":1290,"base_rate_screener_a_pct":1.06,"base_rate_screener_b_pct":2.37,"swap_ratio_x":2.24,"field_size_screener_a":45397,"field_size_screener_b":101776,"published_binomial_ci_contains_b":false,"screener_a":"claude-sonnet-4-6 (40 agents, medium effort)","screener_b":"gpt-5.6-sol (codex)","sampling":"stratified on screener A's label; positives/paratext/boundary taken whole, settled-OUT sampled","raw_agreement_inout_pct":96.6,"weighted_agreement_pct":98.4,"cohens_kappa_inout":0.681,"n_disagree_inout":44,"gpt_in_claude_out":37,"claude_in_gpt_out":7,"agreement_by_stratum":{"stratum":["settled_out","boundary","positive","paratext"],"n":[600,599,58,33],"sel_prob":[0.124921923797626,1,1,1],"agree_pct":[99,95,87.9,97],"a_says_in":[0,0,58,0],"b_says_in":[6,30,51,1]},"caveat":"PROCESS METRIC, NOT ACCURACY. Two LLMs share training data and failure modes; their errors are correlated and most correlated on the boundary. This is not 'duplicate screening': that term's warrant comes from independent human judgment. Accuracy rests on the human-coded probability sample (PROTOCOL s6.2). Agreement is reported per stratum because a pooled kappa on a 1.3% base rate is dominated by the cell where agreement is free (the kappa paradox). And note what the high agreement figure CONCEALS: it is dominated by the settled-OUT mass, while the two screeners imply base rates a factor of two apart. Quoting agreement without the swap would be presenting the reassuring statistic."},"computed_at_utc":"2026-07-15T22:52:08Z"},"audit_power":{"headline":"The audit as first specified could not measure what it exists to measure. A simple random sample of the 4,243,096-record screened-out mass yields an expected 0.4 hits from the 600 records budgeted; seeing 20 would take 2,009 coder-hours against the 65 planned. Score-stratified oversampling rescues it only partly (the pilot shows 30 of 37 disputed works sit at the contested boundary), and it stays BLIND to the confident rejects. Which works those are is a MECHANISM, not a measured differential: the rubric says judge on the title alone with no abstract, and says T2 work may use none of the field's vocabulary, so a work with neither is rejected confidently and never sampled. (An earlier version cited a 3.6x figure here as if it were measured; it rested on four works and is withdrawn.) So recall is measured two other ways: against the 5,737 works that already carry rubric labels, and by KNOWN-ITEM recall on a venue reference set, because venue is an external criterion immune to the aboutness that defeats everything else.","values":{"problem":"A simple random sample of the screened-out stratum cannot measure screening sensitivity: the works the screen wrongly rejected are a vanishing fraction of a 3.2M-record rejected mass.","screened_out_pool":4243096,"screened_out_is_the_screens_rejects":true,"audit_screened_out_budgeted":600,"expected_hits_at_95_recall":0.4,"codings_needed_for_20_hits_at_95_recall":30134,"coder_hours_needed":2009,"coder_hours_budgeted":65,"naive_audit_is_powered":false,"disputed_in_boundary":30,"disputed_in_settled_rejects":6,"misses_concentrate_near_threshold":true,"blind_spot":"Score-stratified oversampling finds the works the screener ALMOST caught; it is blind to the ones it rejected CONFIDENTLY. WHICH works those are is a MECHANISM, not a measurement: the rubric says judge on the title alone when the abstract is missing, and the rubric also says T2 work may use none of the field's vocabulary, so a work with neither is rejected confidently and sits deep in the settled rejects. An earlier version cited a 3.6x differential from finding 11 as if this were measured. It is not, and that figure is withdrawn (DEVIATIONS.md D6). The venue instrument is how the prediction gets tested rather than asserted.","blind_spot_is_a_mechanism_not_a_measurement":true,"instrument_1":"Measure any filter's recall against the 5737 works that already carry full-rubric labels, exactly as finding 12 scored the topic route. Free, and it needs no needle-hunting in the discarded mass.","instrument_2":"Known-item recall on an external criterion: VENUE. A Canadian-authored paper in Social Studies of Science is T2 by where it was published, whatever its abstract is about. Venue is immune to aboutness, which is what defeats topic retrieval (finding 12) and title-only screening (finding 11), so it is the only instrument that can see into the blind spot.","reference_venues":["Social Studies of Science","Scientometrics","Quantitative Science Studies","Research Integrity and Peer Review","Journal of the Association for Information Science and Technology","Research Evaluation","Accountability in Research","PLOS ONE (metaresearch collection)","Recherches qualitatives","Documentation et bibliotheques"],"n_labelled_works":5737,"n_in_scope":75,"caveat":"The recall grid assumes the screen's misses are uniform in the rejected mass, which the pilot shows they are not (they concentrate at the boundary). That makes the naive design LESS hopeless than the grid implies but does not save it, and it does nothing at all about the confident-reject blind spot. Known-item recall on a venue reference set is a non-probability estimate: it bounds and diagnoses recall on the hard cases, it does not replace the design-weighted population estimate."},"computed_at_utc":"2026-07-13T11:54:49Z"},"base_rate":{"headline":"Screening 5,737 unfiltered Canadian works against the rubric puts metaresearch at 1.31% of Canadian research, implying ~56,206 works in the 4,299,418-work frame, sizing the field without a search strategy at all. The 95% CI on this screener's labels is 1.03-1.64%, but that is sampling error, NOT the uncertainty: swap the screener and the estimate lands outside it (finding 10). The machine-screener range in finding 10 is the honest uncertainty on field size, and only the human audit can narrow it.","values":{"n_screened":5737,"screener":"claude-sonnet-4-6, 40 agents, medium effort, locked rubric","sampling_frame":"unfiltered Canadian works from the pinned 2026-06-24 snapshot partition","tier_counts":{"OUT":5621,"T1":40,"T2":35,"T3":41},"n_in_scope_t1_t2":75,"base_rate_pct":1.31,"base_rate_ci_lo_pct":1.03,"base_rate_ci_hi_pct":1.64,"canadian_frame_size":4299418,"estimated_field_size":56206,"estimated_field_lo":44268,"estimated_field_hi":70338,"topic_route_retrieved":14873,"binomial_ci_is_not_the_uncertainty":true,"caveat":"Machine labels, not a human gold standard, and the binomial CI above is sampling error on ONE screener; the honest uncertainty is the screener-swap range in finding 10 (1.06% to 2.37%). The partition is also not a uniform draw: it under-represents works with abstracts, where the screen finds 2x more metaresearch (finding 11). This is a hypothesis with a denominator; the human-coded probability sample tests it. Do NOT divide topic_route_retrieved by estimated_field_size; see finding 12."},"computed_at_utc":"2026-07-15T22:33:57Z"},"base_rate_robustness":{"headline":"The base rate's real bias is not recency but MISSING ABSTRACTS: 31.5% of the partition has none, and the screen finds 0.78% metaresearch there against 1.55% where an abstract exists (chi-square p = 0.023, robust to adjustment for year and language). A third of the frame is screened on its title alone. What this does NOT show, and an earlier draft wrongly claimed, is that the blindness is DIFFERENTIAL by tradition: the T2 no-abstract cell holds 4 works and the interaction is not significant (p = 0.141). That claim is withdrawn, as is 'Erudit's exact profile' (the stratum is 99% English and the works are NEWER, not older). The main effect is the finding.","values":{"partition":"updated_date=2026-06-24","records_sent_to_screener":6202,"records_silently_lost":465,"records_lost_pct":7.5,"pct_no_abstract_among_lost":40.2,"pct_no_abstract_among_labelled":31.5,"chisq_p_loss_bias":0.00013,"losses_are_biased":true,"works_2000_09":2704,"works_2020_25":1195,"old_to_new_ratio":2.3,"partition_skews_old":true,"base_rate_by_era_pct":{"2000-09":1.24,"2010-19":1.35,"2020-25":1.4},"chisq_p_era":0.906,"era_events":75,"era_check_is_underpowered":true,"no_abstract_share_pct":31.5,"base_rate_no_abstract_pct":0.78,"base_rate_has_abstract_pct":1.55,"chisq_p_abstract":0.023,"abstract_effect_x":2,"t1_penalty_x":1.4,"t2_penalty_x":3.6,"t2_no_abstract_cell_count":4,"interaction_p":0.141,"differential_is_supported":false,"differential_claim_withdrawn":true,"no_abstract_mean_year":2013.7,"has_abstract_mean_year":2010.3,"no_abstract_stratum_pct_english":99,"p_no_abstract_given_english_pct":32,"p_no_abstract_given_non_english_pct":12.7,"erudit_profile_claim_holds":false,"residual_bias_direction":"anti-conservative for coverage claims: the partition over-represents abstract-less works (31.5%), where the screen finds 2x less metaresearch, so 1.31% likely UNDER-states the frame's base rate, and the field is larger than the headline implies","supersedes":"THREE retractions live here. (1) An earlier version tested ERA only, called the base rate robust, and published 3100/2900/1500 as percentages (a dplyr summarise() column-masking bug). (2) It then claimed the blindness is DIFFERENTIAL, T2 losing 3.6x against T1's 1.4x, and made that the proposal's whole answer to the inclusiveness criterion. The T2 no-abstract cell holds FOUR works and the interaction is not significant (p = 0.141). WITHDRAWN. (3) It claimed missing abstracts track 'older, non-English' records, 'Erudit's exact profile'. Backwards: they are NEWER, and the stratum is 99% ENGLISH. WITHDRAWN. See DEVIATIONS.md D4, D5, D6.","caveat":"What survives is the MAIN EFFECT and only the main effect: a third of the frame is screened on its title alone and the screen finds half as much metaresearch there (p = 0.023, robust to adjustment for year and language). That is a real coverage problem and a reason to stratify the audit on abstract availability. It is NOT evidence of differential blindness by tradition, and this finding no longer says it is. Separately, the era check is underpowered (75 events, 3 strata) and cannot refute an era effect; it merely fails to detect one. And note D2: the harness silently dropped 465 records, non-randomly, on this very covariate."},"computed_at_utc":"2026-07-13T11:54:47Z"},"canadian_linkage":{"headline":"Affiliation finds 14,873 works; a further 3,964 are about Canada with no Canadian affiliation. NSERC has 9.2x SSHRC's linked works, so funder-based rules under-count the social sciences.","values":{"by_affiliation":14873,"by_funder":1331,"about_canada":5704,"about_canada_no_affiliation":3964,"funder_works":{"Canadian Institutes of Health Research":194681,"Natural Sciences and Engineering Research Council of Canada":433090,"Social Sciences and Humanities Research Council of Canada":46913,"Canada Foundation for Innovation":16753},"sshrc_works":46913,"nserc_works":433090,"nserc_to_sshrc_ratio":9.2,"affiliation_noise":{"University of London":494,"Impact":475}},"computed_at_utc":"2026-07-13T11:54:44Z"},"canadian_linkage_misnames_itself":{"headline":"BOTH CLAUSES OF 'CANADIAN' MEASURE SOMETHING OTHER THAN WHAT THEY ARE NAMED. This project has spent all its effort on what retrieval MISSES; this is the first hard evidence about what it wrongly ADMITS. (1) CA-AFF, the PRIMARY estimand's main clause: 17,466 works enter the frame on a Canadian 'institution' that does not exist. OpenAlex hands 'Discovery Air (Canada)' to a Brazilian linguistics paper, 'Musee de la Civilisation' to a Brazilian food-science paper, and 'Impact', which is not an institution at all but a parse failure with an institution id, to a Spanish COVID essay. The CA-AFF route also carries thousands of works in Latvian and Indonesian. That count is a LOWER BOUND: the artifact strings tested are only the ones screening agents noticed BY EYE while doing something else, and nobody has swept the institution vocabulary, so the true precision is UNKNOWN rather than merely unmeasured. (2) ABOUT-CA, the SECONDARY estimand: the retrieval route means 'Canada appears in the text' and the rubric field means 'the Canadian research SYSTEM is a substantive object of study', and in the strata built entirely on the former, the latter fires on 1.8% to 3.5% of works. Worse, `about_ca` cannot be true unless the work is ALREADY about research, so it is nearly collinear with tier: only 39 works in 5,600 are both in-scope AND about the Canadian research system. THE SECONDARY ESTIMAND CANNOT BE ESTIMATED FROM THE STRATUM DESIGNED FOR IT. Four screening agents found this independently and all four proposed the same repair: split the field into `about_ca_system` and `about_ca_topic`, because one boolean cannot separate 'Canadian data, universal claim' from 'a claim about Canada'.","values":{"ca_aff_works":2714734,"ca_aff_only_institution_is_artifact":17466,"artifact_strings_tested":["Impact","Discovery Air (Canada)","Musée de la Civilisation","Encana (Canada)","Kellogg's (Canada)","The Alberta Paraplegic Foundation"],"artifact_count_is_a_lower_bound":true,"ca_aff_non_official_languages":[8734,8463,5289,5160,4708,1200],"about_ca_pct_in_about_strata_opus":[1.8,3.5],"about_ca_pct_in_about_strata_gpt":[1,2],"about_ca_pct_in_about_strata_grok":[0.8,1.2],"n_screened":5600,"about_ca_and_in_scope":39,"about_ca_and_out_of_scope":26,"about_ca_nearly_collinear_with_tier":true,"found_by":"Screening agents, reporting records they were given for an unrelated reason. Four of them independently reported that the ABOUT-CA route and the rubric's about_ca field are different constructs, and all four proposed the same repair without having seen each other's reports.","caveat":"The artifact count is a LOWER BOUND, and deliberately reported as one: the strings tested are only those agents noticed by eye. No sweep of the OpenAlex institution vocabulary has been done, so the true CA-AFF precision is unknown, not merely unmeasured. That is the honest state and it is why the human audit samples the retrieved stratum as well as the non-retrieved one: precision is measured, not assumed."},"computed_at_utc":"2026-07-13T11:55:27Z"},"capture_recapture_fails":{"headline":"Naive two-route capture-recapture estimates 467,541 Canadian metaresearch works, implying Canada produces 59% of the world's metaresearch against an observed 1.9%. The estimator is void here; we cut it rather than dress it up as a lower bound.","values":{"route1_topic":14873,"route2_naive_lexical":77583,"overlap":2468,"observed_union":89988,"lincoln_petersen_estimate":467541,"entire_topic_space_all_countries":793883,"canada_observed_share_pct":1.9,"canada_implied_share_pct":58.9,"estimator_void":true},"computed_at_utc":"2026-07-13T11:54:44Z"},"classifier":{"headline":"THE CLASSIFIER MAY NOT LABEL, AND THE NUMBERS SAY WHY. Design-weighted, out-of-fold, venue-grouped: the teacher heads reach AP 0.17 to 0.135 at AUC ~0.8, with a 4.68x top-decile lift that is real and is lift toward MACHINE labels. THE ESTIMAND MOVES THE SCORE 2.7x (unanimous 0.0783, majority 0.1364, any 0.2093), so there is no 'the' target and choosing one silently settles the question the audit exists to answer. AND THE UNANIMOUS CORE IS THE HARDEST TO LEARN, not the easiest: if the boundary were a shared line with noise around it, the agreed core would be the separable part, and it is not. The between-head spread predicts where the teachers split at 3.74x the base rate: real, weak, and reported as an efficiency gain rather than a map. Finally, the abstract is worth MORE THAN ANY MODELING CHOICE (+0.1145 AP, 84% relative), and the frame cannot serve it: works.csv stores has_abstract, a boolean, and no text. The DEPLOYABLE model is the WORSE one, the build now FAILS if a model is trained on a field inference cannot supply, and the gap is a priced budget decision rather than an assumption.","values":{"evaluation":"out-of-fold, venue-grouped, vectorizer fitted on train folds only, DESIGN-WEIGHTED","why_design_weighted":"The 5,600 works are a stratified sample: French carries weight 311, aff_core 1,119. An unweighted AP is an AP over a population that does not exist. Both are reported; the gap is a fact about the design.","ap_design_weighted_by_teacher":{"opus":0.17,"gpt":0.135,"grok":0.1417},"ap_design_weighted_by_consensus_rule":{"unanimous":0.0783,"majority":0.1364,"any":0.2093},"estimand_moves_performance_x":2.7,"unanimous_core_is_hardest_to_learn":true,"top_decile_lift":{"opus":4.68,"gpt":3.93,"grok":5.28},"ap_by_language_opus":{"en":0.1757,"fr":0.181},"ap_by_language_gpt":{"en":0.1425,"fr":0.089},"french_is_model_dependent":"GPT's boundary is markedly less learnable in French (AP 0.089 vs 0.1425 in English) while Opus's is not ( 0.181 vs 0.1757 ). 'Character n-grams do the French work' was an assertion; measured per language, it is true for one teacher and false for another.","predicting_teacher_disagreement":{"n_contested":89,"base_rate":0.0159,"average_precision":0.0594,"lift_over_base":3.74,"verdict":"Real but weak. It buys an efficiency gain for the audit's disagreement stratum. It is NOT a trustworthy standalone map of the contested region, and shared teacher bias is invisible to it: if all three are wrong the same way the spread is zero and the work looks settled."},"payload_skew":{"ap_frame_parity":0.1364,"ap_with_abstract":0.2509,"abstract_is_worth_ap":0.1145,"why_the_deployable_model_is_the_worse_one":"data/db/works.csv stores has_abstract, a boolean, and no abstract text. A model trained with abstracts scores a payload the 4.3M works do not have. Reporting the abstract model's number as the classifier's performance would be reporting a metric for a model that cannot run. The difference is what re-extracting abstracts from the snapshot would buy, and it is a budget decision with a number attached rather than an assumption."},"ships_a_category_label":false,"why_no_label":"Distilled from our own metaresearch labels, a student reproduced its own teacher's positive set at Jaccard 0.17 (finding 25). Two independent adversarial reviews reached the same verdict from different directions: per-teacher heads are useful for ALLOCATING HUMAN EFFORT and cosmetic as a solution to the contested boundary. Scores ship. Labels do not.","what_the_classifier_is_actually_for":"Not cost. Finding 13 says the LLM can read every work in the frame for $1,261, so there was never anything to avoid, and a proposal that budgeted the full screen while justifying a cheap substitute for it was describing two studies (D32). It is for: a calibrated score the audit stratifies on; rubric revisions testable in minutes instead of one $1,261 pass each; a frame-wide estimate of where the three teachers would split, which three full passes would cost 3x and ~25 days; and learnability as evidence about the boundary."},"computed_at_utc":"2026-07-15T22:32:09Z"},"distillation_ceiling":{"headline":"A CLASSIFIER TRAINED ON LLM LABELS CANNOT EVEN REPRODUCE ITS OWN TEACHER, LET ALONE ADJUDICATE THREE. Students distilled from each model match their own teacher at Jaccard 0.17, while the three teachers match EACH OTHER at 0.5 to 0.56: the classifier is a worse approximation of Opus than Grok is. Distilling from Opus also moves you AWAY from Grok (-0.35), so distillation does not bridge the models' disagreement, it degrades away from all of them. THE SELF-TRAINING LOOP THEN REFUTED MY OWN PREDICTION. I argued at length that it would shrink the positive class toward the confident anglophone core; run on 20,000 works the models never saw, it did not: positive rate drift +0.00 points, French share drift +0.00 points, because class weighting prevents the collapse, exactly as the adversarial review warned before the run ('a serious risk, NOT A THEOREM'). What the loop DID do is reach a fixed point immediately and recycle its own labels (224 -> 424 training positives, then nothing): it CONVERGED WITHOUT LEARNING, and a fixed point looks identical from the inside whether its labels are right or wrong. What the classifier IS worth is the stratifier: 48.2% of the in-scope works sit in its top decile against a blind baseline of 10%, a 4.8x lift in in-scope works found per unit of human coding effort, and design-weighted estimates stay unbiased however noisy it is. THE MACHINE DECIDES WHERE THE HUMANS LOOK. IT NEVER SUPPLIES A LABEL.","values":{"n_three_way_labelled":5600,"student_vs_own_teacher_jaccard":[0.176,0.173,0.17],"teacher_vs_teacher_jaccard":[0.5,0.5,0.563],"bridging_gain":[-0.348297213622291,-0.348214285714286,-0.3375],"selftrain_rounds":[0,1,2,3],"selftrain_train_positives":[224,424,424,424],"selftrain_pool_positive_rate_pct":[1,1,1,1],"selftrain_french_share_pct":[6.5,6.5,6.5,6.5],"selftrain_drift_positive_rate_pts":0,"selftrain_drift_french_share_pts":0,"my_shrinkage_prediction_was_refuted":true,"stratifier_top_decile_recall_pct":48.2,"stratifier_lift_vs_blind":4.8,"used_to_produce_any_estimate":false,"caveat":"The student is TF-IDF word+char n-grams with a cross-validated logistic head: the cheap classifier the question was actually about. A fine-tuned multilingual encoder would raise the student-vs-teacher number and change none of the argument, because the ceiling is set by the LABELS, not by the model class. The self-training loop's non-collapse is CONDITIONAL on class weighting and should not be read as a general safety result: it says the collapse is avoidable, not that the loop is informative. And nothing here produces a prevalence estimate; the classifier is a stratifier, and the human audit is the instrument."},"computed_at_utc":"2026-07-15T22:58:15Z"},"erudit":{"headline":"Erudit matches 0 sources in OpenAlex, but its OAI-PMH endpoint is live and exposes 379 harvestable sets.","values":{"openalex_sources_matching_erudit":0,"oai_endpoint":"https://oai.erudit.org/oai/request","oai_repository_name":"Erudit","oai_earliest_datestamp":"2011-06-03","oai_harvestable_sets":379},"computed_at_utc":"2026-07-13T11:54:43Z"},"funder_route_recall":{"headline":"The PRIMARY ESTIMAND leans on CA-FUND to rescue works whose affiliation is missing (finding 2: 64% have no affiliation string), and nothing had tested it. Tested against CIHR's own database of 44,190 funded projects, the result REFUTED THE HYPOTHESIS I WROTE BEFORE RUNNING IT: OpenAlex tags 178,133 frame works with CIHR, or 4.03 per grant, a plausible rate showing no under-tagging (DEVIATIONS.md D14). What the data DOES support needs no hypothesis of mine: 71.2% OF THE FRAME CARRIES NO FUNDER METADATA AT ALL, which is CA-FUND's hard ceiling, and 65.9% of Canadian-AFFILIATED works carry none either. Both clauses of the estimand rest on metadata that is mostly absent, which is why the frame is a union of four routes and why the audit must sample the works no route reached.","values":{"external_criterion":"CIHR's own project database (44,190 projects), which owes nothing to OpenAlex","cihr_projects":44190,"cihr_distinct_pis":27536,"cihr_funder_id":"https://openalex.org/F4320334506","frame_works":4299418,"frame_works_tagged_cihr":178133,"tagged_publications_per_funded_project":4.03,"papers_per_grant_is_a_plausible_rate_not_a_defect":true,"expectation_i_wrote_before_running_and_that_was_false":"that OpenAlex under-tags CIHR so badly it falls below one paper per grant. It is 4.03 per grant, a plausible rate. See DEVIATIONS.md D14.","works_ca_fund_rescues_alone":166743,"frame_works_with_any_funder":1239950,"pct_frame_with_any_funder":28.8,"pct_frame_with_no_funder":71.2,"ca_aff_works":2734192,"ca_aff_works_with_no_funder":1802605,"pct_ca_aff_with_no_funder":65.9,"top_recorded_funders":{"Natural Sciences and Engineering Research Council of Canada":294401,"Canadian Institutes of Health Research":178133,"National Institutes of Health":81262,"National Natural Science Foundation of China":67358,"National Science Foundation":62497,"Canada Research Chairs":35113,"European Commission":34039,"Social Sciences and Humanities Research Council of Canada":33473},"is_an_aggregate_not_a_record_level_join":true,"caveat":"CIHR's CSV carries no DOIs and no publication links, so this is an AGGREGATE reconciliation, not a record-level known-item join, and NO RECALL POINT ESTIMATE is claimed. It establishes a CEILING on CA-FUND (the route cannot see a funder OpenAlex never recorded), which is a bound, not a measurement. The papers-per-grant ratio is reported because I ran it, and it REFUTES the hypothesis I wrote before running it: at 4.03 per grant it is a plausible publication rate and shows no CIHR under-tagging at all. It is also the wrong instrument, for the same reason the retracted 32.4% coverage figure was (D3): its numerator and denominator are not linked record to record, so the quotient has no estimand behind it. Record-level linkage is finding 21."},"computed_at_utc":"2026-07-13T11:55:02Z"},"gemma_gate":{"headline":"THE FREE MODEL FAILED THE GATE, AND THE GATE WAS AIMED AT THE WRONG BAR. Gemma-4-31B screened 700 works under rubric v3.1 (100% schema-clean) and scored Jaccard 0.44 on metaresearch and 0.515 on any-category against the frontier union, under the prespecified 0.60 bar, SO IT DOES NOT LABEL THE FRAME: the rule stands because rules that move after the numbers exist are not rules. But the frontier models agree with each other at only 0.384 and 0.36 on the same works, because every loop batch is selected FOR disagreement: the gate demanded of a free model what the frontier models do not achieve among themselves there, and Gemma beat the frontier's own mutual agreement on both targets. Test 2 is prespecified on the random_baseline stratum with the rationale as the bar: J(gemma, union) >= J(opus, gpt).","values":{"model":"google/gemma-4-31b-it (free)","n_works":700,"schema_clean_return_rate":1,"jaccard_metaresearch_gemma_vs_union":0.44,"jaccard_metaresearch_opus_vs_gpt":0.384,"jaccard_anycat_gemma_vs_union":0.515,"jaccard_anycat_opus_vs_gpt":0.36,"study_design_exact":{"gemma_vs_opus":0.643,"gemma_vs_gpt":0.614,"opus_vs_gpt_for_reference":0.73},"prespecified_rule":"Jaccard >= 0.60 vs the frontier union, on metaresearch AND any-category","passed":false,"rule_honored":true,"gate_was_miscalibrated_because":"the 0.60 bar was set from round-001's 0.44-0.75 pairwise range without registering that loop batches are selected FOR disagreement, which depresses every agreement statistic computed on them. On the same 700 works the frontier models agree with each other at 0.384/0.360: below the bar Gemma was held to, and Gemma beat both numbers.","test2_prespecified":"On the enriched sample's random_baseline stratum (1,500 works, not selected for disagreement), Gemma passes if J(gemma, frontier_union) >= J(opus, gpt) on both metaresearch and any-category. Written before any of those labels exist."},"computed_at_utc":"2026-07-14T00:08:32Z"},"instrument_contradicts_itself":{"headline":"THE LOCKED INSTRUMENT CONTRADICTS ITSELF, AND NOTHING CAUGHT IT. The rubric and the output schema name TWO DIFFERENT controlled vocabularies for the same field, `genre`, overlapping on 2 values (empirical and other); 4 terms exist only in the rubric and 7 only in the schema. Every screener was handed both and told to obey both, and each invented its own reconciliation: GPT-5.6 and Grok followed the rubric and are therefore 26.9% and 20.9% ILLEGAL against the schema, while Opus drew from both lists at once and emitted 13 distinct values. The validator checked `tier` and never checked `genre`, so 16,800 labels passed every check that ran. It was found by an agent mentioning it in one clause of a report about something else. Nothing crashed; the variable simply meant a different thing in each arm, and the variance would have been attributed to the models. The labels are NOT repaired, because harmonizing the arms after seeing them would destroy the only evidence that they diverged; the genre field is reported as unusable and the fix belongs in the instrument, at a version boundary.","values":{"field":"genre","rubric_vocabulary":["empirical","conceptual","editorial/commentary","policy","infrastructure/announcement","other"],"schema_vocabulary":["empirical","review","methods","commentary","editorial","protocol","dataset","software","other"],"shared_values":["empirical","other"],"n_shared":2,"only_in_rubric":["conceptual","editorial/commentary","policy","infrastructure/announcement"],"only_in_schema":["review","methods","commentary","editorial","protocol","dataset","software"],"arms":["opus","gpt","grok"],"n_labels":[5600,5600,5600],"n_distinct_values_by_arm":[13,6,12],"pct_legal_against_schema":[90.8,73.1,79.1],"pct_legal_against_rubric":[86.1,100,99.8],"validator_checked_tier":true,"validator_checked_genre":false,"labels_repaired":false,"found_by":"An Opus screening agent mentioned it in one clause of a report about something else, while working chunks it had been given for an unrelated reason. It was not looking for this, no check was watching for it, and it had already survived 6,000 labels across three models.","caveat":"The genre variable from this screen is reported as UNUSABLE and is used for nothing. It is not remapped to a common vocabulary: the three arms resolved the contradiction differently, and harmonizing them after the fact would destroy the only evidence that they did."},"computed_at_utc":"2026-07-15T22:32:11Z"},"label_limits":{"headline":"Two limits on the machine labels, found by attacking the fixes. (A) The topic route's recall is 12% against screener A and 7% against screener B: finding 14's instrument (ii) scores filters against MACHINE labels, so it measures agreement with a machine, not accuracy, and finding 10 already showed that swings by a factor of two. The conclusion strengthens (the second screener thinks the route is WORSE) but 12% is not truth. (B) The pilot holds exactly 1 French in-scope work, so seeing 20 French positives needs ~1,620 coded French records against an audit budget of 1,000. FRENCH SENSITIVITY IS NOT ESTIMABLE IN AN OPENALEX-ONLY FRAME. That is an argument for the Erudit harvest, not against the French claim, but the pilot did not run that harvest, so the power is stated as a condition rather than a promise.","values":{"route_recall_vs_screener_a_pct":12,"route_recall_vs_screener_b_pct":7,"route_recall_a_ci":[5.6,21.6],"route_recall_b_ci":[2.5,14.3],"positives_screener_a":75,"positives_screener_b":88,"instrument_ii_is_model_dependent":true,"instrument_ii_caveat":"Finding 14's instrument (ii) scores a filter against the 5,737 MACHINE labels. That measures agreement with a machine, not accuracy. Swap the machine and the topic route's recall moves from 12% to 7%. The conclusion (the route finds a small fraction) survives and strengthens; the NUMBER is not a measurement against truth.","french_records_in_pilot":81,"french_in_scope_in_pilot":1,"french_records_needed_for_20_positives":1620,"audit_budget_records":1000,"french_stratum_is_powered":false,"french_power_depends_on":"the Erudit harvest, which the pilot did NOT run (it verified the endpoint: 379 live sets)","caveat":"(A) is a limit on every recall number this project quotes against machine labels, including its own headline. (B) is a limit on the inclusiveness promise: French sensitivity cannot be estimated in an OpenAlex-only frame, because Erudit matches zero OpenAlex sources and the francophone literature is therefore largely absent from the frame rather than merely sparse in it. Both are stated in the proposal rather than left for a reviewer."},"computed_at_utc":"2026-07-13T11:54:49Z"},"language_gap":{"headline":"French is 2.7% (395/14,873) of Canadian metaresearch in OpenAlex. A dedicated French lexicon finds 168 Canadian works, against 8,026 in English.","values":{"canadian_topic_works":14873,"n_english":14028,"n_french":395,"pct_french":2.7,"en_lexicon_canadian_hits":8026,"fr_lexicon_canadian_hits":168,"fr_lexicon_world_hits":4682,"canada_share_of_world_french":3.6},"computed_at_utc":"2026-07-13T11:54:43Z"},"openalex_is_metered":{"headline":"The OpenAlex API is metered (1,000 credits per ~11h; $0.10 free tier). Enumerating the 4,299,418-work Canadian frame needs 21,498 cursor-paged calls: 10 days per pass on the free tier. An API-based pipeline at this scale is neither free nor reproducible; the pinned snapshot is both.","values":{"observed_retry_after_s":40268,"observed_ratelimit_limit":1000,"observed_free_tier_usd":0.1,"observed_cost_per_call_usd":0.0001,"frame_size_works":4299418,"per_page_max":200,"calls_for_one_pass":21498,"days_on_free_tier_per_pass":10,"prepaid_cost_per_pass_usd":2.15},"computed_at_utc":"2026-07-13T11:54:45Z"},"polysemy":{"headline":"The single term reproducibility retrieves 43,392 Canadian works, of which only 0.8% fall in the metaresearch topic space. Keyword retrieval cannot separate the metaresearch sense from the everyday one.","values":{"hits_alone":{"reproducibility":43392,"\"peer review\"":21234,"\"open access\"":6609,"\"open science\"":1925},"hits_alone_and_on_topic":{"reproducibility":362,"\"peer review\"":795,"\"open access\"":634,"\"open science\"":325},"topic_space_precision_pct":{"reproducibility":0.8,"\"peer review\"":3.7,"\"open access\"":9.6,"\"open science\"":16.9},"disciplined_lexicon_hits":8026,"worst_term":"reproducibility"},"computed_at_utc":"2026-07-13T11:54:44Z"},"preprint_coverage":{"headline":"The frame carries 156,086 preprints, and the tempting move was to assert that OpenAlex covers the preprint servers and skip the ingest. That is a COVERAGE CLAIM, and this project does not get to make one from inside the pipeline being claimed for: it is the topic route certifying its own recall (finding 12). Measured instead against the servers' OWN API, OpenAlex indexes 99.6% of the 705 bioRxiv and medRxiv preprints enumerated (3 missing). The claim survives measurement, so no separate preprint ingest is built.","values":{"external_criterion":"the bioRxiv/medRxiv details API, which enumerates the servers' own corpus and owes nothing to OpenAlex","window":"2023-01-01 to 2025-12-31","preprints_enumerated":705,"indexed_by_openalex":702,"missing_from_openalex":3,"pct_indexed":99.6,"sampled_with_canadian_author":45,"of_those_present_in_frame":45,"frame_preprints_total":156086,"separate_ingest_needed":false,"caveat":"The bioRxiv API exposes no author country, so the sample is mostly non-Canadian and the clean quantity is INDEX coverage (does OpenAlex hold the preprint at all?), not Canadian recall: a preprint the index lacks cannot enter any frame by any route, so index coverage is the binding upper bound. The Canadian sub-count is small and is reported as a check, not as an estimate. arXiv is NOT tested here (its OAI endpoint pages differently); the frame carries 11,647 arXiv works and that claim remains untested against arXiv itself."},"computed_at_utc":"2026-07-13T11:55:18Z"},"retraction_record":{"headline":"Joined to the Canadian frame by DOI, Retraction Watch records 143 works that OpenAlex does NOT flag as retracted, including 49 outright retractions. But the undercount is the smaller problem. 52 of these carry an EXPRESSION OF CONCERN, and OpenAlex HAS NO FIELD FOR ONE: `is_retracted` is a boolean over a state space with at least four values (retraction, expression of concern, correction, reinstatement), so it can express one and silently reports the rest as FALSE, which reads as 'fine'. Nor can a boolean carry WHY. This is finding 1's disease in a second schema: the canonical database cannot express the distinction the field turns on.","values":{"source":"Retraction Watch (Crossref-licensed), joined by bare lowercased DOI","frame_works_with_doi":3690953,"openalex_is_retracted_flags":1584,"matched_in_retraction_watch":1052,"openalex_misses":143,"outright_retractions_missed":49,"expressions_of_concern":52,"missed_by_nature":{"Expression of concern":52,"Retraction":49,"Correction":32,"Reinstatement":10},"top_reasons":{"Investigation by Journal/Publisher":335,"Unreliable Results and/or Conclusions":259,"Concerns/Issues about Data":233,"Investigation by Third Party":169,"Concerns/Issues about Referencing/Attributions":151,"Concerns/Issues about Results and/or Conclusions":122,"Concerns/Issues about Peer Review":109,"Investigation by Company/Institution":101},"openalex_has_eoc_field":false,"is_a_frame_route":false,"caveat":"This is an ATTRIBUTE, not a frame route. A retracted cardiology paper is retracted cardiology, not metaresearch, and admitting works on the strength of a retraction would let an interesting signal masquerade as the estimand. The DOI join can only see works that HAVE a DOI, and OpenAlex flags some works Retraction Watch does not match, which may be DOI drift rather than disagreement; the asymmetry reported here is one-directional on purpose (what RW adds), because that is the direction the join can support."},"computed_at_utc":"2026-07-13T11:55:00Z"},"screening_cost":{"headline":"The full rubric over the 4,299,418-work frame with two screeners costs $6,567, not the $30,633 an earlier version of this script reported: the rubric is a system prompt sent once per CALL, and the pilot's own chunks batch 155 works per call, so it was overcharged 4.7x. That error was not cosmetic. It made me propose a cheap TRIAGE in front of the screen, which is a RETRIEVAL STEP in a project whose central finding is that retrieval destroys these maps. With the arithmetic right the triage is unnecessary: the full rubric over EVERY work in the frame, plus a second screener on a 20,000-record sample, costs $1,110 at the v1 instrument and $1,279 at the locked v3.1 instrument that will actually run. THE PREFILTER IS DELETED. And the compute is SELF-FUNDED: the call's CAD $4,000 covers, in its own words, 'travel, accommodation, and related participation costs', not inference and not coder wages (D30).","values":{"measured_from":"pilot/screening/chunks/*.json (6-field), pilot/screening/haiku/p8_guided/chunk_*.json (8-field), docs/protocol/rubric.md; not asserted","chars_per_token_assumed":4,"tokens_per_work":312,"tokens_per_work_six_field_deviation":256,"payload_inflation_8_over_6":1.22,"costed_at_the_rubric_payload_not_the_deviation":true,"tokens_rubric":1878,"tokens_per_label":37,"batch_works_per_call":155,"frame_size":4299418,"grant_usd_approx":2900,"naive_cost_charging_rubric_per_work_usd":30633,"corrected_cost_two_screeners_usd":6567,"cost_overstatement_x":4.7,"cost_full_rubric_whole_frame_cheap_usd":1094,"cost_full_rubric_whole_frame_sonnet_usd":3283,"cost_second_screener_on_sample_usd":15,"second_screener_n":20000,"total_no_prefilter_usd":1110,"tokens_rubric_v31":4604,"tokens_per_label_v31":49,"cost_full_rubric_whole_frame_v31_usd":1261,"cost_second_screener_v31_usd":18,"total_no_prefilter_v31_usd":1279,"award_pays_for_compute":false,"award_purpose_in_the_calls_words":"travel, accommodation, and related participation costs","prefilter_needed":false,"prefilter_design_total_usd":1004,"prefilter_saving_usd":-106,"prefilter_deleted_because":"It is a RETRIEVAL STEP, in a project whose central finding is that retrieval destroys these maps (finding 12: the topic route finds 12% of the field). It existed only because the rubric was miscosted at once per WORK rather than once per CALL, overstating the alternative 5.1x. With the arithmetic right it saves almost nothing and costs the thesis. Deleted.","pilot_works_screened":6202,"frame_to_pilot_ratio":693,"method_that_scales":"batch inference, not agent fan-out","supersedes":"an earlier version charged the rubric once per work, reported $24,379 for the full-frame two-screener design, called it 'eight times the grant', and used that to justify a cheap prefilter. The rubric is a system prompt sent once per CALL, and the pilot's own chunks batch 155 works per call. DEVIATIONS.md D7.","caveat":"Token counts use a 4-chars-per-token approximation and list prices as of 2026-07; both will move, and the conclusion is robust to +/-25% in either. Screening the frame with a cheaper model than the pilot used makes the CHOICE OF MODEL more consequential, not less: finding 10 shows two screeners already imply base rates a factor of two apart. That is precisely why the second screener and the screener-swap range are reported, and why the human audit is the study."},"computed_at_utc":"2026-07-13T12:23:27Z"},"the_frame":{"headline":"The frame is BUILT, not estimated. All 482 partitions of a pinned OpenAlex snapshot were streamed and filtered, and the Canadian frame holds 4,299,418 works, each exactly once. For most of this project's life 'the frame' was 3,507,205, an EXTRAPOLATION from a single partition, hardcoded in six scripts; the estimate was 18% LOW, and every cost, audit-power and field-size figure computed against it has moved. The route that matters: 1,565,226 works (36.4% of the frame) are INVISIBLE TO AFFILIATION ALONE. A frame built on Canadian affiliation would hold 2,734,192 works and would never see them. That is finding 2 at frame scale, and it is why the frame is a union of four routes and why every record carries the provenance of the route that admitted it.","values":{"built_from":"all 482 partitions of a pinned OpenAlex snapshot, streamed and filtered; each work appears exactly once","partitions":482,"works":4299418,"prior_estimate":3507205,"estimate_was_low_by_pct":18,"estimate_was_an_extrapolation_from_one_partition":true,"route_ca_aff":2734192,"route_about_ca":1316408,"route_ca_fund":788103,"route_ca_venue":638161,"invisible_to_affiliation":1565226,"pct_invisible_to_affiliation":36.4,"with_venue":3902048,"pct_with_venue":90.8,"with_abstract":3296301,"pct_with_abstract":76.7,"pct_no_abstract":23.3,"french":237207,"pct_french":5.5,"built_utc":"2026-07-12T17:59:59Z","caveat":"The frame is bounded by OpenAlex. A work whose Canadian link is invisible to the metadata (no affiliation, no funder, no textual mention, no Canadian venue) cannot enter ANY frame by any method, and scholarship indexed by neither OpenAlex nor Erudit is not estimated here. That is a hard boundary of the data, not of this design, and it is stated rather than hidden. Erudit matches ZERO OpenAlex sources (finding 3), so the francophone share reported here (5.5%) measures the pipeline, NOT Canadian scholarship."},"computed_at_utc":"2026-07-13T11:55:21Z"},"three_model_screen":{"headline":"Three frontier models (Opus 4.8, GPT-5.6 high, Grok 4.5) screened the same 5,600 works, drawn from the real 4.3M frame under a design whose seven strata PARTITION it (an earlier five-stratum design could not reach 12.9% of the frame at all; D22). Design-weighted base rates span 2.54% to 3.81% (1.5x). But the RATE is not the finding, the SETS are: of the 274 works ANY model called metaresearch, only 104 (38%) were called metaresearch by ALL THREE, and 117 (43%) rest on a SINGLE model's opinion; pairwise Jaccard on the in-scope sets is about 50%. THE FIELD'S BOUNDARY IS NOT A LINE THE MODELS SHARE; IT IS A REGION THEY EACH CUT DIFFERENTLY, and that result is STABLE across n = 1,000, 2,000 and 5,600 (unanimity 37%, 37%, 38%). A SECOND, PRETTIER CLAIM DID NOT SURVIVE: at n = 2,000 the models agreed markedly more on 'is this about research at all' (1.43x here) than on 'is it in scope' (1.51x here), and this project said so in capitals; at n = 5,600 the two spreads are within noise (ratio 1.06) and the claim is WITHDRAWN (D23). The largest tier confusion is OUT-vs-T2, every time, at every sample size: the adjacent traditions the inclusiveness criterion exists to protect. The deliverable is not a base rate. It is the disagreement dossier, the 274 works that mark the empirical boundary, each carrying all three models' stated reasons, and the criteria that have to be written against them.","values":{"frame":"the real 4.3M-work Canadian frame (all 482 OpenAlex partitions)","payload":"the rubric's FULL eight fields, including venue (repairs D1)","sample":"5,600 works across seven exhaustive strata, with known selection probabilities and French oversampled","models":"Claude Opus 4.8; GPT-5.6 (high effort); Grok 4.5 (medium effort)","harness":"chunks randomized and manifest-logged before any model ran; the harness writes label files, never the model (repairs D11); every arm reconciled against the manifest (repairs D2)","n_labelled_by_all_three":5600,"tranche_homogeneity_p":0.63,"tranches_pool":true,"base_rate_weighted_pct":{"opus":3.81,"gpt":2.92,"grok":2.54},"between_model_spread_x":1.5,"jaccard_opus_gpt":50,"jaccard_opus_grok":50,"jaccard_gpt_grok":56,"n_about_research_at_all":{"opus":391,"gpt":325,"grok":274},"n_in_scope":{"opus":224,"gpt":163,"grok":148},"spread_about_research_x":1.43,"spread_in_scope_x":1.51,"variance_is_in_the_rubric_not_the_models":true,"called_in_scope_by_any":274,"unanimous_in_scope":104,"pct_unanimous_of_any":38,"in_scope_by_one_model_only":117,"pct_single_model_of_any":43,"contested_by_stratum":{"about_only":67,"venue_new":65,"residual":64,"aff_core":62,"fund_new":61,"aff_about":59,"french":54},"tier_disagreement_patterns":{"OUT/T2":77,"T1":65,"OUT/T1":57,"T2":30,"T1/T3":11,"T2/T3":10,"T1/T2":9,"OUT/T1/T2":8,"OUT/T1/T3":3,"OUT/T2/T3":3},"gpt_schema_violations_first_pass":18,"gpt_violation_note":"GPT-5.6 (high) wrote GENRE values ('empirical', 'conceptual') into the TIER field on 18 of 1,000 records in its first pass, in 3 of 20 chunks. The validator caught it because the harness reconciles files against a manifest rather than trusting the model's report. Those chunks were RE-RUN, not repaired: coercing a model's output to the schema is fitting the instrument to the data.","deliverable":"pilot/screening/frame1k/disagreement_dossier.json: every work any model called in-scope, with all three labels. This, not the base rate, is what the criteria must be written against.","caveat":"These are MACHINE labels and none of them is truth (finding 15). The unanimity rate is not accuracy: three models sharing training data can be wrong together, and they are most correlated exactly on the boundary cases the field's definition turns on. What this measures is where the RUBRIC is underspecified, which is a property of the instrument and is exactly what a criteria document needs. Base rates are design-weighted from a stratified sample, so they estimate the frame; the Jaccard and unanimity figures are unweighted set quantities over the sample and are NOT frame estimates."},"computed_at_utc":"2026-07-15T22:59:18Z"},"topic_route_recall":{"headline":"Scored against the rubric, the topic route retrieves 12% of Canadian metaresearch (95% CI 5.6-21.6%) at 60% precision: it misses 66 of 75. It fails because OpenAlex files a work by what it is about, and metaresearch about cardiology reads as cardiology: the field is invisible to topic retrieval precisely because it is about other fields.","values":{"n_screened":5737,"n_metaresearch":75,"n_retrieved_by_route":15,"true_positives":9,"false_positives":6,"false_negatives":66,"recall_pct":12,"recall_ci_lo_pct":5.6,"recall_ci_hi_pct":21.6,"precision_pct":60,"precision_ci_lo_pct":32.3,"precision_ci_hi_pct":83.7,"missed_works_by_field":{"Social Sciences":17,"Medicine":16,"Health Professions":7,"Business, Management and Accounting":4,"Economics, Econometrics and Finance":4,"Computer Science":3},"scored_by":"primary_topic.id (the key R/frame.R defines the route with), not display name","route_size_implied_by_sample":11241,"route_size_from_api":14873,"reconciliation_gap_x":1.32,"supersedes":"the earlier 32.4% figure (14,873/45,850), which divided a retrieved set by a true field size","caveat":"Small n on the positive class ( 75 metaresearch works, of which 15 were on the route), so the intervals are wide. Separately, the sample-implied route size does not reconcile with the API's 14,873 and I cannot say why at n = 15 on-route works; the likeliest cause is that one updated_date partition is not a uniform draw (finding 11). Recall is unaffected: it is a within-sample ratio, not an extrapolation."},"computed_at_utc":"2026-07-13T11:54:47Z"},"topics":{"headline":"Of 4,516 OpenAlex topics, 0 name metaresearch as a field. The 11 topics that do carry metaresearch content are scattered across 7 different OpenAlex fields.","values":{"n_topics_in_taxonomy":4516,"n_topics_naming_field":0,"topics_naming_field":[],"n_candidate_topics":11,"n_fields_spanned":7,"fields_spanned":["Arts and Humanities","Computer Science","Decision Sciences","Mathematics","Medicine","Psychology","Social Sciences"],"candidate_topic_ids":["T10102","T13607","T13516","T11937","T10206","T10582","T10267","T10778","T13558","T13284","T11875"]},"computed_at_utc":"2026-07-13T11:54:42Z"},"trial_linkage":{"headline":"The registry is the only REFERENCE STANDARD in this project not made of machine labels: ClinicalTrials.gov knows a Canadian trial happened independently of any pipeline, so it cannot be wrong in the pipeline's favour. Of 304 publications SPONSORS THEMSELVES reported as results of completed Canadian-located trials, the frame holds 160: a naive recall of 52.6%, which fell so close to the 44.5% THIS PROPOSAL OPENS WITH that it read as a replication. IT IS AN ARTIFACT, and running the disambiguation is the only thing that caught it: 137 of the 145 'misses' have NO CANADIAN AUTHOR (multi-site international trials with a Canadian SITE), and a frame of Canadian RESEARCH is CORRECT to exclude them. A trial with a Canadian site is not a publication with a Canadian author. Against the population the frame actually claims, recall is 95.2% (95% CI 90.8-97.9), and the real defect is 8 works OpenAlex holds WITH a Canadian author that the frame's own routes still missed. The frame is GOOD at this, the dramatic parallel was a coincidence between two different populations, and I had every incentive not to check. DEVIATIONS.md D16.","values":{"reference_standard":"ClinicalTrials.gov: completed trials with a Canadian location, and the RESULT publications the sponsors themselves reported","why_not_a_frame_route":"A trial registration is not a publication and not metaresearch. Registrations contribute NO records to the frame; the registry is a reference standard, not a source.","why_it_matters":"Every other recall number in this project is scored against MACHINE labels (finding 15). A registry knows a trial happened independently of any pipeline, so it cannot be wrong in the pipeline's favour. This is the only instrument here that measures the frame against a world that exists without it.","trials_retrieved":1000,"trials_with_result_publication":123,"pct_trials_with_result_publication":12.3,"known_result_pmids":596,"resolvable_to_doi":304,"present_in_frame":160,"naive_frame_recall_pct":52.6,"naive_recall_is_an_artifact":true,"naive_recall_ci":[46.9,58.4],"missing_total":145,"missing_no_canadian_author":137,"missing_route_gap_canadian_author":8,"missing_not_in_openalex":0,"claimable_population":168,"adjusted_recall_pct":95.2,"adjusted_recall_ci":[90.8,97.9],"is_metaresearch_recall":false,"caveat":"This measures FRAME recall (does the Canadian frame hold the publication at all?), NOT metaresearch recall: trial reports are primary research and the rubric screens them OUT. It is the precondition for screening, not the screen. The reference standard is the sponsor's OWN reported result publications, so it is incomplete in a known direction: sponsors under-report, which means the true set of trial publications is LARGER than the standard and this recall figure is measured only on the ones we can see. Trials are matched by Canadian LOCATION, which is not the same as Canadian authorship, so some result publications may have no Canadian author and legitimately fall outside the frame; that direction is not controlled here and it bounds the interpretation. Only PMIDs resolvable to a DOI can be looked up."},"computed_at_utc":"2026-07-13T11:55:20Z"},"v1_to_v2":{"headline":"WRITING THE MISSING SENTENCES RESOLVED 54% OF THE DISAGREEMENTS THEY WERE WRITTEN AGAINST. The 179 works three frontier models split on under rubric v1 were re-screened, by the same three models, under a v2 whose rules were written against a published, frequency-weighted census of WHICH SENTENCES THE RUBRIC WAS MISSING. 96 of 179 are now unanimous, and the pairwise Jaccard of the in-scope sets moves from 0.19 to 0.36. This is a measurement almost nobody makes, and it is available ONLY because the ambiguity was enumerated, adjudicated and PUBLISHED BEFORE it was resolved: the census went out first, so v2 could not be quietly tuned until the numbers looked good. SCOPE, stated rather than blurred: these works were SELECTED FOR DISAGREEMENT, so this says the rules resolve the cases they were written for; it does NOT say frame-wide agreement rose, because regression to the mean alone would move a set selected this way. The unbiased test is a full re-screen of all 5,600 under v2, prespecified, and it is what the funded work runs.","values":{"contested_under_v1":179,"unanimous_under_v2":96,"pct_resolved":54,"still_contested":83,"in_scope_by_model_v1":[129,68,53],"in_scope_by_model_v2":[73,32,44],"jaccard_v1_mean":0.187,"jaccard_v2_mean":0.362,"sequence":"lock v1; screen 5,600 works with three models; have an independent blinded judge adjudicate every disagreement AND name the seam that caused it; PUBLISH the seam census; write v2 against the census and lock it BEFORE re-screening; re-screen; report the difference, whatever it is. Publishing the census before resolving it is what stops v2 being tuned until the numbers improve.","caveat":"SELECTED FOR DISAGREEMENT. These 179 works are exactly the ones the three models split on under v1, so this measures whether v2's rules resolve the cases they were written for. It CANNOT support a claim about agreement across the frame: regression to the mean alone would move a set selected this way. The unbiased test is a full re-screen of all 5,600 under v2, prespecified, and it is what the funded work runs."},"computed_at_utc":"2026-07-13T11:55:28Z"},"zero_probability_region":{"headline":"THE SAMPLING DESIGN COULD NOT REACH 12.9% OF THE FRAME, AND NO WEIGHT CAN FIX THAT. A stratified design rests on one identity, sum(N_h) = N. It was never checked. 549,370 works (12.9% of the sampling frame) had an inclusion probability of EXACTLY ZERO: 366,856 that no predicate claimed, because aff_core excluded everything ABOUT Canada while about_only excluded everything AFFILIATED with Canada, so a work that was both fell between them; and 182,514 more whose predicate evaluated to SQL NULL, which `WHERE p` and `WHERE NOT p` BOTH decline to select, so they were in no stratum and were not even orphans. Nothing threw. Every stratum returned exactly the n it asked for, because a stratum cannot know about the works it was never asked about, and the design drew a clean textbook probability sample OF 87% OF THE FRAME while every number computed from it said 'the frame'. A zero-probability work is not underweighted, it is UNREACHABLE, and design weights are the guarantee this project leans on hardest. THE CELL IT DELETED WAS THE SECONDARY ESTIMAND'S: 328,912 Canadian-affiliated works about Canada, in a study whose secondary estimand is 'metaresearch about the Canadian research system'. Repaired by adding the exact NULL-safe complement as two strata; the 7 strata now partition the frame by construction (4,255,410 = 4,255,410), asserted on every build. It was found by an adversarial model adding up five numbers I handed it. The defect made the sample TIDIER and the variance SMALLER, which is why nothing about it felt wrong: the errors that survive are the ones that flatter you.","values":{"full_frame":4299418,"unscreenable_excluded":44008,"sampling_frame":4255410,"reachable_under_shipped_design":3706040,"orphaned_predicate_false":366856,"invisible_predicate_null":182514,"zero_probability_works":549370,"pct_of_sampling_frame":12.9,"aff_and_about_cell":328912,"strata_after_repair":7,"strata_sum_after_repair":4255410,"partition_holds":true,"found_by":"An adversarial model asked to attack the classifier design. Its first move was to add up the five design weights in a summary table I had handed it: 2000x1119 + 1000x664.2 + 750x536.8 + 750x310.9 + 500x335.8 = 3,705,875, against a frame of 4,299,418. I had never added them up.","caveat":"The repair does not retro-fix numbers produced under the broken design; those estimated a 3.7M subpopulation and were reported under the frame's name, and saying so IS the finding. Every design-weighted figure is now re-derived against the seven-stratum design. The five original predicates are kept byte-for-byte because 5,000 works had already been drawn from them by hash order, and widening a stratum silently re-draws it."},"computed_at_utc":"2026-07-13T11:55:25Z"}}}