Bias, accuracy and sample size in the systematic linking of historical records
Bibliographic record
Abstract
IntroductionLinking distinct historical sources on an automated basis directs attention to the quality and representativeness of the linked data created by these systems. Linking with time-invariant personal characteristics arguably minimizes bias or departures from representativeness even though a wider set of features might generate more links.
 Objectives and ApproachThe objective of this research is to compare, evaluate and understand bias when two linking methodologies are employed on the same data sources. Our approach to studying this problem is by comparing linked records from Canadian censuses (linking 1871 to 1881) generated by two different linking strategies. The first method is a support vector machine based classification model on time-invariant individual characteristics. Using this method a large number of multiple matches is generated, as records look similar on a small number of time-invariant individual characteristics. The second method adds a second stage of disambiguating multiple matches using family information.
 ResultsWe compare the links produced by the two methods used in the study and we discuss the results. The comparison is in terms of number of links produced, their quality (false positive rate) and the bias of the linked data produced. A complication is that there are many dimensions of bias. Even time-invariant criteria typically generate some bias.
 As expected, the two-step process produces a larger linked sample. Interestingly, it also produces a lower error rate and different patterns of bias. Both methods understate the Quebec-born, French-ethnicity, the unmarried and adolescents. Unexpectedly, the bias in favour of married people is larger using individual (first method) than family information (second method). However, family-based linking does over-represent young children.
 Conclusion/ImplicationsResults suggest that neither method will be universally preferable. Rather, the choice of research question may affect the preferred balance of biases and link rate. Fortunately, the advance of computational capacity allows a researcher to select a method that generate links most appropriate for the problem at hand.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.004 | 0.032 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".