Evaluating the accuracy of forced alignment across Mandarin varieties
Bibliographic record
Abstract
Forced alignment is widely used in phonetics to align transcripts with acoustic signals. These tools are trained on specific language varieties; it is unclear if they generalize to others. Previous research on English by MacKenzie and Turton (2020) finds good agreement between automated and human alignments for varieties that differ from the training variety. Such evaluation has only been carried out for English. We evaluate the level of human-aligner agreement on four Mandarin varieties (Canto, Shanghai, Beijing, and Tianjin). For each variety, two recordings from the HUB5 Corpus (LDC 1998) were aligned manually and by the Montreal Forced Aligner [McAuliffe et al. (2017)] using acoustic models trained on Beijing, Wuhan, and Hekou Mandarin [Schultz (2002)]. We find strong agreement between human and machine-aligned phone boundaries, with 17 ms as the median onset displacement. A mixed model identifies little variation across varieties or according to speech rate, but significant interindividual variation. Notably, despite the generally close agreement between the machine and human alignments, for two of the speakers, more than 10% of the alignments are displaced by over 100 ms. In sum, the Mandarin forced-aligner yields reliable alignments for out-of-training varieties, but manual checking of the results is still crucial.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.007 | 0.028 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.002 | 0.001 |
| Insufficient payload (model declined to judge) | 0.004 | 0.004 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".