Multilingual Research Projects: Non-Latin Script Challenges for Making Use of Standards, Authority Files, and Character Recognition
Bibliographic record
Abstract
Academic research about digital non-Latin script (hereafter: NLS) research data can pose a number of challenges just because the material is from a region where the Latin alphabet was not used. Not all of them are easy to spot. In this paper, I introduce two use cases to demonstrate different aspects of the complex tasks that may be related to NLS material. The first use case focuses on metadata standards used to describe NLS material. Taking the VRA Core 4 XML as example, I will show where we found limitations for NLS material and how we were able to overcome them by expanding the standard. In the second use case, I look at the research data itself. Although the full-text digitization of western newspapers from the 20th century usually is not problematic anymore, this is not the case for Chinese newspapers from the Republican era (1912–1949). A major obstacle here is the dense and complex layout of the pages, which prevents OCR solutions from getting to the character recognition part. In our approach, we are combining different manual and computational methods like crowdsourcing, pattern recognition, and neural networks to be able to process the material in a more efficient way. The two use cases illustrate that data standards or processing methods that are established and stable for Latin script material may not always be easily adopted to non-Latin script research data.Des recherches académiques sur les recherches de textes numériques qui ne sont pas en alphabet latin (désormais NLS) peuvent poser plusieurs défis, car le matériel vient d’une région où l’alphabet latin n’était pas utilisé. Ils ne sont pas tous faciles à trouver. Dans cet article, je vais présenter deux cas d’utilisation pour démontrer les différents aspects des tâches complexes qui pourraient être reliées au matériel NLS. Le premier cas d’utilisation focus sur les standards de métadonnées utilisés pour décrire le matériel NLS. En prenant comme exemple le VRA Core 4 XML, je montre où se trouvent les limitations pour le matériel NLS et comment nous sommes capables de les surmonter en augmentant les standards. Pour le deuxième cas d’utilisation, je regarde les données de recherches elles-mêmes. Même si la numérisation de textes complets de journaux occidentaux du 20e siècle n’est plus problématique, ce n’est pas le cas pour les journaux chinois de l’ère républicaine (1912-1949). Un obstacle majeur est la densité et la complexité de la mise en page, ce qui empêche les solutions OCR (reconnaissance optique de caractères) de se rendre à la partie de reconnaissance des caractères. Dans notre approche, nous avons combiné des méthodes manuelles et computationnelles différentes comme l’externalisation ouverte (crowdsourcing), la reconnaissance de motifs, et le réseau neuronal pour procéder au matériel de manière plus efficace. Les deux cas d’utilisations démontrent que les données standards ou les méthodes de traitement qui sont établies et stables pour le matériel en alphabet latin ne peuvent être utilisées facilement pour des données qui ne sont pas en alphabet latin.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.076 | 0.225 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.006 | 0.009 |
| Science and technology studies | 0.007 | 0.011 |
| Scholarly communication | 0.021 | 0.031 |
| Open science | 0.004 | 0.015 |
| Research integrity | 0.004 | 0.004 |
| Insufficient payload (model declined to judge) | 0.011 | 0.010 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".