Assessing the accuracy of MT and AI tools in translating humanities or social sciences Arabic research titles into English: Evidence from Google Translate, Gemini, and ChatGPT
Bibliographic record
Abstract
Breakthroughs and advances in translation technology by virtue of AI-powered MT tools and techniques contributed significantly to providing near-perfect translation. This study aims to evaluate the accuracy of three translation technologies (Google Translate, Gemini, and ChatGPT) in translating multidisciplinary Arabic research titles in the Humanities and Social Sciences into English. A corpus of 163 titles of Arabic research articles from various disciplines, including media studies, literature, linguistics, education, and political science, was extracted from a Scopus-indexed journal, namely Dirasat: Human and Social Sciences Series. The research methodology in the present study lends itself largely to Koponen’s (2010) translation error strategy framework. Based on the data analysis, the findings showed that the renditions provided by these programs were categorically marked with either sense or syntax errors, which often rendered the translations inaccurate. Many polysemous terms with multiple related senses were mistranslated. The results showed that the Gemini translations contained the least errors. In contrast, the human translations contained the least mistranslation and diction errors. Google Translate and ChatGPT, on the other hand, contained the highest number of equivalence-based errors. Unexpectedly, the human translations contained the highest number of syntactic errors, reflecting a lack of target language proficiency. The study's conclusions and findings would be beneficial to translators, students, and scholars who may consider translating their Arabic study research titles and abstracts through the most commonly used AI tools.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.051 | 0.280 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.006 | 0.008 |
| Science and technology studies | 0.002 | 0.003 |
| Scholarly communication | 0.005 | 0.008 |
| Open science | 0.002 | 0.003 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.002 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".