Anishinaabemowin AI Translation Tool Evaluation Dataset

This dataset contains comprehensive evaluation results from systematic testing of five commercially available AI translation tools for Anishinaabemowin (Ojibwe language). The research addresses critical questions about AI reliability for Indigenous language education, particularly for second-language learners in off-reserve locations with limited access to fluent speakers. Phase 1: Comparative Tool Analysis 5 AI translation tools tested (AnythingTranslate, Musely.ai, Ojibwechat, ChatGPT Indigenous Language Supporter, Claude.ai Pro) 17 Anishinaabemowin words plus 2 complex sentences Structured documentation with validation against authoritative print and online dictionaries Systematic consistency testing through repeated queries to demonstrate all models have errors Data files: 4 CSV files (2025-08-12-14 Ojibwe Translation Tests) Phase 2: Guardrail Development JSON-based cultural protocol guardrails for Indigenous language AI Conversational testing with ChatGPT, Claude.ai, and Perplexity Evaluation of bilingual response formats (Anishinaabemowin first, English in brackets), gentle correction methodologies, and community authority deference Assessment of educational suitability for language learning contexts Data files: AI TEST 2025-08-17 JSON File.md (guardrail specifications) + 4 conversation test files Phase 3: Open Conversation Testing "Cold start" conversations without JSON guardrails Testing of Perplexity, ChatGPT Indigenous Language Bot, and Claude.ai Mixed Anishinaabemowin-English conversation evaluation Error pattern documentation and learning risk assessment Data files: 4 conversation test files KEY FINDINGS Tool Performance: AnythingTranslate and Musely.ai demonstrated complete unreliability with different translations each query ChatGPT, Claude.ai, and Perplexity showed best overall performance with grammatical understanding but persistent word stem errors Ojibwechat demonstrated some understanding but produced grammatical errors and defaults to translation All systems exhibited errors requiring fluent speaker verification Critical Translation Error Example: English: "stop walking" AI Translation (ChatGPT): "gego bimosen" ❌ (means "don't walk") Correct Translation: "boon bimosen" ✓ (means "stop walking") This demonstrates English-based reasoning (negation vs. cessation) that poses significant learning risks. JSON Guardrail Results: Successfully enforced culturally appropriate behaviors (bilingual formats, gentle correction methods) Underlying accuracy issues persist despite behavioral constraints Demonstrates potential for Indigenous pedagogy integration with critical limitations Risk Assessment: Highest risk: Beginner learners without verification access (cannot identify errors) Moderate risk: Intermediate learners with occasional fluent speaker access Lowest risk: Advanced learners with regular community speaker access Critical concern for off-reserve second-language learners (one-third of Indigenous language speakers per 2021 Census) RESEARCH CONTEXT This research responds to the 4% decline in Indigenous language speakers from 1991-2021 (Statistics Canada) and the increasing proportion of second-language learners. With Indigenous communities seeking technological tools to support language revitalization, this dataset provides critical evidence for evaluating AI tool safety and developing community-controlled alternatives. The data supports development of culturally-grounded AI evaluation frameworks—specifically contributing to the proposed "Anishinaabe Turing Test"—that measure AI systems against Indigenous knowledge protocols rather than European cognitive patterns. DATASET CONTENTS Phase 1 Data Files: 2025-08-12-14 Ojibwe Translation Tests [4 CSV files] - Comparative tool analysis results with 17 words and 2 complex sentences Phase 2 Data Files: AI TEST 2025-08-17 JSON File.md - Cultural protocol guardrail specifications in JSON format AI TEST 2025-08-17 JSON [4 MD files] - JSON guardrail conversation testing results (Claude, ChatGPT, Perplexity) Phase 3 Data Files: AI TEST [4 MD files] - Open conversation testing results without guardrails Supporting Materials: methodology.md - Detailed testing protocols and validation procedures README.md - Complete documentation and usage guidelines LICENSE - CC BY 4.0 license text METHODOLOGY Testing employed standardized protocols including: Consistent word/phrase lists across all tools (17 words + 2 complex sentences) Multiple researcher cross-validation Authoritative dictionary verification against Anishinaabemowin print and online sources Systematic documentation for reproducibility Initial consistency tracking to demonstrate all models have errors Iterative testing with multiple queries to assess reliability Cultural Protocols: Research conducted with respect for Indigenous knowledge systems and in support of language revitalization efforts. Testing focused exclusively on publicly available AI tools to assess community safety. No traditional knowledge was used or shared with the AI systems. Researchers worked exclusively with language data the LLMs already possessed, ensuring protection of cultural knowledge. IMPLICATIONS For Researchers: Framework for evaluating AI tools for Indigenous languages Methodology for culturally-grounded AI assessment Evidence of systematic errors in commercial translation systems For Educators: Evidence-based assessment of tool safety for classroom use Documentation of error patterns for post-use correction protocols Verification protocols requiring fluent speaker oversight For Indigenous Communities: Data to inform AI tool adoption decisions before integration into language programs Support for community-controlled AI development initiatives Validation of data sovereignty principles in language technology Protection against potential harm from unreliable AI systems EDUCATIONAL IMPACT AI systems pose high risk for isolated learners who may internalize errors as correct language patterns. Current tools require intensive community oversight and systematic verification protocols. Without fluent speaker access, beginners cannot distinguish between correct and incorrect outputs, leading to potential long-term language acquisition problems. TECHNICAL SPECIFICATIONS Data Formats: CSV (comma-separated values), Markdown (.md) Character Encoding: UTF-8 Testing Period: August 12-17, 2025 Number of AI Systems Tested: 5 translation tools + 3 conversational LLMs Research Context: PhD research, York University Digital Media program Funding Support: IBET (Indigenous and Black Engineering and Technology), Abundant Intelligences Tkaronto Pod CITATION When using this dataset, please cite: McConnell, A., & Ly, J. (2025). Anishinaabemowin AI Translation Tool Evaluation Dataset (Version 1.0.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.17274098 Associated Publication: McConnell, A., & Ly, J. (2025). "Anishinaabemowin Aabajichigan Gaawiin Nibwaakaasii (Efficacy of AI for Ojibwe Language Education)." Connected Minds 2025, Toronto, ON, October 6-8, 2025. BibTeX: @dataset{mcconnell_anishinaabemowin_2025, author = {McConnell, Andrew and Ly, Jasmine}, title = {Anishinaabemowin AI Translation Tool Evaluation Dataset}, year = {2025}, publisher = {Zenodo}, version = {1.0.0}, doi = {10.5281/zenodo.17274098}, url = {https://doi.org/10.5281/zenodo.17274098} } KEYWORDS Indigenous languages, Anishinaabemowin, Ojibwe language, artificial intelligence, machine translation, language revitalization, language education, AI evaluation, Indigenous data sovereignty, cultural appropriateness, language learning, second language acquisition, Indigenous pedagogy, AI ethics, language technology, ChatGPT, Claude, Perplexity, translation tools, JSON guardrails, Indigenous futurities, off-reserve language learning LICENSE Creative Commons Attribution 4.0 International (CC BY 4.0) https://creativecommons.org/licenses/by/4.0/ ACKNOWLEDGMENTS This work is supported by IBET (Indigenous and Black Engineering and Technology) and Abundant Intelligences, Tkaronto Pod. Research conducted as part of PhD studies in Digital Media at York University. GITHUB REPOSITORY Complete dataset and documentation available at: https://github.com/giigdo/AI_Bot_Tests_Q3_2025

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.