Linguistic Markers of AI-Generated Text: A Comparative Analysis of Machine-Identified and Human-Inferred Predictors
Bibliographic record
Abstract
The widespread use of generative AI tools has significantly changed academic and professional writing, due to their ability to produce texts that mimic human writing styles. As a result, there are growing concerns about academic integrity, authorship, and the possible spread of misinformation. This study addresses the challenge of finding clear language features that can distinguish AI-generated texts from human-written ones, which is a gap that current detection tools have not resolved. Prior work shows that AI can produce coherent and context-relevant text by learning from large data sets and that features such as readability, lexical diversity, perplexity and burstiness, and sentiment are useful in detection, though results have been mixed. Our main goal is to determine which of these language features best predict AI authorship and to compare these machine-identified signals with the cues that human reviewers use. We analyze 100 mental health abstracts from 2022, published before the release of ChatGPT from OpenAI, and generate 100 additional abstracts using ChatGPT. We use a quantitative approach, using natural language processing methods such as readability, analytic writing index, lexical diversity, including measures like the measure of textual lexical diversity and type-token ratio, perplexity, burstiness, sentiment, common word groups, term frequency-inverse document frequency scores, voice usage, punctuation, and tone. These measures are then used to train a machine learning model to pick out the top predictors of AI-generated content. In addition, we will conduct a survey of 200 participants (expected) from Toronto Metropolitan University to collect ratings on abstract quality and ask participants to identify if each abstract was written by a human or generated by AI, along with background and AI usage information. We expect our analysis to show that AI-generated abstracts tend to have lower lexical diversity, simpler sentence structures, and lower perplexity, and that human reviewers will struggle to correctly identify AI-generated abstracts, especially when the differences are subtle. The findings add to our existing knowledge of the key language features that signal AI authorship and support the creation of better detection tools that combine machine analysis with human insight, ultimately helping to protect academic integrity and guide ethical authorship.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.006 | 0.071 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.007 | 0.004 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.003 | 0.002 |
| Open science | 0.000 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".