Linguistic Markers of AI-Generated Text: A Comparative Analysis of Machine-Identified and Human-Inferred Predictors
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
The widespread use of generative AI tools has significantly changed academic and professional writing, due to their ability to produce texts that mimic human writing styles. As a result, there are growing concerns about academic integrity, authorship, and the possible spread of misinformation. This study addresses the challenge of finding clear language features that can distinguish AI-generated texts from human-written ones, which is a gap that current detection tools have not resolved. Prior work shows that AI can produce coherent and context-relevant text by learning from large data sets and that features such as readability, lexical diversity, perplexity and burstiness, and sentiment are useful in detection, though results have been mixed. Our main goal is to determine which of these language features best predict AI authorship and to compare these machine-identified signals with the cues that human reviewers use. We analyze 100 mental health abstracts from 2022, published before the release of ChatGPT from OpenAI, and generate 100 additional abstracts using ChatGPT. We use a quantitative approach, using natural language processing methods such as readability, analytic writing index, lexical diversity, including measures like the measure of textual lexical diversity and type-token ratio, perplexity, burstiness, sentiment, common word groups, term frequency-inverse document frequency scores, voice usage, punctuation, and tone. These measures are then used to train a machine learning model to pick out the top predictors of AI-generated content. In addition, we will conduct a survey of 200 participants (expected) from Toronto Metropolitan University to collect ratings on abstract quality and ask participants to identify if each abstract was written by a human or generated by AI, along with background and AI usage information. We expect our analysis to show that AI-generated abstracts tend to have lower lexical diversity, simpler sentence structures, and lower perplexity, and that human reviewers will struggle to correctly identify AI-generated abstracts, especially when the differences are subtle. The findings add to our existing knowledge of the key language features that signal AI authorship and support the creation of better detection tools that combine machine analysis with human insight, ultimately helping to protect academic integrity and guide ethical authorship.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it