Scarlet Cloak and the Forest Adventure: The Issue of False Positives in AI Detection Tools

The advent of Large Language Model (LLM) generative writing tools is producing what increasingly looks like a Kuhnian paradigm shift (Kuhn 1962; Bai̇Doo-Anu and Owusu Ansah 2023; Lambert and Stevens 2023; Kirwan 2023) in the production and assessment of writing. As the tools have been adopted by students and professionals, they have forced educators, judges, policy-makers, publishers, curators, and others whose professions involve the production or evaluation of creative products to adjust their policies, and expectations repeatedly (Köbis and Mossink 2021). While the use of such tools is a problem across the professions, it represents a particular challenge in the case of qualitatively assessed student learning in Post-Secondary Education (PSE). Traditionally, students have been assessed using written work on the assumption that such assignments are representative of the assessed student’s writing, research, and reasoning abilities. Since LLM tools can produce copy at a reasonably high level from minimal prompts, they may represent a fundamental challenge to a well-established disciplinary practice. Heavy-handed initial reactions — banning “the use of AI” and using AI detectors (e.g. Edward Tian [@edward_the6] 2023; Turnitin, n.d.) to root out non-compliance — are already producing problematic results: evidence suggests that second-language speakers and first-language speakers of non-privileged varieties are more likely to produce writing that (falsely) suggests the use of LLM tools (Sample 2023). As such tools become more widely integrated into standard consumer software, it is becoming increasingly difficult to avoid any interaction between human writers and LLM-based text-generating tools: search engines, grammar checkers, paraphrasing and summary tools, and word processors have all begun to use generative AI, meaning that students may be integrating AI-generated text unknowingly into their work — and, perhaps more importantly, doing so in ways that conform to the intended use of these (often instructor-recommended) tools. In this paper we present the result of a study on the impact of a LLM-based tools on student and professional writing based on discussions with students about commonly used applications. The tools considered range from text-generating bots such as ChatGPT to hybrid applications such as EditPad, Writefull, and Grammarly. Our method applied each tool in the intended fashion to a number of different texts written by professional academics or a creative writer, and then checked the results of these interventions (e.g. [as pseudo-prompts] “rewrite,” “paraphrase,” “improve,” “grammar check,” etc.). Our conclusion is that teachers in PSE must approach the use of LLM-based Generative AI (and especially interpreting the results of AI-detectors) with great caution: not only do different tools produce potentially different thumbprints, perfectly legitimate uses of AI-enabled tools result in “false positives” when the results are taken as definitive evidence of academic misconduct. The question we need to ask is not “has this writing been touched by AI?”, but, increasingly, “how was the inevitable presence of AI handled?” After demonstrating the impact of the various tools on the different texts in our sample, we conclude with advice about how to construct suitable (if necessarily interim) assessment policies in this fast-developing area.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.