Predicting Threats to Academic Integrity: A Text-Mining and Scenario Modeling Framework
Bibliographic record
Abstract
Cleverly crafted words become the weapon of choice in the realm of adversarial stylometry, where authors don masks of deception, bending language to confound the very algorithms designed to unveil their true identity" -ChatGPT, May 2023 when asked to define adversarial stylometry.This thesis will combine text-mining and scenario modeling to identify and predict threats.The problem of academic integrity is significant in society today, given all of the technological advancements.This research focused explicitly on contract cheating to narrow the scope.Contract cheating can take place on an online platform where a customized paper can be purchased or arranged informally with family or a friend.A financial transaction is not always present.Because the work acquired through these means is original, it circumvents text-matching detection.Therefore, this presented an interesting problem to study, understand, and test.This interdisciplinary research demonstrated that text-mining and scenario modeling can predict future threats to academic integrity.The text-mining technique of topic modeling was utilized to identify weak signals, and two diverse, separate models were built to account for the abundance of varied information and the complexity of the problem.The similarities between the two models were triangulated through comparison using word embeddings.This aided with validation and allowed causal relationships to be more easily seen by reinforcing and introducing weak signals.Weak signals provide an opportunity or a threat in the business world.We focused on threats solely, as the common thread within the contract cheating literature was the lack of detection.The dominant weak signal of adversarial stylometry was discovered in the topic model through the measurement of distance.Adversarial stylometry refers to the deceptive manipulation of text to avoid authorship detection.This weak signal was applied, a definition and threat model were created, and the weakness was quantified based on a three-tier threat severity.As stories are often more effective as a catalyst for change than models or numbers, ChatGPT generated this output as narrative scenarios in the students and course instructors' voices.As a result, a methodological framework was established that combined explanatory and predictive components.Scholars can replicate the demonstrated methods to predict threats on similar or new problems.Moreover, practitioners can use this methodology as a tool i) to automate threat predictions over time to remain competitive and ii) strategically plan for the future of academic integrity.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.003 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.006 | 0.011 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; both teacher heads agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".