Identifying and Analyzing Provocative Text: An XAI Approach to Classification and Feature Selection
Bibliographic record
Abstract
In this work, several AI models and methods are compared to see whichis best at classifying provocative texts in order to understand which features, such as specific words or kinds of words, are most influential in determining whether a piece of text is classified as provocative. This was done with two different categories of models: the more classical supervised learning models (e.g. RandomForests and Support Vector Machines (SVM) and the more novel transformer based classifiers. After performing a grid search and threshold tuning, the best perfoming classical model was an SVM, which achieved a minority-class F1 of 0.69 on a balanced test set, while the Bayesian XLNet was the best performing transformer, peaking at only 0.28 minority-F1 and 0.61 macro-F1 on skewed data. To understand the underwhelming performance of the models, the uncertainty was using the Bayesian XLNet model and found heteroscedastic aleatoric uncertainty in the data. This means that the task itself has inherent uncertainties. This uncertainty coupled with the skewed dataset explains the struggle of the models well. The models are then used to calculate the SHAP (SHapley Additive exPlanations) values of words to understand what impact individual words have on the classification. While some unexpected terms were found, like get, most words that indicate provocativeness are more explicit, such as stupid. These results are likely due to the underperformance of the models, which were unable to capture more nuanced forms of provocation. Together, the uncertainty and SHAP analyses demonstrate that provocativeness is anuanced, context-dependent phenomenon — data scarcity and label ambiguity limit performance, and no model can fully overcome that innatetask uncertainty. Note: This research paper presents some texts that might be perceived as offensive. Please note that these are only included when necessary for academic reasons and visualisations, and are by no means intended to hurt or offend readers.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.002 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".