Detecting the Presence of COVID-19 Vaccination Hesitancy From South African Twitter Data Using Machine Learning
Bibliographic record
Abstract
Very few social media studies have been done on South African user-generated content during the COVID-19 pandemic and even fewer using hand-labeling over automated methods. Vaccination is a major tool in the fight against the pandemic, but vaccine hesitancy jeopardizes any public health effort. In this study, sentiment analysis on South African tweets related to vaccine hesitancy was performed, with the aim of training AI-mediated classification models and assessing their reliability in categorizing user-generated content. A dataset of 30,000 tweets from South Africa was extracted and hand-labeled into one of three sentiment classes—positive, negative, and neutral. The machine learning models used were long short-term memory (LSTM), bi-LSTM, support vector machine (SVM), bidirectional encoder representations from transformers (BERT)-base-cased and the RoBERTa-base models, whereby their hyperparameters were carefully chosen and tuned using the WandB platform. We used two different approaches when we preprocessed our data for comparison—one was a semantics-based method, while the other was a corpus-based method. The preprocessing of the tweets in our dataset was performed using both of these two different methods, respectively. All models were found to have low F1-scores within the range of 45%–55%, except for BERT and RoBERTa, which both achieved significantly better measures with overall F1-scores of 60% and 61%, respectively. Topic modeling using a latent Dirichlet allocation (LDA) was then performed on both the correctly classified and misclassified tweets of the RoBERTa model to gain insight on how to further improve the accuracy of these models.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.002 |
| Science and technology studies | 0.005 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".