Battling the bots: Defending against fraudulent responses while conducting an international community-engaged web-based survey with people living with Long COVID (Preprint)
Bibliographic record
Abstract
BACKGROUND Web-based surveys involving self-reported questionnaires are vulnerable to fraudulent responses. Advancements in artificial intelligence (AI) and bots has introduced additional challenges to preventing and identifying fraudulent responses to online questionnaires. OBJECTIVE To describe our experiences with fraudulent responses, strategies for preventing and identifying fraudulent responses, lessons learned when conducting a web-based survey with adults living with Long COVID, and recommendations for web-based survey research. METHODS The Long COVID and Episodic Disability Study is an international community-engaged study among adults living with Long COVID in Canada, Ireland, United Kingdom (UK), and United States (US). We conducted a longitudinal web-based survey, with online administration of a self-reported questionnaire at two timepoints (Time One and Time Two), one week apart. We recruited through Long COVID community groups using social media, emails, and word of mouth. The survey was disrupted by fraudulent responses, including bots. To defend data integrity, we implemented the following strategies: a) pausing our initial launch (Wave One), b) developing and implementing screening criteria to identify fraudulent responses, and c) re-launching the web-based survey (Wave Two) with revised recruitment strategies and questionnaire design to prevent, and identify fraudulent responses. RESULTS We received 4663 responses for Time One and 1281 responses for Time Two, of which we retained 798/4663 (17%) and 629/1281 (49%). Strategies for preventing fraudulent responses included enabling survey protection features in survey software, shutting down compromised survey links, avoiding recruitment via public social media groups, and removing mention of a financial incentive from recruitment materials. Strategies for identifying fraudulent responses included monitoring response completion times, start and end time stamps, geolocation, and screening for suspicious email address characteristics and duplicates. CONCLUSIONS Our lessons learned fell into three areas: 1) survey-design and implementation to prevent and identify fraudulent and bot-generated responses; 2) recruitment strategies to mitigate risk of disruption by bots; and 3) responding to disruptions caused by fraudulent and bot responses. We recommend the following tactics to prevent and mitigate the risks of fraudulent and bot responses when administering online web-based questionnaires: a) review current literature and connect with researchers and Research Ethics Boards (REBs) about strategies prior to launching; b) invest in survey software with rigorous info-security technology; c) employ bot-detection features available in survey software prior to launching; d) design questionnaire items to identify bots and fraudulent actors; e) tailor criteria for identifying fraudulent and bot responses to the characteristics of the target population; f) avoid recruitment in public social media groups; g) engage community leaders in tailored and targeted recruitment; h) avoid advertising incentives; i) shut down compromised links rapidly; j) communicate with the REB about disruptions; and k) combine automated with manual methods to identify potentially fraudulent responses in a timely manner. CLINICALTRIAL n/a INTERNATIONAL REGISTERED REPORT RR2-10.1136/bmjopen-2022-060826
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Direct model labels (unvalidated)
Per-model category and study-design labels from the labeling rounds. They are machine output, unvalidated, and the disagreement between models ships as data. No study design here is MEDLINE-validated yet.
| Model arm | Categories | Study design | Confidence |
|---|---|---|---|
| gemma | Metaresearch Domain: Methods · Genre: Empirical About the Canadian research system: no · About a Canadian topic: no | Not applicable | medium |
| gpt | Metaresearch Domain: Methods · Genre: Empirical About the Canadian research system: no · About a Canadian topic: no | Observational | high |
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.015 | 0.043 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.003 | 0.001 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.001 | 0.004 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedLabeled directly by 2 models reading the full record.
The models disagree on parts of this classification; every voice is preserved in the section at the end of the page.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".