Combining Text Mining and Manual Thematic Analysis to Understand Participant Experiences With Surveys
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Participant engagement in large-scale surveys is vital to health and social science research. Understanding the factors that affect participant engagement requires methods that can meaningfully interpret participant feedback. Open-ended survey questions can capture these kinds of nuanced responses, but can be challenging to analyze at scale. To address this challenge, this study combines computational text mining with a manual thematic analysis to examine over 15,000 open-text survey responses. These responses provided details about participants’ experiences with longitudinal public health surveys administered by Alberta’s Tomorrow Project. The text mining analysis consists of both sentiment analysis and topic modelling to identify broad patterns in participant experiences. This is combined with a thematic analysis of a purposive sample of 852 of these responses to validate the computational methods and identify themes not captured by the topic model. The findings suggest that many participants found the survey easy to complete and valued having the opportunity to participate. Challenges experienced included difficulty with recall-based questions, irrelevant response options, and program functionality issues. Providing customizable communication methods to adapt to individual preferences was also widely reported to be important by participants. We find that the conclusions from the analyses were largely consistent across the computational text mining and manual thematic analyses. However, the computational approaches overgeneralize or miscategorize certain responses, highlighting the benefits of incorporating both methods to analyse large volumes of qualitative survey responses.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.048 | 0.006 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it