Study of Methods for Assessing Research Topic Elicitation and pRioritization (SMARTER): Study Protocol to Compare Qualitative Research Methods and Advance Patient Engagement in Research
Bibliographic record
Abstract
BACKGROUND: Involving patients as partners in research is a defining characteristic of patient-centered outcomes research (PCOR). While patients' experiential knowledge of a health condition or treatment may yield research priorities not reflected by researchers and policy makers, the methods for identifying and effectively collaborating with patients are still evolving. Patient registries and crowdsourcing may offer ease of access and convenience to both researchers and patients. Surveys and focus groups, including online modalities, have been described for prioritizing research topics. However, little is known about how these different methods compare in producing consistent priorities and similar perceptions of engagement quality among participants. OBJECTIVE: The aims of this study are (1) to compare how different engagement methods used to elicit patient priorities for research perform as measured by rankings for priorities generated and participant satisfaction; and (2) to determine characteristics of individuals choosing to participate in research prioritization activities. METHODS: Participants in the Back pain Outcomes using Longitudinal Data (BOLD) patient registry, established to evaluate the natural history of back pain among individuals 65 years and older, and participants on the Amazon Mechanical Turk (MTurk) crowdsourcing platform, to provide input on priorities for research via a questionnaire, are invited. For BOLD participants, we subsequently randomize interested respondents to 1 of 3 interactive prioritization activities to further develop priorities: a Delphi panel, an online crowd voting activity, or an in-person facilitated prioritization activity using nominal group technique (NGT). Participants involved in each activity complete a survey to evaluate the quality of the experience and a subset of these participants discuss their experience further in an interview. Descriptive statistics are used to characterize the rankings produced by each method and compare the top 5 rated topics resulting from each prioritization activity. We use rank-ordered logistic regression models to identify associations of the ranked priority topics with baseline patient characteristics. We analyze responses to the evaluation using a mixed-methods approach wherein we tabulate responses to Likert-scale questions and use content analysis to enumerate themes emerging from interviews for the 3 activities. RESULTS: In Phase I, we invite approximately 3000 BOLD participants and 500 Amazon MTurk workers to complete a research topic prioritization survey. Based on these results, we include additional topics into a subsequent prioritization survey. In Phase II, we invite BOLD participants to join 1 of 3 activities: 90 participants for Delphi panel, 100 participants for crowd voting, and 60 participants for focus groups. Of the Phase II participants, 30 will be interviewed to evaluate the activities. CONCLUSIONS: This study informs decisions about how to conduct outreach to patient registry participants for providing input on research priorities, how individuals 65 years and older wish to participate in engagement activities, and how different research prioritization methods compare in terms of rankings generated and participant satisfaction.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Direct model labels (unvalidated)
Per-model category and study-design labels from the labeling rounds. They are machine output, unvalidated, and the disagreement between models ships as data. No study design here is MEDLINE-validated yet.
| Model arm | Categories | Study design | Confidence |
|---|---|---|---|
| gemma | no category Domain: not available · Genre: Protocol About the Canadian research system: no · About a Canadian topic: no | Not applicable | low |
| gpt | Metaresearch Domain: Methods · Genre: Protocol About the Canadian research system: no · About a Canadian topic: no | Qualitative | high |
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.285 | 0.316 |
| Meta-epidemiology (narrow) | 0.005 | 0.005 |
| Meta-epidemiology (broad) | 0.005 | 0.004 |
| Bibliometrics | 0.007 | 0.007 |
| Science and technology studies | 0.006 | 0.006 |
| Scholarly communication | 0.005 | 0.005 |
| Open science | 0.005 | 0.005 |
| Research integrity | 0.007 | 0.008 |
| Insufficient payload (model declined to judge) | 0.060 | 0.018 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedLabeled directly by 2 models reading the full record.
The models disagree on parts of this classification; every voice is preserved in the section at the end of the page.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".