Assessing Facilitator Fidelity to Principles of Public Deliberation: Tutorial
Bibliographic record
Abstract
Public deliberation, or deliberative democracy, is a method used to elicit informed perspectives and justifiable solutions to ethically fraught or contentious issues that affect multiple stakeholder groups with conflicting interests. Deliberative events bring together stakeholders (deliberants) who are provided with empirical evidence on the central issue or concern and then asked to discuss the evidence, consider the issue from a societal perspective, and collectively work toward a justifiable resolution. There is increasing interest in this method, which warrants clear guidance for evaluating the quality of its use in research. Most of the existing literature on measuring deliberation quality emphasizes the quality of deliberants' inputs (eg, engagement and evidence of compromise) during deliberative sessions. Fewer researchers have framed quality in terms of facilitator inputs, and these researchers tend to examine inputs that are consistent with generic group processes. The theory, process, and purpose of public deliberation, however, are distinct from those of focus groups or other group-based discussions and warrant a mechanism for measuring quality in terms of facilitator fidelity to the principles and processes of deliberative democracy. In our public deliberation on ethical conflicts in minor consent for biomedical HIV prevention research, we assessed facilitator fidelity to these principles and processes because we believe that such assessments serve as a component of a comprehensive evaluation of overall deliberation quality. We examined verbatim facilitator remarks in the deliberation transcripts and determined whether they aligned with the 6 principles of public deliberation: equal participation, respect for the opinions of others, adoption of a societal perspective, reasoned justification of ideas, expression of diverse opinions, and compromise or movement toward consensus. In this tutorial, we describe the development of a blueprint to guide researchers in assessing facilitator fidelity, share 3 templates that will assist them in the task, and describe the results of our assessment of facilitator fidelity in 1 of the 4 sites in which we conducted deliberations.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.091 | 0.226 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.004 | 0.003 |
| Science and technology studies | 0.003 | 0.002 |
| Scholarly communication | 0.004 | 0.006 |
| Open science | 0.004 | 0.007 |
| Research integrity | 0.003 | 0.003 |
| Insufficient payload (model declined to judge) | 0.010 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".