Capturing 'what works' in complex process evaluation research: the use of calendar instruments
Bibliographic record
Abstract
Calendar or timeline techniques have developed in parallel and are used in life course research, and health behaviour and treatment studies. Both types of research seek to reconstruct histories or events in order to understand phenomena. Unsurprisingly a research strategy that seeks to represent events from memory is fraught with recall error thereby influencing consistency, completeness, and accuracy of data. Strategies can be employed to improve data quality so informants can more accurately access long term memory. One such strategy involves producing a graphical timeframe against which historical information can be represented. This is said to stimulate memory facilitating accuracy of recall and fidelity of data. There are minor variations in the application of calendar techniques, unsurprising given the different methodological heritage, nevertheless there are common characteristics. These include: graphical display of the dimension of time, use of one or more thematic axis (representing the data domains) and event or landmark cues that temporally bound the research. \nThe Department of Health (England) in 2008 funded a series of public health initiatives in nine ‘Healthy Towns’. These initiatives were targeted on facilitating healthier lifestyles in local populations and importantly learning from projects about “what works”. One “Healthy Town” – Healthy Halifax – funded ten embedded project streams all designed to encourage adoption of health lifestyles by the population living in four wards with poorest health outcomes. The challenge presented to the local evaluation team was capturing which, if any, of the projects made a difference to health lifestyles of local populations. Calendar technique were incorporated in research design to accurately represent the life history of each project and capture the antecedents, attributes and consequences of project delivery that might illuminate ‘what works’. This presentation will offer a critical appraisal of the utility of calendar technique as a methodological approach for capturing process evaluation.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.498 | 0.658 |
| Meta-epidemiology (narrow) | 0.003 | 0.003 |
| Meta-epidemiology (broad) | 0.003 | 0.004 |
| Bibliometrics | 0.014 | 0.019 |
| Science and technology studies | 0.005 | 0.024 |
| Scholarly communication | 0.027 | 0.026 |
| Open science | 0.004 | 0.011 |
| Research integrity | 0.004 | 0.006 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".