A comparative performance analysis of the International Classification of Functioning, Disability and Health and the Item-Perspective Classification framework for classifying the content of patient reported outcome measures
Bibliographic record
Abstract
BACKGROUND: Standardized coding of the content presented in patient reported outcome measures can be achieved using classification frameworks, and the resulting data can be used for ascertaining content validity or comparative analyses. The International Classification of Functioning (ICF) is a framework with a detailed conceptual structure that has been successfully utilized for such purposes through established coding procedures. The Item Perspective Classification (IPC) framework is a newly developed relational coding system that classifies the respondent perspective and conceptual domains addressed in items. The purpose of this study was to compare and describe the performance of these two frameworks when used alone, and in conjunction, for the generation of data pertaining to the content of patient reported outcome measures. METHODS: Six health-related quality of life questionnaires with a total of 159 items were classified by two raters using the Item Perspective Classification framework in conjunction with the International Classification of Functioning. Framework performance indicators included: classification capacity (percent of items amenable to successful classification), coding efficiency (number of codes required to classify items), and content overlap detection (percent of items sharing identical classification codes with at least one other item). Inter-rater reliability of item coding was determined using Krippendorff's alpha. RESULTS: Classification capacity of the IPC framework was 97%, coding efficiency 26, and content overlap detection was 95%; whereas respective values for the ICF were 68%, 114, and 58%. When used in conjunction values were 63%, 129, and 30%. Krippendorff's alpha exceeded 0.97 for all 3 classification indices. CONCLUSION: Inter-rater agreement on classification data was excellent. The IPC framework provided a unique classification of the respondent's judgment during item response and classified more items using fewer categories, indicated greater content overlap across items and was able to describe the relationship between multiple concepts presented within the context of a single item. The ICF provided a unique classification of item content relating to aspects of disability and generated more detailed and precise descriptions. A combined approach provided a rich description (detailed codes) with each framework providing complementary information. The benefits of this approach in instrument development and content validation require further investigation.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.129 | 0.290 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.002 | 0.002 |
| Bibliometrics | 0.012 | 0.010 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.003 | 0.004 |
| Open science | 0.001 | 0.003 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".