Concordance between blinded independent central review committee and physician-assessed responses: Analyses based on a real-world external control arm in relapsed/refractory multiple myeloma using International Myeloma Working Group data.
Bibliographic record
Abstract
e19521 Background: Blindedindependent central review (BICR) committees are often established for oncology trials to assessbestoverall response (BOR), while real-world (RW) studies typically rely on physician-assessed responses. To emulate clinical trial rigor, a BICR committee was implemented in the R5458-ONC-21101 RW data-derived external control arm study (NCT05673967) to evaluate BOR using International Myeloma Working Group (IMWG) criteria. A pre-specified concordance analysis compared BOR assessments between the BICR committee and treating site physicians. Methods: RW data were obtained from chart reviews at 15 participating International Myeloma Foundation IMWG sites. The BICR committee, comprised of three multiple myeloma experts, evaluated BOR through consensus voting. Per protocol, the committee reviewed case report forms consisting of data on serum and urine myeloma protein and free light chain levels; and, where available, bone marrow plasma cell percentage, presence of plasmacytomas, and bone lesion changes. Concordance between BICR- and physician-assessed BOR was evaluated for objective response rate defined as ≥partial response (PR), ≥very good partial response (VGPR), ≥complete response (CR), and stringent complete response (sCR), with non-evaluable responses excluded from the analysis. Cohen’s kappa (κ) was used to measure agreement, with values >0.6 indicating strong concordance. 1 Results: Of 279 evaluable lines of therapy from 203 patients (Table), the BOR of ≥PR was 45.2% by BICR assessment and 42.3% by treating physicians (κ=0.622). Concordance improved for BOR of ≥VGPR (22.2% by BICR vs 24.7% by treating physicians; κ=0.731) but was low for BOR of ≥CR (3.9% vs. 10.8%; κ=0.508) and sCR (1.1% vs. 4.7%; κ=0.364). Conclusions: BICR- and physician-assessed responses showed strong concordance for BOR of ≥PR and ≥VGPR. However, as expected, lower concordance for ≥CR and sCR was observed, as evaluation of CR per IMWG criteria requires a bone marrow biopsy, which is rarely performed in clinical practice. While the reliability of CR and sCR in RW data is less certain, these results suggest that BOR of ≥VGPR or ≥PR may be reliably ascertained in academic settings based on treating physician evaluations. 1. Landis JR, Koch GG. Biometrics 1977;33(1):159–74. Clinical trial information: NCT05673967 . Agreement of BICR- and physician-assessed responses. BICR assessed Physician assessed Observed agreement, % Cohen’s kappa* Number of evaluable lines of therapy 279 279 - - ≥PR, n (%) 126 (45.2) 118 (42.3) 81.4 0.622 ≥VGPR, n (%) 62 (22.2) 69 (24.7) 90.3 0.731 ≥CR, n (%) 11 (3.9) 30 (10.8) 93.2 0.508 sCR, n (%) 3 (1.1) 13 (4.7) 96.4 0.364 *A kappa value >0.6 indicates strong concordance. 1
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.309 | 0.326 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.003 |
| Bibliometrics | 0.005 | 0.004 |
| Science and technology studies | 0.002 | 0.003 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.002 | 0.004 |
| Research integrity | 0.002 | 0.001 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".