Utility of Technologist Editing of Polysomnography Scoring Performed by a Validated Automatic System
Bibliographic record
Abstract
RATIONALE: Automatic scoring of polysomnography records offers many advantages, but excessive editing time seriously limits its use. OBJECTIVES: To identify reasons for excessive editing time, and the clinical utility of such editing, and to develop an approach to optimize the editing process. METHODS: Forty-two polysomnograms scored manually were scored months later by an automatic system (Michele Sleep Scoring). Results were edited by the technologist who scored them initially. Editing actions and time were documented. An Editing Helper algorithm was developed on the basis of these results, and its effectiveness was tested in 60 new records. MEASUREMENTS AND MAIN RESULTS: Technologists performed 253 ± 110 actions, consuming 54.5 ± 26.3 minutes, per file. Of the edits, 33% were either subsequently reversed or not considered in the clinical summary. The electroencephalography pattern in 67% of epochs changed from awake to non-REM sleep, and vice versa, represented neither stable wakefulness nor sleep so that assigning a precise stage was arbitrary. Many opposing changes occurred. Ultimately the impact of editing on summary results was limited. In the second set, the Editing Helper algorithm reduced editing time from 59 ± 26 to 6 ± 7 minutes. Average (±SD) intraclass correlation coefficients for 15 reported variables were 0.77 ± 0.14 for manual versus unedited automatic, 0.89 ± 0.09 for manual versus fully edited automatic, and 0.87 ± 0.08 for manual versus automatic edited according to the Editing Helper's suggestions only, and there was no difference between the last two average intraclass correlation coefficients. CONCLUSIONS: Editing time does not reflect unreliable scoring. Comprehensive editing of a well-validated automatic scoring system is highly inefficient. Editing can be substantially optimized.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.019 | 0.104 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.000 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".