Disease Severity Estimates—Effects of Rater Accuracy and Assessment Methods for Comparing Treatments
Bibliographic record
Abstract
Assessment of disease severity is required for several purposes in plant pathology; most often, the estimates are made visually. It is established that visual estimates can be inaccurate and unreliable. The ramifications of biased or imprecise estimates by raters have not been fully explored using empirical data, partly because of the logistical difficulties involved in different raters assessing the same leaves for which actual disease has been measured in a replicated experiment with multiple treatments. In this study, nearest percent estimates (NPEs) of Septoria leaf blotch (SLB) on leaves of winter wheat from nontreated and fungicide-treated plots were assessed in both 2006 and 2007 by four raters and compared with assumed actual values measured using image analysis. Lin’s concordance correlation (LCC, ρc) was used to assess agreement between the two approaches. NPEs were converted to Horsfall-Barratt (HB) midpoints and were compared with actual values. The estimates of SLB severity from fungicide-treated and nontreated plots were analyzed using generalized linear mixed modeling to ascertain effects of rater using both the NPE and HB values. Rater 1 showed good accuracy (ρc = 0.986 to 0.999), while raters 3 and 4 were less accurate (ρc = 0.205 to 0.936). Conversion to the HB scale had little effect on bias but reduced numerically both precision and accuracy for most raters on most assessment dates (precision, r = −0.001 to −0.132; and accuracy, ρc = −0.003 to −0.468). Interrater reliability was also reduced slightly by conversion of estimates to HB midpoint values. Estimates of mean SLB severity were significantly different between image analysis and raters 2, 3, and 4, and there were frequently significant differences among raters (F = 151 to 1,260, P = 0.001 to P < 0.0001). Only on 26 June 2007 did conversion to the HB scale change the means separation ranking of rater estimates. Nonetheless, image analysis and all raters were able to differentiate control and treated-plot treatments (F = 116 to 1,952, P = 0.002 to P < 0.0001, depending on date and rater). Conversion of NPEs to the HB scale tended to reduce F values slightly (2006: NPEs, F = 116 to 276, P = 0.002 to 0.0005; and, for the HB-converted values, F = 101 to 270, P = 0.002 to 0.0005; 2007: NPEs, F = 164 to 1,952, P = 0.001 to P < 0.0001; and, for HB-converted values, F = 126 to 1,633, P = 0.002 to P < 0.0001). The results reaffirm the need for accurate and reliable disease assessment to minimize over- or underestimates compared with actual disease, and the data we present support the view that, where multiple raters are deployed, they should be assigned in a manner to reduce any potential effect of rater differences on the analysis.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.238 | 0.417 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.003 | 0.004 |
| Bibliometrics | 0.003 | 0.003 |
| Science and technology studies | 0.002 | 0.003 |
| Scholarly communication | 0.003 | 0.004 |
| Open science | 0.003 | 0.004 |
| Research integrity | 0.003 | 0.004 |
| Insufficient payload (model declined to judge) | 0.001 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".