Variations in relevance judgments and the measurement of retrieval effectiveness
Why is this work in the frame?
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
No Canadian affiliation. An affiliation-only frame — the usual design — would never have seen this work. It is one of the works that make the case for inverting the frame.
Machine scores (provisional)
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
- Teacher spread
- 0.230 · how far apart the two teachers sit on this one work
- Validation status
score_only:v0-immature-baseline· verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it
Abstract
No abstract. This is not a gap in this database — OpenAlex has none either. 23.3% of the frame is in this state, and the screen finds HALF as much metaresearch here, so the absence is a measured bias rather than a missing field.
The record
- Venue
- Information Processing & Management
- Topic
- Information Retrieval and Search Behavior
- Field
- Computer Science
- Canadian institutions
- —
- Funders
- National Institute of Standards and TechnologyUniversity of Waterloo
- Keywords
- Relevance (law)Information retrievalComputer scienceNISTSet (abstract data type)Test (biology)Reliability (semiconductor)Ranking (information retrieval)Document retrievalNatural language processing
- Has abstract in OpenAlex
- no