Instance-Optimality in the Noisy Value-and Comparison-Model
Bibliographic record
Abstract
Motivated by crowdsourced computation, peergrading, and recommendation systems, Braverman, Mao and Weinberg [7] studied the query and round complexity of fundamental problems such as finding the maximum (max), finding all elements above a certain value (threshold-ν) or computing the top−k elements (Top-k) in a noisy environment. For example, consider the task of selecting papers for a conference. This task is challenging due to the crowdsourcing nature of peer reviews: the results of reviews are noisy and it is necessary to parallelize the review process as much as possible. We study the noisy value model and the noisy comparison model: In the noisy value model, a reviewer is asked to evaluate a single element: “What is the value of paper i?” (e.g., accept). In the noisy comparison model (introduced in the seminal work of Feige, Peleg, Raghavan and Upfal [17]) a reviewer is asked to do a pairwise comparison: “Is paper i better than paper j?” In this paper, we introduce new lower bound techniques for these classic problems. In comparison to previous work, our lower bounds are much more fine-grained: they focus on the interplay between round and query complexity and the dependency on the output size. In the setting of conference papers, this translates into a trade-off between number of reviews per paper and the number of review rounds necessary in order find the best 100 papers for the conference. We complement these results with simple algorithms which show that our lower bounds are almost tight. We then go beyond the worst-case and address the question of the importance of knowledge of the instance by providing, for a large range of parameters, instance-optimal algorithms with respect to the query complexity. We complement these results by showing that for some family of instances, no instance-optimal algorithm can exist.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.016 | 0.076 |
| Meta-epidemiology (narrow) | 0.003 | 0.002 |
| Meta-epidemiology (broad) | 0.007 | 0.004 |
| Bibliometrics | 0.002 | 0.004 |
| Science and technology studies | 0.002 | 0.005 |
| Scholarly communication | 0.007 | 0.016 |
| Open science | 0.008 | 0.007 |
| Research integrity | 0.006 | 0.011 |
| Insufficient payload (model declined to judge) | 0.013 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".