Instance-Optimality in the Noisy Value-and Comparison-Model
Bibliographic record
Abstract
Motivated by crowdsourced computation, peergrading, and recommendation systems, Braverman, Mao and Weinberg [7] studied the query and round complexity of fundamental problems such as finding the maximum (max), finding all elements above a certain value (threshold-ν) or computing the top−k elements (Top-k) in a noisy environment. For example, consider the task of selecting papers for a conference. This task is challenging due to the crowdsourcing nature of peer reviews: the results of reviews are noisy and it is necessary to parallelize the review process as much as possible. We study the noisy value model and the noisy comparison model: In the noisy value model, a reviewer is asked to evaluate a single element: “What is the value of paper i?” (e.g., accept). In the noisy comparison model (introduced in the seminal work of Feige, Peleg, Raghavan and Upfal [17]) a reviewer is asked to do a pairwise comparison: “Is paper i better than paper j?” In this paper, we introduce new lower bound techniques for these classic problems. In comparison to previous work, our lower bounds are much more fine-grained: they focus on the interplay between round and query complexity and the dependency on the output size. In the setting of conference papers, this translates into a trade-off between number of reviews per paper and the number of review rounds necessary in order find the best 100 papers for the conference. We complement these results with simple algorithms which show that our lower bounds are almost tight. We then go beyond the worst-case and address the question of the importance of knowledge of the instance by providing, for a large range of parameters, instance-optimal algorithms with respect to the query complexity. We complement these results by showing that for some family of instances, no instance-optimal algorithm can exist.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".