Tight Bounds for Lp Samplers, Finding Duplicates in Streams, and Related Problems
Bibliographic record
Abstract
In this paper, we present near-optimal space bounds for Lp-samplers. Given a stream of updates (additions and subtraction) to the coordinates of an underlying vector x \in R^n, a perfect Lp sampler outputs the i-th coordinate with probability |x_i|^p/||x||_p^p. In SODA 2010, Monemizadeh and Woodruff showed polylog space upper bounds for approximate Lp-samplers and demonstrated various applications of them. Very recently, Andoni, Krauthgamer and Onak improved the upper bounds and gave a O(ε^{-p} log^3 n) space εrelative error and constant failure rate Lp-sampler for p \in [1,2]. In this work, we give another such algorithm requiring only O(ε^{-p} log^2 n) space for p \in (1,2). For p \in (0,1), our space bound is O(ε^{-1} log^2 n), while for the $p=1$ case we have an O(log(1/ε)ε^{-1} log^2 n) space algorithm. We also give a O(log^2 n) bits zero relative error L0-sampler, improving the O(log^3 n) bits algorithm due to Frahling, Indyk and Sohler. As an application of our samplers, we give better upper bounds for the problem of finding duplicates in data streams. In case the length of the stream is longer than the alphabet size, L1 sampling gives us an O(log^2 n) space algorithm, thus improving the previous O(log^3 n) bound due to Gopalan and Radhakrishnan. In the second part of our work, we prove an Omega(log^2 n) lower bound for sampling from 0, \pm 1 vectors (in this special case, the parameter p is not relevant for Lp sampling). This matches the space of our sampling algorithms for constant ε> 0. We also prove tight space lower bounds for the finding duplicates and heavy hitters problems. We obtain these lower bounds using reductions from the communication complexity problem augmented indexing.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.016 | 0.111 |
| Meta-epidemiology (narrow) | 0.004 | 0.003 |
| Meta-epidemiology (broad) | 0.005 | 0.004 |
| Bibliometrics | 0.004 | 0.007 |
| Science and technology studies | 0.005 | 0.007 |
| Scholarly communication | 0.009 | 0.033 |
| Open science | 0.010 | 0.011 |
| Research integrity | 0.005 | 0.013 |
| Insufficient payload (model declined to judge) | 0.012 | 0.004 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".