Rarity and the problem of measuring diversity
Bibliographic record
Abstract
Quantifying the diversity of microbial communities is a question of central importance from both a fundamental and an applied point of view. The estimation of microbial diversity has received renewed attention with the advent of large-scale metagenomic studies. Despite the remarkable sample sizes, metagenomic data sets are far from being an exhaustive census of the community on a fine taxonomic level. Hence, we must still ask what the diversity observed in a sample tells us about the diversity of the community being sampled.To answer this question, we study the link between sample composition and community structure using artificial and real communities. By evaluating the sampling properties of an unconstrained set of hypothetical community structures, we identify diversity measures that are possible to estimate and those that cannot be estimated without further assumptions. We analyze nine metagenomic data sets from a wide range of environments, and show that our findings for artificial communities are relevant for real communities.First, we argue that it is impossible to estimate the number of microbial taxa present in a community. The reason for this is simple: the sample data cannot rule out the possibility that the community contains a very large number of extremely rare taxa. We show how lack of information about rare taxa can lead to unbounded uncertainty about the number of taxa present. Second, we argue that it is also impossible to estimate the number of taxa in a relative sense. We illustrate this by applying Chao's estimator to our artificial communities: they are ranked incorrectly in the presence of a potentially large number of rare taxa. Finally, we extend our analysis to generalized diversities, including Shannon's and Simpon's, and show that the Simpson's diversity has the best estimation properties.Our claims about the impossibility of estimating the number of microbial taxa seem to contrast with previous studies. We argue that these studies rely on additional assumptions. By assuming a particular form of the taxa-abundance distribution, an estimate of the number of taxa can be obtained. However, these estimates are necessarily based on unverified extrapolation of the abundances from observed taxa to unobserved rare taxa. The possible presence of a large number of rare taxa invalidates this type of estimate.Chao's estimator is not explicitly based on a particular abundance distribution. Rather, this estimate can be interpreted as giving the lower bound of the estimates that would be obtained from a variety of distributional assumptions. As a result, Chao's estimator yields a possibly huge underestimation of the true diversity, a property that is often overlooked. However, in light of our findings, this lower bound estimate is the only robust statement that can be made about the number of taxa.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.034 | 0.177 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.003 | 0.001 |
| Bibliometrics | 0.010 | 0.007 |
| Science and technology studies | 0.002 | 0.018 |
| Scholarly communication | 0.006 | 0.011 |
| Open science | 0.004 | 0.009 |
| Research integrity | 0.004 | 0.005 |
| Insufficient payload (model declined to judge) | 0.001 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".