Bibliographic record
Abstract
Being named recipient of the 2004 ACM Sigmetrics Achievement Award has done several things to me. It brought me surprise that I would be singled out from the many people who have made significant and sustained contributions to the field of performance evaluation. It also brought me deep appreciation for all the students and colleagues with whom I have worked and come to know as friends over the years. Finally, it has caused me to ponder and reminisce about many of the research projects and consulting studies in which I have participated.In this talk, I will describe various systems I have used and studied, various applications of interest, and various models that I, and others, have used to try to gain insights into the performance of systems. Some lessons of possible future relevance that emerge from this retrospective look at a wide variety of projects are the following: Exact Answers Are Overrated -- While exact solutions of mathematical models are intellectually satisfying, they are often not needed in practice. Analytic Models Have a Role -- Analytic models can be used to obtain quick and inexpensive answers to performance questions in many situations where neither simulation nor experimentation are feasible. Assumptions Matter -- Subtle changes to the assumptions that underlie an analytic model can substantially alter the conclusions reached based on the model. After considering all the methods of analysis, simulation and experimentation, my recommendation for the very best means to attain substantially improved computer system performance is: Wait thirty years!
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.008 | 0.003 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.003 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".