Understanding open source software peer review: review processes, parameters and statistical models, and underlying behaviours and mechanisms
Bibliographic record
Abstract
Peer review is seen as an important quality assurance mechanism in both industrial development and the open source software (OSS) community. The techniques for performing inspections have been well studied in industry; in OSS development, peer review practices are less well understood. In contrast to industry, where reviews are typically assigned to specific individuals, in OSS, changes are broadcast to hundreds of potentially interested stakeholders. What is surprising is that this approach works very well, despite concerns that reviews may be ignored, or that discussions will deadlock because too many uninformed stakeholders are involved. In this work we use a multi-case study methodology to develop a theory of OSS peer review. There are three research stages. In the first stage, we examine the policies of 25 OSS projects to understand the review processes used on successful OSS projects. We also select six projects for further analysis: Apache, Subversion, Linux, FreeBSD, KDE, and Gnome. In the second stage, using archival records from the six projects, we construct a series of metrics that produces measures similar to those used in traditional inspection experiments. We measure the frequency of review, the size and complexity of the contribution under review, the level of participation during review, the experience and expertise of the individuals involved in the review, the review interval, and number of issues discussed during review. We create statistical models of the review efficiency, review interval, and effectiveness, the issues discussed during review, to determine which measures have the largest impact on review efficacy. In the third stage, we use grounded theory to analyze 500 instances of peer review and interview ten core developers across the six projects. This approach allows us to understand why developers decide to perform reviews, what happens when reviews are ignored, how developers interact during a review, what happens when too many stakeholders are involved during review, and the effect of project size on the review techniques. Our findings provide insights into the simple, community-wide mechanisms and behaviours that developers use to effectively manage large quantities of reviews and other development discussions. The primary contribution of this work is a theory of OSS peer review. We find that OSS reviews can be described as (1) early, frequent reviews (2) of small, independent, complete contributions (3) that, despite being asynchronously broadcast to a large group of stakeholders, are reviewed by a small group of self-selected experts (4) resulting in an efficient and effective peer review technique.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.002 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".