Peer Portal: Quality enhancement in thesis writing using self-managed peer review on a mass scale
Bibliographic record
Abstract
This paper describes a specially developed online peer-review system, the Peer Portal, and the first results of its use for quality enhancement of bachelor’s and master’s thesis manuscripts. The peer-review system is completely student driven and therefore saves time for supervisors and creates a direct interaction between students without interference from supervisors. The purpose is to improve thesis manuscript quality, and thereby use supervisor time more efficiently, since peers review basic aspects of the manuscripts and give constructive suggestions for improvements. The process was initiated in 2012, and, in total, 260 peer reviews were completed between 1st January and 15th May, 2012. All peer reviews for this period have been analyzed with the help of content analysis. The purpose of analysis is to assess the quality of the students work. The results are categorized in four groups: 1) excellent (18.1%), 2) good (22.7%), 3) fragmented (18.5%), and 4) poor (40.7%). The overall result shows that almost 40% of the students produced excellent or good peer reviews and almost as many produced poor peer reviews. The result shows that the quality varies considerably. Explanations of these quality variations need further study. However, alternative hypotheses followed by some strategic suggestions are discussed in this study. Finally, a way forward in terms of improving peer reviews is outlined: 1) development of a peer wizard system and 2) rating of received peer reviews based on the quality categories created in this study. A Peer Portal version 2.0 is suggested, which will eliminate the fragmented and poor quality peer reviews, but still keep this review system student driven and ensure autonomous learning.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.030 | 0.065 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.004 | 0.003 |
| Science and technology studies | 0.002 | 0.001 |
| Scholarly communication | 0.004 | 0.003 |
| Open science | 0.002 | 0.006 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.026 | 0.016 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".