Judging by Numbers: How Will Judicial Analytics Impact the Justice System and Its Stakeholders?
Bibliographic record
Abstract
In 2019, the French government passed an unprecedented law that banned the public from analyzing information in reported court decisions to draw insights about the judicial behaviour of individual judges. The penalty for breaking this law is steep: violators face up to five years in prison. In our view, a French-style ban is not normatively defensible in Canada given our protection of freedom of expression and our strong open courts principle. The public should be able to analyze information that is in the public domain. We do believe, however, that Canada – like France – faces important questions about how to respond to the fast-growing field of judicial analytics. Although studying judges is not new, judicial analytics tools allow for much faster and more powerful analysis of large amounts of information. Judicial analytics tools for public use already exist but, for reasons explained in the article, such tools are likely to become even more powerful and readily accessible in the near-to-medium future, resulting in unprecedented public insight into judges and the work of judging. We term this phenomenon “mainstreamed judicial analytics.” It is this future world of mainstreamed judicial analytics that is the focus of our article. What happens in a world where technology allows us to instantaneously draw up a detailed profile of a judge’s past behaviour with a click of a smartphone button? What happens when we have a plethora of “stats” on how judges react to particular types of litigants, lawyers, legal arguments or even individual words? What happens when we can pull up reports on how a judge’s behaviour may be impacted by the day of the week, time of day or even the weather? Motivated by these and related questions, this article provides an analysis of the future of judicial analytics, its likely impacts, and potential responses to the rise of this technology in Canada. We conclude that the key potential benefit of mainstreamed judicial analytics is significantly increased transparency into the work of judging. Such transparency could provide an opportunity for the public to better critique and more effectively operate within the justice system. Also, judges could use information produced by judicial analytics tools to reflect on and improve upon their practices, where needed. Meaningful transparency, however, is not a guaranteed output. There will remain some practical complications to producing high-quality information even with “mainstreamed” tools. We also identify potential risks resulting from increased surveillance of judges, including the potential for unwanted strategic behaviour and negative impacts on health and well-being. Finally, we note that lawyers and judges will need to become familiar with this technology in order to competently perform their jobs.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.024 | 0.136 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.004 | 0.004 |
| Science and technology studies | 0.005 | 0.010 |
| Scholarly communication | 0.025 | 0.028 |
| Open science | 0.002 | 0.005 |
| Research integrity | 0.003 | 0.005 |
| Insufficient payload (model declined to judge) | 0.011 | 0.004 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".