The Language of Supreme Court Briefs: A Large-Scale Quantitative Investigation
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
I. BACKGROUND AND INTRODUCTION In 2003, we initiated a long-term project to investigate empirically the language used in United States Supreme Court briefs. (1) The exploratory stage was open-ended, largely without any particular results initially sought or predicted. We wanted to collect and categorize as much linguistic data from Supreme Court briefs as possible, and analyze such data as thoroughly as we could--and let the results lead to possible topics for publication, rather than vice versa. Indeed, at times we hoped (admittedly with quite a bit of skepticism) that we might find statistically significant differences between the linguistic styles of winning and losing briefs, and be able to offer profitable advice for practitioners based on such information. But even without any unrealistic Holy Grail outcomes, we nonetheless were confident such a study would be able to provide useful advice to legal practitioners and educators, as well as possibly interesting outcomes for scholars of legal advocacy or linguistics. Our first publication, in the American Journal of Trial Advocacy, (2) was based on a less complete database, and was narrower in scope, because it focused on the language of only one short component of the brief, the question presented. Still, this earlier article did find interesting relationships between linguistic and other variables (time, party, and the like) in Supreme Court briefs, and concluded with advice for Court advocates. (3) The scope of the current article is more extensive. Our database consists of nearly every brief on the merits presented to the Court for the thirty-five years between 1969 and 2004. (4) We initially downloaded about 9,000 briefs, and then chopped them up for analysis into about a quarter of a million separate brief components such as Table of Contents, Table of Authorities, Summary of Argument, and the like. To clean up and analyze the briefs, eight original PERL software programs were written for this project. We decided to download every brief, rather than a smaller number based on an appropriate statistical sampling, for two reasons. For one, downloading every brief allowed us to sidestep any sampling concerns in the first place. But more importantly, although our database is comprehensive for our purposes, we were curious about how style might vary depending on a large number of legal issues, and of course even with a full set of briefs over thirty-five years, some legal issues appear rarely (or not at all). A. Other Empirical Studies of the Language of Legal Advocacy Our project is certainly not the first to use quantitative methods to investigate the language of written (or oral) legal advocacy. The first published work that applied computational linguistics to analyze the language of judicial briefs focused on the University of Michigan affirmative action litigation, as decided by the Supreme Court in Gratz v. Bollinger (5)and Grutter v. Bollinger. (6) Over a hundred amicus briefs had been filed in these companion cases, so the authors had a healthy corpus of advocacy language for analysis. Using programs that counted the appearance of key words in each brief, they were able to show that quantitative methods alone could successfully predict the policy positions that were being advocated; statistically significant differences were found in the language of amicus briefs supporting the respondent, as opposed to amicus briefs supporting the petitioner. (7) In other work, scholars have usefully polled large numbers of active judges to ascertain what stylistic factors in appellate briefs are most favored and disfavored by decisionmakers. (8) Empirical work has also found positive relationships between success and attorney qualifications in oral arguments before the Supreme Court of Canada. (9) Only recently has the United States Supreme Court allowed oral argument transcripts to be released that identify a Justice by name in recording questions they pose to advocates. …
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.007 | 0.005 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it