Source Code for the 'Corpus of Decisions: International Court of Justice' (CD-ICJ-Source)
Bibliographic record
Abstract
Overview This code in the R Programming Language downloads and processes the full set of decisions and appended opinions rendered by the International Court of Justice (ICJ) as published on its website into a rich and structured human- and machine-readable data set. It is the basis of the Corpus of Decisions: International Court of Justice (CD-ICJ). All data sets created with this script will always be hosted permanently open access and freely available at Zenodo, the scientific repository of CERN. Each version is uniquely identified with a persistent Digitial Object Identifier (DOI), the Version DOI. The newest version of the data set will always available via the link of the Concept DOI: https://doi.org/10.5281/zenodo.3826444 Citation A peer-reviewed academic paper describing the construction and relevance of the data set entitled 'Introducing Twin Corpora of Decisions for the International Court of Justice (ICJ) and the Permanent Court of International Justice (PCIJ)' was published open access in the Journal of Empirical Legal Studies (JELS). It is also available in print at JELS 2022, Vol. 19, No. 2, pp. 491-524. If you use the data set for academic work, please cite both the JELS paper and the precise version of the data set you used for your analysis. New in Version 2023-10-22 Full recompilation of data set Scope extended up to case number 190: Aerial Incident of 8 January 2020 (Canada, Sweden, Ukraine and United Kingdom v. Islamic Republic of Iran) Add fix for lowercase components in URL basenames Updated Python toolchain Align docker config with Debian as host system Updates The CD-ICJ will be updated two times per year, ideally every six months. In case of serious errors an update will be provided at the earliest opportunity and a highlighted advisory issued on the Zenodo page of the current version. Minor errors will be documented in the GitHub issue tracker and fixed with the next scheduled release. The CD-ICJ is versioned according to the day the data was acquired from the website of the Court, in the ISO format YYYY-MM-DD. Its initial release version was 2021-11-23. Notifications regarding new and updated data sets will be published on my academic website at www.seanfobbe.com or via Mastodon at @seanfobbe@fediscience.org Functionality This code will produce 21 ZIP archives: 2 archives of CSV files containing the full machine-readable data set (English/French) 2 archives of CSV files containing the full machine-readable metadata (English/French) 2 archives of TXT files containing all machine-readable texts with a reduced set of metadata encoded in the filenames (English/French) 2 archives of PDF files containing all human-readable texts with enhanced OCR (English/French) 2 archives of PDF files containing all human-readable majority opinions with enhanced OCR (English/French) 2 archives of PDF files of documents dated 2004 and earlier containing monolingual documents with enhanced OCR (English/French) 2 archives of PDF files as originally published by the ICJ (English/French) 2 archives of TXT files containing text as generated by Tesseract for documents dated 2004 or earlier (English/French) 2 archives of TXT files containing extracted text from the original documents (English/French) 1 archive PDF files that were unlabelled on the website (intended for replication and review only) 1 archive of analysis data and diagrams 1 archive containing all source files The integrity and veracity of each ZIP archive is documented with cryptographically secure hash signatures (SHA2-256 and SHA3-512). Hashes are stored in a separate CSV file created during the data set compilation process. System Requirements Docker Docker Compose 25 GB disk space on hard drive Parallelization will automatically be customized to your machine by detecting the maximum number of cores A full run of this script takes approximately 11 hours on a machine with a Ryzen 3700X CPU using 16 threads, 64 GB DDR4 RAM and a fast SSD Instructions Step 1: Prepare Folder Copy the full source code to an empty folder, for example by executing: $ git clone https://github.com/seanfobbe/cd-icj Always use a dedicated and empty (!) folder for compiling the data set. The scripts will automatically delete all PDF, TXT and many other file types in its working directory to ensure a clean run. Step 2: Create Docker Image The Dockerfile contains automated instructions to create a full operation system with all necessary dependencies. To create the image from the Dockerfile, please execute: $ bash docker-build-image.sh Step 3: Compile Dataset If you have previously compiled the data set, whether successfuly or not, you can delete all output and temporary files by executing: $ Rscript delete_all_data.R You can compile the full data set by executing: $ bash docker-run-project.sh Results The data set and all associated files are now saved in your working directory. Academic Publications (Fobbe) Website — www.seanfobbe.com Open Data — zenodo.org/communities/sean-fobbe-data Code Repository — zenodo.org/communities/sean-fobbe-code Regular Publications — zenodo.org/communities/sean-fobbe-publications Contact Did you discover any errors? Do you have suggestions on how to improve the data set? You can either post these to the Issue Tracker on GitHub or write me an e-mail at fobbe-data@posteo.de
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.032 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.002 | 0.002 |
| Bibliometrics | 0.006 | 0.007 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.005 | 0.003 |
| Open science | 0.003 | 0.004 |
| Research integrity | 0.001 | 0.003 |
| Insufficient payload (model declined to judge) | 0.499 | 0.381 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".