Bibliographic record
Abstract
The CESART Evaluation Package was produced within the French national project CESART (Evaluation of terminology extraction tools), as part of the Technolangue programme funded by the French Ministry of Research and New Technologies (MRNT). The CESART project enabled to carry out a campaign for the evaluation of terminology extraction tools. This project is an extension of the evaluation campaign of terminology resource acquisition tools that was carried out for written corpora (ARC A3) within the AUPELF campaigns (Actions de recherche Concertées, 1996-1999). This package includes the material that was used for the CESART evaluation campaign. It includes resources, protocols, scoring tools, results of the campaign, etc., that were used or produced during the campaign. The aim of these evaluation packages is to enable external players to evaluate their own system. The campaign is distributed over two actions: 1)Term extraction for the building of a terminology reference which applications are the enrichment of the reference and the free indexing of documents.2)Extraction of semantic relations (synonymy) from a list of “focal” terms.The CESART evaluation package contains the following data and tools:Three domain-specific corpora in French were built: one medical corpus, one educational corpus, and one political corpus. The first two were used as test corpora, while the third one (political corpus) was used as a masking corpus. The corpora were encoded in UTF-8 and XML. They are available in two different versions, one for DOS and one for UNIX.1)The medical corpus consists of web pages extracted from Santé Canada (http://www.hc-sc.gc.ca/index_f.html). 2)The corpus in the educational field contains articles extracted from the SPIRAL magazine specialised in pedagogy and research in education. 3)The political corpus is composed of texts extracted from the Official Journal of the European Union. The table below gives some statistics on the corpora used for the evaluation:<table border="0" width="100%" cellspacing="0" cellpadding="2" class="infoBoxContents"><tr align=center><td>Corpus (specialised)</td><td>Medicine (test corpus)</td><td>Education (test corpus)</td><td>Politics (masking corpus)</td></tr><tr align=center><td align=left><strong>Number of documents</strong></td><td>7,514</td><td>149</td><td>1,477</td></tr><tr align=center><td align=left><strong>Number of segments</strong></td><td>255,161</td><td>12,109</td><td>9,024</td></tr><tr align=center><td align=left><strong>Number of words</strong></td><td>9,000,000</td><td>535,000</td><td>240,000</td></tr></table>Two reference lists were built from two terminology databases in a specialised domain. The list of medical terms, based on the terminology provided by the CISMeF team (www.chu-rouen.fr/terminologiecismef), is available from the IST/Inserm (http://mesh.inserm.fr/mesh). This list contains 22,861 entries. As for the educational domain, the reference list is based on the Motbis thesaurus (http://www.thesaurus.motbis.cndp.fr/site/) and consists of 36,081 entries.A description of the project is available at the following address:http://www.technolangue.net/article.php3?id_article=200 (in French language)
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.950 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; both teacher heads agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".