Domino: SAIC's English Entity-Linking System.
Bibliographic record
Abstract
The Domino system was SAIC’s student-intern entry to the English Entity-Linking track of the 2012 TAC-KBP competition. This paper describes how Domino was developed using components from the CUNY-BLENDER system and discusses the features and rules that were added to Domino. It analyzes Domino’s performance, and suggests ways in which we plan to improve the system in the future. 1.Building the Domino Baseline System 1.1 Motivation and Constraints Entity linking is a central task that analysts in the intelligence community (IC) often perform. Analysts must try to determine, for example, whether a person who is referred to in an intercepted email is the same as a person who is reported in some news article to have engaged in some terrorist activity. SAIC, which supports IC analysts in many different ways, is interested in developing methods to help automate the entitylinking process. There are many similarities between the IC entity-linking task and the TAC-KBP EntityLinking track, which focuses on linking named entities in news articles or blog posts with Wikipedia articles. We --a group of students from the University of Maryland who spent the summer of 2012 at SAIC, and our supervisor at SAIC --therefore decided, in mid-June 2012, to enter the TAC-KBP Entity Linking competition. Because most of us --and in particular, the developers among us –were undergraduates with little experience in Natural Language Processing, and because we knew we had only two months to pull together a system, we decided that we would try to use existing resources as much as possible. Our aim was to get an existing entity-linking system and modify it in order to improve output results. We were especially interested in generalizing the entitylinking system so that it would be useful for more than just Wikipedia’s domain. SAIC’s customers will often be interested in people who keep a low profile and who would be unlikely to have an entry in Wikipedia. 1.2 Harnessing Existing Resources We were fortunate that the researchers who had developed CUNY-BLENDER, the CUNY’s entry to multiple TAC-KBP tracks (entity linking and slot filling) in 2010 [Chen et al., 2010] had made CUNY-BLENDER’s codebase available to anyone who wanted to use it. We decided to build our system on top of the CUNY-BLENDER pipeline. Originally, we had envisioned that DOMINO would be a superset of CUNY-BLENDER. We had hoped to quickly get CUNY-BLENDER running, establish a baseline, and then spend most of our time experimenting with new features which would enhance Domino’s performance. In fact, we needed to make significant adjustments and modifications to CUNY-BLENDER. As a result, while Domino is based on CUNY-BLENDER, it is neither a subset nor a superset of it. 1.3 Overview of CUNY-BLENDER The architecture for the CUNY-BLENDER pipeline is shown below. Figure 1: CUNY-BLENDER’S architecture The general procedure for entity linking is as follows:
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.004 | 0.014 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.006 | 0.004 |
| Science and technology studies | 0.002 | 0.001 |
| Scholarly communication | 0.005 | 0.010 |
| Open science | 0.003 | 0.006 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.055 | 0.048 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".