Relation Extraction with Synthetic Explanations and Neural Network
Bibliographic record
Abstract
The state-of-the-art for Relation Extraction, defined as the detection of existing relations between a pair of entities in a sentence, relies on neural networks that require a large number of training examples to perform well. To address that cost, Distant Supervision has become the preferred choice for collecting labeled sentences. However, Distant Supervision has many limitations and often introduces noise into the training set. Recent work has shown an alternative way of training neural methods for relation extraction, namely to provide a small number of annotated sentences and explanations for why those sentences express the relation. Training classifiers with this approach results in accuracy comparable to Distant Supervision, but requires humans to annotate the sentences and provide the explanations. In this paper, we show a way to generate synthetic explanations from a small number of relational trigger words, for each relation, whose resulting explanations achieve comparable accuracy to human produced ones. We validate the method on five relation extraction tasks with different entity types (person-person, person-location, etc.). Furthermore, experiments on two public datasets demonstrate the effectiveness of our generated synthetic explanations, with 6% improvement in accuracy on relation extraction and 19% improvement in F1-score on generating labeled training sentences compared to the next best methods.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.009 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".