What Makes Sentences Semantically Related? A Textual Relatedness Dataset and Empirical Study
Bibliographic record
Abstract
What Makes Sentences Semantically Related? A Textual Relatedness Dataset and Empirical Study This repository contains data and code for the paper What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Study. We hope that this work will spur further research on understanding sentence--sentence relatedness, methods of sentence representation, measures of semantic relatedness, and their applications. Citing our work Please use the following BibTex entry to cite us if you use our dataset or any of the associated analyses: @inproceedings{abdalla2023makes, title={What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Study}, author={Abdalla, Mohamed and Vishnubhotla, Krishnapriya and Mohammad, Saif M.}, year={2023}, address = {Dubrovnik, Croatia}, publisher = "Association for Computational Linguistics", booktitle = "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume" } Dataset Description The dataset consists of 5500 English sentence pairs that are scored and ranked on a relatedness scale ranging from 0 (least related) to 1 (most related). Why Semantic Relatedness? Closeness of meaning can be of two kinds: semantic relatedness and semantic similarity. Two sentences are considered semantically similar when they have a paraphrasal or entailment relation, whereas relatedness accounts for all of the commonalities that can exist between two sentences. Semantic relatedness is central to textual coherence and narrative structure. Automatically determining semantic relatedness has many applications such as question answering, plagiarism detection, text generation (say in personal assistants and chat bots), and summarization. Prior NLP work has focused on semantic similarity (a small subset of semantic relatedness), largely because of a dearth of datasets. In this paper, we present the first manually annotated dataset of sentence--sentence semantic relatedness. It includes fine-grained scores of relatedness from 0 (least related) to 1 (most related) for 5,500 English sentence pairs. The sentences are taken from diverse sources and thus also have diverse sentence structures, varying amounts of lexical overlap, and varying formality. Comparative Annotations and Best-Worst Scaling Most existing sentence-sentence similarity datasets were annotated, one item at a time, using coarse rating labels such as integer values between 1 and 5 @ representing coarse degrees of closeness. It is well documented that such approaches suffer from inter- and intra-annotator inconsistency, scale region bias, and issues arising due to the fixed granularity. The relatedness scores for our dataset were, instead, obtained using a comparative annotation schema. In comparative annotations, two (or more) items are presented together and the annotator has to determine which is greater with respect to the metric of interest. Specifically, we use Best-Worst Scaling, a comparative annotation method}, which has been shown to produce reliable scores with fewer annotations in other NLP tasks. We use scripts from https://saifmohammad.com/WebPages/BestWorst.html to obtain relatedness scores from our annotations. Loading the Dataset - The sentence pairs, and associated scores, are in the file sem_text_rel_ranked.csv in the root directory. The CSV file can be read using: python import pandas as pd str = pd.read_csv('sem_text_rel_ranked.csv') row = str.loc[0] sent1, sent2 = row['Text'].split("\n") score = row['Score'] - Relevant columns: - Text: Sentence pair, separated by the newline character. - Score: The semantic relatedness score between 0 and 1. - Additionally: - the SourceID column indicates the source dataset from which the sentence pair was drawn (see Table 2 of our paper) - The SubsetID column indicates the sampling strategy used for the source dataset - and the PairID is a unique identifier for each pair that also indicates its Source and Subset. Raw Annotations from Amazon Mechanical Turk - The `mturk_data/` subdirectory provides the raw MTurk annotations obtained with our comparative annotation setup. - Each row of `mturk_data/bws_annotations.csv` consists of four sentence pairs along with human annotations for the most related (column `BestItem`) and the least related (column `WorstItem`) pair. - File `mturk_data/id2sents.csv` pairs each sentence pair with the corresponding SourceID, SubsetID, and PairID that indicates the source dataset (see Table 2 of our paper). - See file `mturk_data/task_intructions.txt` for the instructions provided to annotators for our task. Datasheet for STR-2022 The datasheet for our dataset is in the document `STR2022-datastatement.pdf` in the root folder of this repository. Ethics Statement Any dataset of semantic relatedness entails several ethical considerations. We talk about this in Section 8 of our paper. Creators - Mohamed Abdalla (University of Toronto) - Krishnapriya Vishnubhotla (University of Toronto) - Saif M. Mohammad (National Research Council Canada) Contact: msa@cs.toronto.edu, vkpriya@cs.toronto.edu, saif.mohammad@nrc-cnrc.gc.ca
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.016 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.004 | 0.003 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.001 | 0.002 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.006 | 0.004 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".