Hence, Socrates is mortal: A Benchmark for Natural Language Syllogistic Reasoning

Syllogistic reasoning, a typical form of deductive reasoning, is a critical capability widely required in natural language understanding tasks, such as text entailment and question answering.To better facilitate research on syllogistic reasoning, we develop a benchmark called SYLLOBASE that differs from existing syllogistic datasets in three aspects: (1) Covering a complete taxonomy of syllogism reasoning patterns; (2) Containing both automatically and manually constructed samples; and (3) Involving both the generation and understanding tasks.We automatically construct 50k template-based syllogism samples by mining syllogism patterns from Wikidata and ConceptNet.To improve our dataset's naturalness and challenge, we apply GPT-3 to paraphrase the templatebased data and further manually rewrite 1,000 samples as the test set.State-of-the-art pretrained language models can achieve the best generation ROUGE-L of 38.72 by T5 and the best multi-choice accuracy of 72.77% by RoBERTa on SYLLOBASE, which indicates the great challenge of learning diverse syllogistic reasoning types on SYLLOBASE.Our datasets are released at https://github.com/ casually-PYlearner/SYLLOBASE.ConceptNet An open, multilingual knowledge graph (1) Triplet Extraction (Section 3.1) (2) Syllogism Construction (Section 3.2) (3) Paraphrasing (GPT-3) (Section 3.3) ① (human, capable of, mortal), (Socrates, is a, human) ② (meteoritics, subclass of, astronomy), (astronomy, subclass of, exact science) Premise 1: All human are mortal.Premise 1: Some astronomy are meteoritics.Premise 2: All Socrates are human.Premise 2: All astronomy are exact science.Conclusion: All Socrates are mortal.Conclusion: Some exact science are meteoritics Pattern: All 𝑚 are 𝑝, all 𝑠 are m ⟶ All 𝑠 are 𝑝.Pattern: Some 𝑚 are 𝑝, all 𝑚 are 𝑠 ⟶ Some 𝑠 are 𝑝.Premise 1: It is a fact that all human beings are mortal.Premise 2: Socrates was a classic Greek philosopher credited as one of the founders of Western philosophy.Conclusion: It is true that Socrates was mortal as well.Premise 1: Meteoritics, which is a specific type of astronomy, involves the study of meteors and meteorites.Premise 2: Astronomy is an exact science meaning precise observation and mathematical calculations are implemented.Conclusion: Some precise sciences are used to further expand upon knowledge of meteorites and meteors.Table 2: Examples of syllogisms from our test set.Categorical Syllogism Premise 1: Carbon dioxide is a chemical compound.Premise 2: Chemical compounds are considered pure substances.Conclusion: Pure substances include carbon dioxide.Hypothetical Syllogism Premise 1: When you make progress in your project, you may want to celebrate.Premise 2: Having a party is a good choice if you want to celebrate.Conclusion: You may want to have a party if you achieve great progress in your project.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.