Evaluating reasoning systems
Bibliographic record
Abstract
A review of the literature on evaluating reasoning systems reveals that it is a very broad area with wide variation in depth and breadth of research on metrics and tests.Consolidation is hampered by nonstandard terminology, differing methodologies, scattered application domains, unpublished algorithmic details, and the effects of domain content and context on the choice of metric and tests.The field of information metrology, which applies to reasoning as a kind of information processing, is still emerging from ad hoc experience in evaluating narrow kinds of information systems.This report begins to bring order to the area by categorizing reasoning systems according to their capabilities.The characteristics of each category can be used as a basis for evaluating and testing reasoning systems claiming to be in that category.Capabilities are analyzed along several dimensions, including representation languages, inference, and user and software interfaces.The report groups representation languages by their relation to first-order logic, and model-theoretic properties, such as soundness and completeness.Inference procedures are divided into deduction, induction, abduction, and analogical reasoning.Capabilities of user and software interfaces are described as they apply to reasoning systems.The report introduces information metrology, model theory, and inference to facilitate understanding of the reasoning categories presented.It concludes with recommendations for future work.Applying the results of this report to evaluation of specific reasoning systems requires further refinement of reasoning categories as needed to support development of test cases of interest to reasoning system users.Then development can proceed on generic test cases for reasoning categories.These are independent of the application, reasoning tool, and computational platform.Finally, based on the generic tests, development can start on specific test cases that are dependent on application, reasoning system tool, and computational platform.These steps can also be applied to user interface and software interface capabilities.This work can only be completed in partnership with users of reasoning systems and reasoning tool providers, to focus effort and reach the level of completeness necessary to put these metrics into practice.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.025 | 0.165 |
| Meta-epidemiology (narrow) | 0.002 | 0.000 |
| Meta-epidemiology (broad) | 0.002 | 0.001 |
| Bibliometrics | 0.007 | 0.004 |
| Science and technology studies | 0.001 | 0.004 |
| Scholarly communication | 0.009 | 0.010 |
| Open science | 0.002 | 0.005 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.010 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".