Methods for Classifying Errors on the Raven’s Standard Progressive Matrices Test - eScholarship
Bibliographic record
Abstract
Methods for Classifying Errors on the Raven’s Standard Progressive Matrices Test Maithilee Kunda (mkunda@gatech.edu) 1 Isabelle Soulieres (soulieres.isabelle@uqam.ca) 2 Agata Rozga (agata@gatech.edu) 1 Ashok K. Goel (goel@cc.gatech.edu) 1 School of Interactive Computing, Georgia Tech, 85 Fifth Street NW, Atlanta, GA 30308 USA Departement de Psychologie, Universite du Quebec a Montreal C.P. 8888 succursale Centre-ville, Montreal (Quebec) H3C 3P8 Canada Abstract Although many psychometric tests, like Raven’s Progressive Matrices, are commonly evaluated according to total score, additional variables can lend insight into the underlying cognitive processes. We examine conceptual errors on the Raven’s Standard Progressive Matrices (SPM) test. We present a complete classification of error types on the SPM using a two-kind coding scheme, yielding ≥ 95% inter-rater reliability. We also examine how to extract error data from a computational model, and we present a method for measuring errors through systematic ablation to create a “population” of models whose performance can be examined as a group. We present a preliminary analysis of error patterns on the SPM from typically developing individuals, individuals diagnosed with autism, and a computational model called ASTI. We discuss what the error patterns suggest regarding cognition on the SPM and routes towards improving the ASTI model. Keywords: ablation experiments; computational modeling; error patterns; mental imagery; psychometrics; Raven’s Progressive Matrices; visual representations. Introduction Raven’s Progressive Matrices (RPM) is a widely used series of intelligence tests that consist of multiple choice visual analogy problems, as in Fig. 1. Each problem contains a matrix of geometric figures with one figure missing; the correct missing figure that completes the matrix pattern must be selected from a set of answer choices. Performance is generally measured in terms of overall score, i.e. number correct, which can then be used as an index into normative test data to determine an IQ score or percentile ranking for that individual. While total score is certainly an important variable, serving as a coarse measure of an individual’s overall ability, there are alternative dimensions of performance that may provide a finer-grained view of an individual’s cognitive processing: 1) Per-item accuracy, e.g. differential item functioning, takes into account potential variation even when individuals may obtain the same total score (Facon & Nuchadee, 2010; Lynn, Alik, & Irwing, 2004; Van Herwegen, Farran, and Annaz, 2011). 2) Reaction time can be used to understand the stages of processing in solving a single item (Bethell-Fox, Lohman, & Snow, 1984) or to compare performance across individuals or groups (Soulieres et al., 2009). 3) Patterns of errors—for a problem answered incorrectly, which of the given distracters is selected?—have been studied as a window into cognitive strategy (Bromley, 1953; Gunn & Jarrold, 2004; Miller & Raven, 1939; Van Herwegen, Farran, and Annaz, 2011; Vodegel Matzen et al., 1994). All of these dimensions represent measurable aspects of the “output” of a human cognitive system taking the RPM test. The “input” to such a system, in addition to the test itself, can be conceptualized as the set of cognitive functions drawn upon while solving the test. Unlike the output measures, it is difficult to directly measure cognitive functioning. Some studies have used eye-tracking as a measure of visual attention (Bethell-Fox, et al., 1984; Carpenter, Just, & Shell, 1990), and some have used verbal reporting protocols (Carpenter et al., 1990) though verbal report may bias the cognitive strategies used by participants (DeShon, Chan, & Weissbein, 1995). Another way to elucidate these invisible cognitive mechanisms is to construct computational models of various aspects of RPM problem solving and then inspect these models in relation to human behavioral data. Aspects of RPM (or RPM-like) problem solving that have been investigated using computational models include: Figure 1: Example of an RPM-like problem.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.022 | 0.101 |
| Meta-epidemiology (narrow) | 0.003 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.010 | 0.004 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.003 | 0.003 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.011 | 0.005 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".