Methods for Classifying Errors on the Raven’s Standard Progressive Matrices Test - eScholarship

Methods for Classifying Errors on the Raven’s Standard Progressive Matrices Test Maithilee Kunda (mkunda@gatech.edu) 1 Isabelle Soulieres (soulieres.isabelle@uqam.ca) 2 Agata Rozga (agata@gatech.edu) 1 Ashok K. Goel (goel@cc.gatech.edu) 1 School of Interactive Computing, Georgia Tech, 85 Fifth Street NW, Atlanta, GA 30308 USA Departement de Psychologie, Universite du Quebec a Montreal C.P. 8888 succursale Centre-ville, Montreal (Quebec) H3C 3P8 Canada Abstract Although many psychometric tests, like Raven’s Progressive Matrices, are commonly evaluated according to total score, additional variables can lend insight into the underlying cognitive processes. We examine conceptual errors on the Raven’s Standard Progressive Matrices (SPM) test. We present a complete classification of error types on the SPM using a two-kind coding scheme, yielding ≥ 95% inter-rater reliability. We also examine how to extract error data from a computational model, and we present a method for measuring errors through systematic ablation to create a “population” of models whose performance can be examined as a group. We present a preliminary analysis of error patterns on the SPM from typically developing individuals, individuals diagnosed with autism, and a computational model called ASTI. We discuss what the error patterns suggest regarding cognition on the SPM and routes towards improving the ASTI model. Keywords: ablation experiments; computational modeling; error patterns; mental imagery; psychometrics; Raven’s Progressive Matrices; visual representations. Introduction Raven’s Progressive Matrices (RPM) is a widely used series of intelligence tests that consist of multiple choice visual analogy problems, as in Fig. 1. Each problem contains a matrix of geometric figures with one figure missing; the correct missing figure that completes the matrix pattern must be selected from a set of answer choices. Performance is generally measured in terms of overall score, i.e. number correct, which can then be used as an index into normative test data to determine an IQ score or percentile ranking for that individual. While total score is certainly an important variable, serving as a coarse measure of an individual’s overall ability, there are alternative dimensions of performance that may provide a finer-grained view of an individual’s cognitive processing: 1) Per-item accuracy, e.g. differential item functioning, takes into account potential variation even when individuals may obtain the same total score (Facon & Nuchadee, 2010; Lynn, Alik, & Irwing, 2004; Van Herwegen, Farran, and Annaz, 2011). 2) Reaction time can be used to understand the stages of processing in solving a single item (Bethell-Fox, Lohman, & Snow, 1984) or to compare performance across individuals or groups (Soulieres et al., 2009). 3) Patterns of errors—for a problem answered incorrectly, which of the given distracters is selected?—have been studied as a window into cognitive strategy (Bromley, 1953; Gunn & Jarrold, 2004; Miller & Raven, 1939; Van Herwegen, Farran, and Annaz, 2011; Vodegel Matzen et al., 1994). All of these dimensions represent measurable aspects of the “output” of a human cognitive system taking the RPM test. The “input” to such a system, in addition to the test itself, can be conceptualized as the set of cognitive functions drawn upon while solving the test. Unlike the output measures, it is difficult to directly measure cognitive functioning. Some studies have used eye-tracking as a measure of visual attention (Bethell-Fox, et al., 1984; Carpenter, Just, & Shell, 1990), and some have used verbal reporting protocols (Carpenter et al., 1990) though verbal report may bias the cognitive strategies used by participants (DeShon, Chan, & Weissbein, 1995). Another way to elucidate these invisible cognitive mechanisms is to construct computational models of various aspects of RPM problem solving and then inspect these models in relation to human behavioral data. Aspects of RPM (or RPM-like) problem solving that have been investigated using computational models include: Figure 1: Example of an RPM-like problem.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.