Identifying Misconceptions using Structural Assessment of Knowledge
Bibliographic record
Abstract
Identifying Misconceptions using Structural Assessment of Knowledge David L. Trumpower (david.trumpower@uottawa.ca) Harold Sharara (hshar044@uottawa.ca) University of Ottawa, Faculty of Education, 145 Jean-Jacques-Lussier Street Ottawa, ON K1N 6N5 Canada Keywords: knowledge organization; diagnostic assessment; Pathfinder; problem solving. Introduction Domain expertise requires not only an abundance of knowledge, but also well organized knowledge. Knowledge organization has been measured using a technique known as structural assessment of knowledge (Goldsmith, Johnson, & Acton, 1991) in which ratings of concept relatedness are transformed via scaling algorithm into a network representation. The quality of a network is determined by some quantitative measure of its overall similarity to a referent network. Although overall similarity has been shown to be a valid measure of domain knowledge (e.g., Goldsmith, et al., 1991), it is not particularly diagnostic in nature. This can be realized by noting that two networks may have the same overall similarity to a referent network although they might differ with respect to the specific links that they share with that referent. In this study, we assess the absence of specific links in structural knowledge representations (rather than an overall similarity measure) in an attempt to diagnose misconceptions, as indicated by performance on different problem types, in a computer programming domain. Pointer and the concepts Position, Increment, and Assign were said to possess the ‘Pointer schema’. PFnets that contained links between the concept Go-To and the concepts Step and If-Then were said to possess the ‘Go-To schema’ (see Figure 1). Of the 35 participants, 8 possessed the Pointer schema, whereas 12 possessed the Go-To schema. Only 3 participants possessed both schemas. Participants who possessed the Pointer schema solved more type P problems successfully than those who did not possess the Pointer schema, t(24.64)=2.81, p=.01. However, there was no difference in the number of other type problems solved by those who did and did not possess the Pointer schema, p>.05. Likewise, participants who possessed the Go-To schema solved more type G problems successfully than those who did not possess the Go-To schema, t(32.58)=2.69, p=.01, but there was no difference in the number of other type problems solved by those who did and did not possess the Go-To schema, p>.05. These results demonstrate that the absence of specific links in PFnets can be used to diagnose different misconceptions. This more fine grained assessment of structural knowledge representations may be useful for formative evaluations and focusing of instruction. Method Thirty-five undergraduate psychology students who had no prior computer programming experience were allowed 15 minutes to study a simple, custom designed computer programming language. The language was modeled after Pascal but was limited in scope, consisting of just 12 key concepts. Participants then rated the relatedness of all pairwise combinations of the 12 concepts on a 5-point scale (1=Not at all related, 5=Very related). Finally, participants attempted to solve a set of computer programming problems. Five of the problems required knowledge of the relationships among the concepts Position, Pointer, Assign, and Increment for successful solution (type P problems), whereas three problems required knowledge of the relationships among the concepts If-Then, Go-To, and Step (type G problems). Position Pointer List Increment Letter Ordered Switch Assign Instruction Step Go-To If-Then Figure 1. Expert’s PFnet with the Pointer schema and the Go-To schema highlighted in italics and bold, respectively. References Results & Discussion Participants’ relatedness ratings were transformed into network representations (PFnets) using the Pathfinder scaling algorithm (Schvaneveldt, 1990). PFnets were then analyzed for the presence of specific subsets of links, or schemas. PFnets that contained links between the concept Goldsmith, T. E., Johnson, P. J., & Acton, W. H. (1991). Assessing structural knowledge. Journal of Educational Psychology, 83, 88-96. Schvaneveldt, R. W. (1990). Pathfinder associative networks: Studies in knowledge organization. Norwood, NJ: Ablex.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".