A Cognitive Model for Automatic Student Assessment: Classification of Errors in Engineering Dynamics
Bibliographic record
Abstract
Abstract Development of a Cognitive Model for Automatic Student Assessment: Classification of Student Errors in Engineering Statics and DynamicsThe taxonomy of human errors is critical for professions dealing with public safety such asengineering and healthcare. Once errors are identified, preventative and corrective measures canbe sought. From a pedagogical perspective, detection of error is also important. When studentsmake an error it is important to determine why the error was made and then begin the process ofcorrecting it by providing them with individualized feedback and assignments. For anexperienced educator, student assessment can be a fairly straightforward task. However, during atypical course, when a student is in the learning stage of development, assessment of thestudent's work is typically performed by a marker. This undergraduate or graduate student oftenhas teaching skills that are not fully developed and therefore may have difficulty properlyassessing the students' work. Guidelines or rules can be provided to the marker in the form of arubric but can be either too general or assignment specific. In addition, with decreasing educationbudgets and increasing popularity of distance learning in the form of massive open onlinecourses, automatic assessment methods are becoming increasingly important.This paper summarizes the errors that students made in two first year engineering final exams:engineering statics and dynamics. A cognitive model is then developed based on an errorclassification scheme rooted in Action Theory, which serves to classify errors as either mistakesor slips in logic. Simple rules are then developed into an algorithm for a computer to categorizeerrors based on a priori concepts such as: student experience level and priority level given on thecourse concept inventory as well as a posteriori concepts such as the frequency of the error. Theassessment algorithm is then “trained” using two final exams. The coefficients in the algorithmare optimized by comparing the results from experienced educators and that of the algorithm.The optimized algorithm is then tested on a third independent final exam and the resulting marksare compared. The cognitive model used is found to be general and easily extendible for use inother STEM courses. Results of the study show that the model performs well for probabilityregions where there is a clear distinction between slips and mistakes, but becomes moredependent on the model coefficients when the probability of the mistake, P(M)=0.5. Extension ofthis model is shown to be useful for automatically assessing a students' work as well as providingtimely and meaningful feedback.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".