A Cognitive Model for Automatic Student Assessment: Classification of Errors in Engineering Dynamics
Notice bibliographique
Résumé
Abstract Development of a Cognitive Model for Automatic Student Assessment: Classification of Student Errors in Engineering Statics and DynamicsThe taxonomy of human errors is critical for professions dealing with public safety such asengineering and healthcare. Once errors are identified, preventative and corrective measures canbe sought. From a pedagogical perspective, detection of error is also important. When studentsmake an error it is important to determine why the error was made and then begin the process ofcorrecting it by providing them with individualized feedback and assignments. For anexperienced educator, student assessment can be a fairly straightforward task. However, during atypical course, when a student is in the learning stage of development, assessment of thestudent's work is typically performed by a marker. This undergraduate or graduate student oftenhas teaching skills that are not fully developed and therefore may have difficulty properlyassessing the students' work. Guidelines or rules can be provided to the marker in the form of arubric but can be either too general or assignment specific. In addition, with decreasing educationbudgets and increasing popularity of distance learning in the form of massive open onlinecourses, automatic assessment methods are becoming increasingly important.This paper summarizes the errors that students made in two first year engineering final exams:engineering statics and dynamics. A cognitive model is then developed based on an errorclassification scheme rooted in Action Theory, which serves to classify errors as either mistakesor slips in logic. Simple rules are then developed into an algorithm for a computer to categorizeerrors based on a priori concepts such as: student experience level and priority level given on thecourse concept inventory as well as a posteriori concepts such as the frequency of the error. Theassessment algorithm is then “trained” using two final exams. The coefficients in the algorithmare optimized by comparing the results from experienced educators and that of the algorithm.The optimized algorithm is then tested on a third independent final exam and the resulting marksare compared. The cognitive model used is found to be general and easily extendible for use inother STEM courses. Results of the study show that the model performs well for probabilityregions where there is a clear distinction between slips and mistakes, but becomes moredependent on the model coefficients when the probability of the mistake, P(M)=0.5. Extension ofthis model is shown to be useful for automatically assessing a students' work as well as providingtimely and meaningful feedback.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction distillée sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Apprise à partir de 10 348 étiquettes directes de Codex et de 10 348 étiquettes directes de Gemma. Le mode candidate est l'union des têtes enseignantes seuillées; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont ni des étiquettes humaines ni des étiquettes directes de modèles de pointe.
Scores Codex et Gemma par catégorie
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,001 | 0,000 |
| Méta-épidémiologie (sens strict) | 0,000 | 0,000 |
| Méta-épidémiologie (sens large) | 0,000 | 0,000 |
| Bibliométrie | 0,000 | 0,000 |
| Études des sciences et des technologies | 0,000 | 0,000 |
| Communication savante | 0,000 | 0,000 |
| Science ouverte | 0,000 | 0,000 |
| Intégrité de la recherche | 0,000 | 0,000 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,000 | 0,000 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; un appel candidat d’une seule tête enseignante, pas un consensus.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».