Assessment of Thinking Levels in Students' Answers
Notice bibliographique
Résumé
Authors' Note: This research was supported in part by a grant to J.J. Pear from the Social Sciences and Humanities Research Council of Canada. D.E. Crone-Todd was supported by a fellowship from the Social Sciences and Humanities Research Council of Canada. The authors gratefully acknowledge Ms. Sabrina Berry's assistance with this research project. Abstract Having first developed a method, based on Bloom's taxonomy (1956), for assessing the thinking levels required by study questions in computer-mediated courses (Crone-Todd, Pear & Read, 2000), we developed a method for assessing the levels at which students answer the questions. Reliability measures between two independent assessment groups were high (i.e., > 80%). The assessment procedure can serve diagnostic and research purposes in determining how to enable students to increase their thinking levels in post-secondary courses. ********** Assessment of Thinking Levels in Students' Answers One of the most important goals of post-secondary education is to promote the use of critical, or higher-order, thinking skills. To this end, educators must find ways to identify, teach, and encourage the use of these skills in their courses. One of the largest hurdles in this process is developing a precise operational definition, or set of definitions, for what is meant by thinking. There is, however, a lack of consensus concerning the definition of this construct. For example, higher-order thinking may be reasoned (Newman, 1991a, b), comparing elements in terms of sameness (Carnine, 1991), application of concepts or principles (Hohn, Gallagher, & Byrne, 1990; Semb & Spencer, 1976), making discipline-related judgments that are effective (Paul & Heaslip, 1995), or argumentation that is systematic and active (Mayer & Goodchild, 1990). It seemed to us that all of these definitions include various components of what is considered thinking, or thinking that requires combining elements in different ways than those provided in a textbook or other course materials. A set of definitions that appears to incorporate all of the definitions above is Bloom's (1956) taxonomy of objectives in the cognitive domain. The taxonomy, which incorporates behavioral definitions of cognitive processes, has been used in a variety of educational settings. Despite its popularity, however, those using the taxonomy for research purposes have encountered problems with its reliability (e.g., Calder, 1983; Gierl, 1997; Kottke & Schuster, 1990; Roberts, 1976; Seddon, 1978; Seddon, Chokotho, & Merritt, 1981). Recently, Crone-Todd, Pear, & Read (2000) used a modified version of Bloom's (1956) taxonomy in the cognitive domain to identify the thinking levels required by study questions in a computer-aided personalized system of instruction (CAPSI) course. The purpose of the study was to begin the development of a more reliable measure of higher-order thinking in CAPSI-taught using guided study questions (e.g., Pear & Crone-Todd, 1999; http://home.cc.umanitoba.ca/~capsi) than had been previously reported in the literature. Following the taxonomy, the thinking levels were: (a) Level 1 - Knowledge, (b) Level 2 - Comprehension, (c) Level 3 - Application, (d) Level 4 - Analysis, (e) Level 5 - Synthesis, and (f) Level 6 - Evaluation. Briefly, in the modified taxonomy, Level 1 corresponds to rote learning, Level 2 involves the ability to state an answer in one's own words, Level 3 is the ability to apply what one has learned to new problems or situations, Level 4 is the ability to break down concepts into smaller components, Level 5 is the ability to combine concepts to create new knowledge, and Level 6 is the ability to rationally argue or discuss a position with regard to a given topic. Levels 1 and 2 may be considered lower-order thinking (because they do not involve generation of new concepts or knowledge), while levels 3 through 6 may be consider higher-order thinking (see Crone-Todd et. …
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction machine sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Le volet Gemma est une étiquette directe du modèle pour chaque travail de la base, lue sur la notice réduite au titre. Le volet Codex est un classifieur appris des 10 348 étiquettes directes de Codex et calibré sur les taux pondérés de l'échantillon; les champs sans appui suffisant ne portent aucun appel Codex. Le mode candidate est l'union des deux volets; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont pas des étiquettes humaines.
Scores du classifieur distillé par catégorie (deux têtes)
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,002 | 0,018 |
| Méta-épidémiologie (sens strict) | 0,000 | 0,000 |
| Méta-épidémiologie (sens large) | 0,000 | 0,000 |
| Bibliométrie | 0,002 | 0,001 |
| Études des sciences et des technologies | 0,000 | 0,000 |
| Communication savante | 0,001 | 0,000 |
| Science ouverte | 0,000 | 0,001 |
| Intégrité de la recherche | 0,000 | 0,001 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,009 | 0,002 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; un appel candidat d’une seule source (Gemma direct ou Codex distillé), pas un consensus.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».