Is the Course Working? An Account of Our Development of an Instrument to Measure the Science Attitudes and Skills of Undergraduate Students Outside of Science Disciplines
Bibliographic record
Abstract
After a redesign of our school year structure, our science team developed an introduction to science course focused on teaching science to non-science majors early in their post-secondary studies. The goal of this course was not to prepare students for further pursuit of science degrees; instead, we wanted to equip them with the skills and attitudes necessary to understand the scientific world in which we live. Consequently, we wondered whether these skills and attitudes were being met throughout the course; was the course working? When searching the literature, we did not identify any instrument that simultaneously and concisely measured general science skills and attitudes. Given this gap and based on our desire to measure science skills and attitudes for non-science majors at our campus, this research team developed Augustana Interdisciplinary Scientific Literacy Evaluation (AISLE) in order to provide a measurement of students’ science skills and abilities in a general science course at the post-secondary level. However, as we would come to know, this process was not as simple as might seem. The purpose of this paper is to provide an account of the development and validation of the AISLE for those who wish to use the instrument or for others in the SoTL community looking to develop similar tools. We also offer an account of using the AISLE in our course to measure students’ science skill and attitude development. In the end, our STEM-based instructional team learned that what appeared to be straight forward assessment development, was, in fact, a far more involved and complicated process.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.012 | 0.010 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.003 | 0.003 |
| Scholarly communication | 0.004 | 0.003 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.004 |
| Insufficient payload (model declined to judge) | 0.005 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".