Assessing the multilevel validity of program-level inferences based on aggregate student perceptions about their general learning
Bibliographic record
Abstract
Aggregate survey results are commonly used by universities in Canada to compare effective educational practices across program majors within a university and between equivalent majors across campuses. Despite this recurrent practice, many researchers neglect to examine the multilevel validity of inferences made from program-level responses. This study illustrates the importance of determining the multilevel validity of program-level inferences prior to making conclusions based on survey data. Survey responses regarding student perceptions about their general learning outcomes and ratings about the learning environment were collected from the National Survey of Student Engagement (NSSE) and the Undergraduate Experience Survey (UES). The analytic procedures used in this study included two-level exploratory multilevel factor analyses (MFA) and three statistical approaches to determine the appropriateness of aggregation: analysis of variance (ANOVA), the within and between analysis (WABA), and the unconditional multilevel model. Multilevel regression models were applied to survey data to examine the relationships of program-level characteristics with perceived student learning outcomes. The results led to four conclusions regarding the use of student survey results aggregated to the program level. First, results from the MFA revealed that the multilevel structure of items regarding perceived learning were consistent across the student and program levels for most samples, but the multilevel structure of items regarding the learning environment was not supported at the program level. Second, results from the ANOVA and unconditional multilevel models indicated that aggregation to the program level for perceived learning was statistically appropriate for three out of the four study samples; however, WABA results indicated that aggregation to a level lower than the program major was more suitable. Aggregation to the program level was not supported for any the learning environment scales across all three procedures. Third, aggregation was variable dependent as demonstrated by lower levels of within-program agreement on ratings of the learning environment, but larger levels of agreement with perceived learning outcomes. Finally, student-level perceptions about learning were partially influenced by student- and program-level characteristics; however, program means were not highly reliable and results did not support making program comparisons. Implications for educational research and recommendations for further research were discussed.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".