Frequency Analysis of Terminology on Engineering Examinations
Bibliographic record
Abstract
Abstract Frequency Analysis of Terminology on Engineering ExaminationsThere have always been differences between instructor expectations of what students “shouldknow” and the actual background experience that students have entering an engineeringprogram. The divergence between this assumed knowledge and the actual knowledge base maybe increasing as the student population diversifies. Previous work has noted the impact ofdiversity in this regard. The issue is not just wide differences in preparation in basic math, orscience, or communication ability, but diversity in the cultural background of students. Whilewe frequently laud diversity we have not always followed this up by supporting inclusivity in ourclassrooms and finding ways to bridge cultural differences that may exist. Specifically, when wecontextualize technical material to situate an engineering problem in a real-world scenario,students are subject to a test of their background experience – so, instead of clarifying a technicalconcept, the context may make the concept more inaccessible. This may also compromise theinclusivity of the learning environment, causing students to doubt their suitability for studyingengineering.Current engineering students bring with them a wealth of knowledge that is as diverse as thebackgrounds and cultures they represent. However, students may be unfairly disadvantagedduring examinations, for example, if they are expected to understand terminology that assumes aspecific set of a priori experience; instead of assessing whether the student understands thetechnical material, assessments may inadvertently test vocabulary. This is especially importantfor examinations because the closely-supervised setting prohibits assistance. Yet, pedagogicallywe would prefer to assess understanding of concepts in authentic situations, not in the abstract.And a number of effective methods, such as model-eliciting activities (MEA’s), are based onauthentic contextualization.This represents an instance where learner characteristics are misaligned with the expectations ofthe learning environment, and there has been little research in this particular area of engineeringeducation. The goal of the current study is to evaluate the frequency of this type ofmisalignment. As raw data we are using an exam bank that contains final examinations collectedover a number of years for all engineering courses at a large engineering school. A frequencyanalysis of the words and terms used on the exams has been carried out, excluding coursespecific technical terminology. At this point in the study we are assuming that infrequently usedwords and terms are typically less familiar to students. This is an assumption that will be testedin a subsequent phase of the study.The results of the frequency analysis are analyzed with respect to: 1. The types of words and terms that are used on exams. 2. The types of courses where contextualization appears to be used most often. 3. The types of contexts that appear most frequently.The results will be discussed within the theoretical framework of learner characteristics andinteraction with the learning environment. In particular, we will examine these results withreference to the literature on Universal Instructional Design (UID), and current work on learnerdiversity.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".