Assessment of Training Effectiveness Adjusted for Learning (ATEAL) Part I: Method Development and Validation
Bibliographic record
Abstract
Training programs are a popular method, in industry globally, to increase awareness of desired concepts to employees and employers and play a critical part in changing or supporting performance improvements. The predominant method to assess the effectiveness of training programs is to have the participants answer Multiple Choice Question (MCQ) and True/False (T/F) questions after the training; however, the metrics typically used to report the outcome of such assessments have drawbacks that make it difficult for the trainer and organization to easily identify the concepts that need more focus and those that do not. This study introduces measures of the Assessment of Training Effectiveness Adjusted for Learning (ATEAL) method, which compensate the assessment scores for prior knowledge and negative training impact in quantifying the effectiveness of each concept taught. The results of this method are compared to the results of the most popular methods currently used. A simulation of various scenarios and the training effectiveness metrics that result from them is used to illustrate the sensitivity and limitation of each method. Results show that the proposed coefficients are more sensitive in detecting prior knowledge and negative training impact. Additionally, the proposed ATEAL method provides a quick and easy way to assess the effectiveness of the training concept based on the assessment results and provides a directional guide on the changes that need to be made to improve the training program for the participants. A companion paper expands the concepts using results from actual training sessions in multiple industries.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.160 | 0.309 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.004 |
| Bibliometrics | 0.003 | 0.003 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.003 | 0.002 |
| Open science | 0.002 | 0.003 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.007 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".