LO30: Using a Massive Online Needs Assessment (MONA) to develop a Free Open Access Medical education (FOAM) curriculum
Bibliographic record
Abstract
Introduction/Innovation Concept: The boom in online educational resources for medical education over the past decade has changed how physicians learn and keep up to date with new literature. While nearly all emergency medicine residents use online resources, few of these resources were designed to target knowledge gaps. Novel methods are required to identify learning needs to allow the targeted development of learner-centered curricula. Methods: A multidisciplinary team attempted to determine the feasibility of conducting a Massive Online Needs Assessment (MONA) to assess the perceived and unperceived educational needs in thrombosis and bleeding. An open, online survey was launched via Google Forms and disseminated using the online educational resource CanadiEM.org and social media platforms Twitter and Facebook with the goal of reaching participants of the Free Open Access Medical education (FOAM) community. Curriculum, Tool, or Material: The survey was designed to identify knowledge gaps and contained demographic, free text, and multiple choice questions. It took individuals approximately 30 minutes to complete and was incentivized with entry into a draw for one of four $250 Amazon Gift cards. Feasibility was defined a priori as 150 responses from at least 4 specialties in 4 or more countries. This sample was deemed the minimum number required to identify knowledge gaps (defined as <50% correct answers). The survey was open from September 20 to December 10, 2016. We received 198 complete responses from 20 countries. Respondents included staff physicians (n=109), residents (n=46), medical students (n=29), nurses (n=8), paramedics (n=4), a pharmacist (n=1) and a physician assistant (n=1). The survey entry page hosted on CanadiEM.org received page views from 866 unique IP addresses. As such, a conservative approximation of the completion rate per unique viewer was 22% (198/866). Conclusion: It is feasible to use a MONA to collect data on the perceived and unperceived needs of an online community. Such needs assessments could be used to make online resources more learner-centered.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.015 | 0.028 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.001 | 0.002 |
| Open science | 0.001 | 0.004 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.012 | 0.004 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".