The One-Legged High Jumper and the Perils of Prediction: Predicting Success for Students Based on Their Background Is More Accurate in the Aggregate Than in Individual Situations, Where It Should Never Be Applied
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Pretty much everyone in education knows student outcomes are deeply affected by family background--that poverty, parental education, family interaction, and ethnicity are strongly correlated with how well students do and with whether they graduate from high school or go on to postsecondary education. We also know that students who fall behind often have a hard time catching up. That knowledge has helped design more effective school programs that can reduce achievement gaps. Yet, while it is important and powerful, knowledge about these correlations has a danger connected with it as well. That danger is the belief that we can predict an individual student's future based on his or her past. Lots of evidence shows that belief is not only wrong but can lead to lower expectations and self-fulfilling prophecies. Here's the problem. Educators will often say that they can look at children at age 8, or 10, or 12, and know which ones will be successful. In the aggregate, for large groups, such predictions have quite a high degree of validity, but they're much less useful when it comes to knowing the futures of individual students who will often surprise us--and themselves. Many studies have found that a large number of students defy negative expectations based on their backgrounds. A Canadian study (OECD, 2010) showed that nearly 40% of 15-year-olds with very low reading skills were in postsecondary education six years later, and another showed that students with poor literacy skills at age 15 made the largest gains in literacy by age 24 (OECD, 2012). A U.S. longitudinal study of 4,000 students (Hernandez, 2011) showed that poor reading at 3rd grade did predict failure to graduate from high school, but nonetheless more than 75% of poor readers in 3rd grade did eventually graduate, including about 70% of struggling readers from poor families. A thorough review of studies of high school graduation found that even complex prediction models with many variables did not have enough accuracy to make them useful in working with individual students (Gleason & Dynarski, 2002). In every study of this kind, a significant number of students who seem to have everything against them end up having good results. And if you don't trust the statistics, just think of examples in your own life and circle. All of us know people who overcame difficult life circumstances. When I speak to groups of educators, I often ask if there is someone in the audience who, at age 15 or 16, would have been regarded by her or his teachers as highly unlikely to succeed. I have yet to meet a group of educators where there was not someone who told me that in high school he or she was regarded by the school as having few prospects but who nonetheless went on to college and is now a successful teacher or principal. One of my favorite real-life examples of defying the odds is Arnie Boldt. Boldt lost a leg in a farm accident when he was three years old, yet became a world-class high jumper (you can find videos of him on YouTube) whose best jumps were well over 6 feet! And now, quite a bit later in life--he became an educator, by the way--he is a world-class paralympic cyclist! Looking at photos and videos of his accomplishments, one can't help but wonder who looked at a kid with one leg and saw a star high jumper? More to the point, how many other students are not getting the chance to develop their talents because we see the missing leg and not the talented person? Boldt himself is entirely modest about his amazing accomplishments. But, in the volunteer work he continues to do with young people who have lost limbs, he notes that the problem is usually not the kids' willingness to try, it's the fear and caution of the adults that gets in the way of their progress. In telling these stories I'm not suggesting that background factors don't matter; a huge amount of evidence tells us that they do. …
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it