Robustness to 3D Object Transformations in Humans and Image-Based Deep Neural Networks
Bibliographic record
Abstract
Recent work at the intersection of psychology, neuroscience, and computer vision has advocated for the use of more realistic visual tasks in modeling human vision. Deep neural networks have become leading models of the primate visual system. However, their behavior under identity-preserving 3D object transformations, such as translation, scaling, and rotation, has not been thoroughly compared to humans. Here, we evaluate both humans and image-based deep neural networks, including vision-only and vision-language models trained with supervised, self-supervised, or weakly supervised objectives, on their ability to recognize objects undergoing such transformations. Humans (n=220) and models (n=169) were asked to categorize images of 3D objects, generated with a custom pipeline, into 16 object categories recognizable by both. Humans were time-limited to reduce reliance on recurrent processing. We find that both humans and models are robust to translation and scaling, but models struggle more with object rotation and are more sensitive to contextual changes. Humans and models agree on which in-depth object rotations are most challenging -- when humans struggle, models do too -- but humans are more robust and show more consistent category confusions with one another than with any model. By testing model families trained on different amounts of data and with different learning objectives, we show that data richness plays a substantial role in supporting robustness -- potentially more so than vision-language alignment. Our benchmark excludes models trained on video, multiview, or 3D data, but is in principle compatible with such models and may support their evaluation in future work. This study underscores the importance of using naturalistic visual tasks to model human object perception in complex, real-world scenarios, and introduces a benchmark - ORBIT (Object Recognition Benchmark for Invariance to Transformations) - for evaluating and developing computational models of human object recognition. Code and data for ORBIT are available at: https://github.com/haideraltahan/ORBIT.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.004 | 0.018 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.002 |
| Scholarly communication | 0.002 | 0.003 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".