Comparing artificial neural network models with varied objectives to probe the role of sensory representation in primate visual memorability
Bibliographic record
Abstract
Imagine a typical experience of scrolling through numerous photos on social media. While most images blend into obscurity, certain ones, like a playful kitten tangled in yarn, linger in our memories. This selective retention, known as image memorability, raises intriguing questions about its neural basis, particularly how the sensory representations in the cortex facilitate this behavior. In this study, we investigate this process, leveraging recent strides in computer vision through artificial neural network (ANN) models. These models, designed to mimic the ventral visual pathway in primates, offer a vast hypotheses space for the brain function that are critical for determining the memorability of an image. We compare two distinct types of ANNs: models geared toward basic object categorization (e.g., ResNet-50, AlexNet, GoogLeNet) and those tailored to predict image memorability (e.g., MemNet, ViTMem) on a set of 200 images (20 images from 10 distinct object categories) from the MS-COCO dataset. Consistent with previous results, we observed that both these model classes can predict which images humans find most memorable. However, our results show that they produce significantly different internal representations as assessed by representational similarity analysis (comparing representations across model classes in architecture-specific and non-specific manner). In addition, here, we used neural data recorded across the macaque inferior temporal (IT) in 6 monkeys to address two primary questions. 1) Can both model classes accurately predict variance in the neural data? 2) Do they predict distinct variances in the responses of the individual neurons? Surprisingly, our results show that memorability models (MemNet) predict significantly more variance in the neural data than object categorization models with matched architecture (AlexNet) – emphasizing a) the need to further probe these model classes as encoding models of the ventral stream, b) provide a new normative framework to think about the evolution of sensory representations.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.011 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.002 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".