Psychophysical Evaluation of Saliency Algorithms
Bibliographic record
Abstract
Significant effort has been spent evaluating the performance of saliency algorithms at predicting human fixations in natural images. However, many other aspects of human visual attention have received relatively little focus in the saliency literature but have been richly characterized by psychophysical investigations. In order to make use of this data, Bruce et al. (2015) have recommended the development of an axiomatic set of model constraints grounded in this body of psychophysical knowledge. We aim to provide a step towards this goal by linking human visual search response time to saliency algorithm output. Duncan and Humphreys (1989) theorized that subject response time in visual search tasks is correlated with similarity between search items (with search time increasing as targets become more similar to distractors). This result fits well with the widely held notion that saliency is largely driven by stimulus uniqueness, but has not been explicitly tested against the performance of saliency algorithms. To do so systematically, we need a well-characterized human performance curve for a given set of visual search stimuli. Arun (2012) produced a performance curve for oriented bars which shows the relationship between human response time and target-distractor orientation differences over the range 7-60°. Here we use Arun's stimuli as input to a range of current saliency algorithms and discover that performance falls into three broad categories: algorithms which cannot consistently find the target, those which consistently find the target but have no differentiated performace with target-distractor difference, and those which are able to deliver a human performance-like curve. In this way we provide a new performance criterion that is more closely aligned with the use of saliency as an early selection mechanism. Future work will look at the full set of Wolfe's (1998) features which can elicit efficient search for singleton targets. Meeting abstract presented at VSS 2016
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.026 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.003 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".