Psychophysical Evaluation of Saliency Algorithms
Bibliographic record
Abstract
Significant effort has been spent evaluating the performance of saliency algorithms at predicting human fixations in natural images. However, many other aspects of human visual attention have received relatively little focus in the saliency literature but have been richly characterized by psychophysical investigations. In order to make use of this data, Bruce et al. (2015) have recommended the development of an axiomatic set of model constraints grounded in this body of psychophysical knowledge. We aim to provide a step towards this goal by linking human visual search response time to saliency algorithm output. Duncan and Humphreys (1989) theorized that subject response time in visual search tasks is correlated with similarity between search items (with search time increasing as targets become more similar to distractors). This result fits well with the widely held notion that saliency is largely driven by stimulus uniqueness, but has not been explicitly tested against the performance of saliency algorithms. To do so systematically, we need a well-characterized human performance curve for a given set of visual search stimuli. Arun (2012) produced a performance curve for oriented bars which shows the relationship between human response time and target-distractor orientation differences over the range 7-60°. Here we use Arun's stimuli as input to a range of current saliency algorithms and discover that performance falls into three broad categories: algorithms which cannot consistently find the target, those which consistently find the target but have no differentiated performace with target-distractor difference, and those which are able to deliver a human performance-like curve. In this way we provide a new performance criterion that is more closely aligned with the use of saliency as an early selection mechanism. Future work will look at the full set of Wolfe's (1998) features which can elicit efficient search for singleton targets. Meeting abstract presented at VSS 2016
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".