Tracking eye movements to uncover the nature of visual-linguistic interaction in static and dynamic scenes
Bibliographic record
Abstract
These studies examined the role of sentence and visual context in the access to verb-complement information, using a new eye tracker and change blindness paradigm. Participants' eye movements were monitored as they viewed pictures of objects (Experiment 1) or dynamic scenes (Experiment 2), and listened to related sentences. In Experiment 1, two sets of sentences were contrasted, a highly constraining causative construction in which there was a close conceptual relation between the verb and its direct object (e.g., "The woman burned the candle ") and a neutral construction with a transitive perception verb (e.g., "The woman admired the candle "). Starting at three different points within the presentation of the verb (onset, middle, offset) and noun (onset, offset, offset+200 ms), participants' task was to look and indicate whether the target objects mentioned in the sentences (e.g., " candle ") were present in the visual displays. Results indicate that semantic information extracted at the verb can be used to constrain the domain of reference in the scene and in some cases predict the referent of the grammatical complement of the verb, depending on tasks demands, conceptual consolidation, of the scene, and the presence of competitor objects. In Experiment 2, two different classes of verbs were contrasted, denominal and non-denominal verbs, which either implicitly (e.g., "The woman will iron the shirt ") or explicitly (e.g., "The woman will chop the vegetables with the knife ") named the instrument nouns. These movies were edited, unbeknownst to the participants, so that the real referents of the verb's grammatical object (e.g., " shirt/vegetables ") or instrument (e.g., " iron/knife ") gradually dissolved. Participants' task was to provide a detailed description of each scene, and later perform a recognition task. Although results indicate that information extracted from the sentence helped identify, describe and remember scene details, visual context seems to take precedence over linguistic input properties in guiding eye movements. In conclusion, the processor appears to incrementally integrate all available knowledge, linguistic and non-linguistic, with the aim to rapidly interpret the linguistic description of what we see in the world and how we may interact with it.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.000 | 0.001 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".