CAVE : Detecting and Explaining Commonsense Anomalies in Visual Environments

CAVE: Commonsense Anomalies in Visual Environment 🏠 Project Page📄 Paper (EMNLP 2025)💻 Code Dataset Details Dataset Description CAVE is the first benchmark of real-world visual anomalies for evaluating Vision-Language Models (VLMs). It is curated from images captured in real-life settings (photographs and screenshots taken by individuals), sourced from Reddit. The benchmark is grounded in cognitive science literature on how humans detect and resolve anomalies. Each image is annotated with rich, multi-task annotations that support three open-ended tasks (anomaly description, explanation, and justification), one visual grounding task (anomaly localization via bounding boxes), and classification along four dimensions (anomaly category, severity, surprisal, and complexity) that characterize the anomaly. CAVE reveals that state-of-the-art VLMs struggle substantially with visual anomaly perception and commonsense reasoning: the best model (GPT-4o) achieves only ~57% F1-score on anomaly detection even with advanced prompting strategies. Curated by: Rishika Bhagwatkar, Syrielle Montariol, Angelika Romanou, Beatriz Borges, Irina Rish, Antoine Bosselut Affiliations: EPFL, MILA Language: English License: CC-BY-4.0 Published at: EMNLP 2025 (Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing) Dataset Sources Project Page: https://smontariol.github.io/cave-visual-anomalies/ Paper: CAVE: Detecting and Explaining Commonsense Anomalies in Visual Environments Contact: rishika.bhagwatkar@mila.quebec, syrielle.montariol@epfl.ch Uses Intended Uses CAVE is designed to evaluate VLMs on their ability to: Detect real-world commonsense anomalies in images (anomaly description). Explain why a detected situation is anomalous (anomaly explanation). Justify how an anomaly might have occurred (anomaly justification). Localize anomalies within images via bounding boxes (anomaly localization). Classify anomalies by their visual manifestation type and numerical features (severity, surprisal, complexity). It also serves as a resource for studying the alignment between human and machine processing of visual anomalies, and for developing improved prompting strategies or fine-tuning approaches for anomaly-related tasks. Out-of-Scope Uses CAVE is a benchmark for evaluation purposes. Its small size (361 images) makes it unsuitable as a training set. It should not be used to deploy anomaly detection systems in safety-critical settings without additional validation. Dataset Structure Overview CAVE consists of 361 images: 309 anomalous and 52 normal (non-anomalous) images. Anomalous images contain up to 3 anomalies each, totaling 334 annotated anomalies. Each anomaly is paired with a unique bounding box. Annotation Fields Each sample includes the following fields: Field Description image The image (photograph or screenshot) image_description Short description of the image content (without describing the anomaly) anomaly_description Textual description of what is anomalous in the image anomaly_explanation Explanation of why the situation is anomalous (commonsense reasoning) anomaly_justification Plausible explanation of how the anomaly might have occurred anomaly_category Category of the anomaly's visual manifestation (see taxonomy below) bounding_box Coordinates of the bounding box demarcating the anomalous region severity 1–5 score: does the anomaly require immediate action? surprisal 1–5 score: how much does the situation deviate from expectations? complexity 1–5 score: how hard is the anomaly to detect? Anomaly Category Taxonomy Anomalies are categorized by how they visually manifest, inspired by MMBench's taxonomy of visual reasoning types: Category Description Example Entity Presence An object is present when it shouldn't be A black bear in an industrial building Entity Absence An expected object is missing A person using a cutter without protective gear Entity Attribute An object has an anomalous attribute (color, shape, label, orientation, usage) A snack packet opened from the wrong side Spatial Relation An object is incorrectly positioned relative to another Furniture blocking an emergency button Uniformity Breach A disruption in an expected uniform/symmetrical pattern One tile with a different orientation Textual Anomaly Text in the image conveys an unexpected or contradictory message A "KEEP RIGHT" sign with an arrow pointing left Dataset Creation Images were collected from four Reddit subreddits that specialize in content featuring unusual or uncommon situations: r/ocdtriggers r/mildlyconfusing r/mildlyinfuriating r/OSHA The top 1,000 posts from each subreddit were downloaded using the PRAW library. Images were filtered through both automatic and manual processes to remove: Unclear or ambiguous content Non-realistic images NSFW or sensitive content Images with text annotations, circles, or other overlaid marks Images below icon resolution Annotation proceeded in two rounds, with Amazon Mechanical Turk followed by Expert Verification & Consolidation, with 3 independent raters per anomaly for severity, surprisal, and complexity scores. Citation @inproceedings{bhagwatkar-etal-2025-cave, title = "{CAVE} : Detecting and Explaining Commonsense Anomalies in Visual Environments", author = "Bhagwatkar, Rishika and Montariol, Syrielle and Romanou, Angelika and Borges, Beatriz and Rish, Irina and Bosselut, Antoine", booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing", month = nov, year = "2025", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2025.emnlp-main.1379/", doi = "10.18653/v1/2025.emnlp-main.1379", pages = "27110--27151", } Acknowledgements The authors acknowledge support from Canada CIFAR AI Chair Program, Canada Excellence Research Chairs Program, Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29), EPFL Center for Imaging, Sony Group Corporation, and a Meta LLM Evaluation Research Grant. Computational resources were provided by MILA - Quebec AI Institute.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.