Automatic Image Tagging and Captioning Using Transformer-Based Vision-Language Models
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
The rapid expansion of visual data in sectors like healthcare, e-commerce, and social media increases the demand for efficient photo tagging and labelling systems. Many of the automatic photo tagging and labelling techniques in use today struggle with the complex relationships between visual content and natural language, which reduces their accuracy and scalability. Recent advances in transformerbased models, particularly Vision-Language Transformers (ViLT), have greatly simplified the interaction between images and textual claims. These models simultaneously handle visual and textual input using transformers. This improves feature extraction and semantic matching. This paper investigates the automated tagging and description of pictures using transformer-based vision-language models. Our proposed approach generates tags and descriptions for pictures that fit in their present context by combining modern vision transformers with language models. The system is made up of two main parts: a vision encoder that takes in a picture and pulls out visual features; and a text decoder that uses the extracted features to make useful subtitles or tags. We also present a multi-modal training approach that lets the model learn from both written and visual data at the same time. This makes it better at many real-world tasks. We did a lot of tests on standard datasets to show that our suggested model is much better than current ones at making subtitles and tags that are accurate, fluent, and relevant. The results show that transformer-based vision-language models can be used to automatically understand images and create material for a wide range of purposes.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it