Automatic Image Tagging and Captioning Using Transformer-Based Vision-Language Models
Bibliographic record
Abstract
The rapid expansion of visual data in sectors like healthcare, e-commerce, and social media increases the demand for efficient photo tagging and labelling systems. Many of the automatic photo tagging and labelling techniques in use today struggle with the complex relationships between visual content and natural language, which reduces their accuracy and scalability. Recent advances in transformerbased models, particularly Vision-Language Transformers (ViLT), have greatly simplified the interaction between images and textual claims. These models simultaneously handle visual and textual input using transformers. This improves feature extraction and semantic matching. This paper investigates the automated tagging and description of pictures using transformer-based vision-language models. Our proposed approach generates tags and descriptions for pictures that fit in their present context by combining modern vision transformers with language models. The system is made up of two main parts: a vision encoder that takes in a picture and pulls out visual features; and a text decoder that uses the extracted features to make useful subtitles or tags. We also present a multi-modal training approach that lets the model learn from both written and visual data at the same time. This makes it better at many real-world tasks. We did a lot of tests on standard datasets to show that our suggested model is much better than current ones at making subtitles and tags that are accurate, fluent, and relevant. The results show that transformer-based vision-language models can be used to automatically understand images and create material for a wide range of purposes.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.002 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.003 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".