NLP-based Traffic Scene Retrieval via Representation Learning
Bibliographic record
Abstract
Many automated systems require the interpretation of visual information, i.e., images, videos, and natural language input, i.e., speech or text, to comprehend their surroundings and communicate with interacting humans. One such hybrid application of computer vision using images and videos and natural language processing (NLP) recognizes traffic scenes, a crucial and challenging problem in automated transportation systems. Scene classification is just one of many areas where recent convolutional neural network (CNN) frameworks have proven to be highly effective. Still to be fully explored for application to problem-solving in the real world is CNN's impressive, truly representative learning capability. However, newer CNN implementations, such as YOLO and DeepSort, show promise for object detection. The BERT model is the benchmark for text embeddings and the most efficient method currently available. Hence, we aim to retrieve the vehicles from the traffic videos using natural language-based description, i.e., text. The paper proposes a novel approach that combines YOLOv7, the recent version of YOLO, DeepSort algorithms for object detection, i.e., detecting the vehicles from the traffic scene from the frames of the videos and the transfer learning model, i.e., BERT model for text embeddings. Additionally, a Kalman filter is utilized to track the cars by providing the id and will retain them in the other frames of the videos. The machine learning model performs the similarity checking, i.e., siamese neural networks. The experiments are performed on the standard dataset of AI city challenge 2022. Moreover, the results depict that the proposed approach achieves 28.49 % of Recall@5, 42.08 % of Recall@10, and 20.73 % of MRR, indicating the proposed method's effective approach.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.002 |
| Meta-epidemiology (narrow) | 0.002 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.005 | 0.004 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.001 | 0.004 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.007 | 0.007 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".