NLP-based Traffic Scene Retrieval via Representation Learning
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Many automated systems require the interpretation of visual information, i.e., images, videos, and natural language input, i.e., speech or text, to comprehend their surroundings and communicate with interacting humans.One such hybrid application of computer vision using images and videos and natural language processing (NLP) recognizes traffic scenes, a crucial and challenging problem in automated transportation systems.Scene classification is just one of many areas where recent convolutional neural network (CNN) frameworks have proven to be highly effective.Still to be fully explored for application to problem-solving in the real world is CNN's impressive, truly representative learning capability.However, newer CNN implementations, such as YOLO and DeepSort, show promise for object detection.The BERT model is the benchmark for text embeddings and the most efficient method currently available.Hence, we aim to retrieve the vehicles from the traffic videos using natural language-based description, i.e., text.The paper proposes a novel approach that combines YOLOv7, the recent version of YOLO, DeepSort algorithms for object detection, i.e., detecting the vehicles from the traffic scene from the frames of the videos and the transfer learning model, i.e., BERT model for text embeddings.Additionally, a Kalman filter is utilized to track the cars by providing the id and will retain them in the other frames of the videos.The machine learning model performs the similarity checking, i.e., siamese neural networks.The experiments are performed on the standard dataset of AI city challenge 2022.Moreover, the results depict that the proposed approach achieves 28.49 % of Recall@5, 42.08 % of Recall@10, and 20.73 % of MRR, indicating the proposed method's effective approach.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.004 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it