Deep Learning at a Distance: Remotely Working to Surveil Sharks
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Abstract In the months following the novel Coronavirus pandemic outbreak, the world has seen an immediate and unprecedented global shift towards remote learning and working. In the academic field specifically, it has fundamentally shifted how the process of learning happens. Throughout the summer of 2020, we had the opportunity to observe how doing research remotely would affect the complicated dynamics of working in a cross-disciplinary team. Our project centered around utilizing machine learning technologies to detect sharks in videos taken from drones, as well as a few possible applications of this technology. Traditionally, a project such as this would involve weekly in-person meetings, with in-person collaboration opportunities on such things as developing our Neural Network computational model, designing user interfaces, and discussing shark behaviors with domain experts such as marine biologists. However, due to the circumstances of the pandemic, we had to make do with weekly online Zoom meetings, as well as figuring out how to collaborate with each other to do the technical aspects of this project remotely. Our team of engineering students comes from a variety of backgrounds including computer science, software engineering, biomedical engineering, and marine biology. The project itself incorporates the use of drones to collect video footage, Machine Learning to process the images, and marine biology in order to analyze the behavior of sharks in their natural habitat in a noninvasive way. Collaboration with a team of marine biologists specializing in sharks at a different university was essential, but our inability to meet with them in person imposed a significant hurdle. Working remotely with a team of this size and range of skills was a learning process during which we overcame numerous logistical, technical, and personal obstacles. In the end, however, we succeeded in developing a system that is capable of locating objects of interest in the video footage and to assign those objects to categories such as shark, seal, tuna, boat, surfer, paddleboarder, swimmer, and others. This system is the basis for a front-end to be used by marine biologists in the behavior of sharks and other marine life: whereas in the past, marine biology students would spend endless hours watching drone video footage to identify snippets of interest, they can now focus on the behavior analysis of the animals. Although the immediate results of this project were obtained in the identification of sharks, work is underway to expand this to other domains. While object recognition has been studied and applied widely, our specific situation involving drones flying over water posed additional challenges such as the presence of glare, waves, and foam. Further challenges are the identification of relatively small objects at varying depths in the water, viewed from a moving object (the drone) at different heights, speeds, and angles. From a technical perspective, the use of a centralized database, advanced labeling software, sophisticated Machine Learning tools, and powerful cloud computing facilities allowed us to do meaningful work during this time while keeping ourselves organized and productive. Though we could not physically meet each other or any sharks, this summer project was an invaluable learning experience with innovation in the application of Artificial Intelligence to show for it.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.003 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.011 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it