Deep Learning at a Distance: Remotely Working to Surveil Sharks
Bibliographic record
Abstract
Abstract In the months following the novel Coronavirus pandemic outbreak, the world has seen an immediate and unprecedented global shift towards remote learning and working. In the academic field specifically, it has fundamentally shifted how the process of learning happens. Throughout the summer of 2020, we had the opportunity to observe how doing research remotely would affect the complicated dynamics of working in a cross-disciplinary team. Our project centered around utilizing machine learning technologies to detect sharks in videos taken from drones, as well as a few possible applications of this technology. Traditionally, a project such as this would involve weekly in-person meetings, with in-person collaboration opportunities on such things as developing our Neural Network computational model, designing user interfaces, and discussing shark behaviors with domain experts such as marine biologists. However, due to the circumstances of the pandemic, we had to make do with weekly online Zoom meetings, as well as figuring out how to collaborate with each other to do the technical aspects of this project remotely. Our team of engineering students comes from a variety of backgrounds including computer science, software engineering, biomedical engineering, and marine biology. The project itself incorporates the use of drones to collect video footage, Machine Learning to process the images, and marine biology in order to analyze the behavior of sharks in their natural habitat in a noninvasive way. Collaboration with a team of marine biologists specializing in sharks at a different university was essential, but our inability to meet with them in person imposed a significant hurdle. Working remotely with a team of this size and range of skills was a learning process during which we overcame numerous logistical, technical, and personal obstacles. In the end, however, we succeeded in developing a system that is capable of locating objects of interest in the video footage and to assign those objects to categories such as shark, seal, tuna, boat, surfer, paddleboarder, swimmer, and others. This system is the basis for a front-end to be used by marine biologists in the behavior of sharks and other marine life: whereas in the past, marine biology students would spend endless hours watching drone video footage to identify snippets of interest, they can now focus on the behavior analysis of the animals. Although the immediate results of this project were obtained in the identification of sharks, work is underway to expand this to other domains. While object recognition has been studied and applied widely, our specific situation involving drones flying over water posed additional challenges such as the presence of glare, waves, and foam. Further challenges are the identification of relatively small objects at varying depths in the water, viewed from a moving object (the drone) at different heights, speeds, and angles. From a technical perspective, the use of a centralized database, advanced labeling software, sophisticated Machine Learning tools, and powerful cloud computing facilities allowed us to do meaningful work during this time while keeping ourselves organized and productive. Though we could not physically meet each other or any sharks, this summer project was an invaluable learning experience with innovation in the application of Artificial Intelligence to show for it.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.002 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.002 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".