Bibliographic record
Abstract
Video data include a significant amount of useful information that can be exploited by di↵erent organizations to gain more insights into running their business.The data are large and growing at a high rate because cameras, nowadays, are installed everywhere, gathering information all the time.Therefore, the data need large storage to be preserved and large computing power to be processed.Di↵erent technologies such as Apache Spark, Apache Storm, and Apache Hadoop have been widely used to perform big data processing on computer clusters.This thesis introduces general solutions and algorithms that can be used with di↵erent technologies, such as Apache Spark, Apache Storm, and Apache Hadoop, to improve the performance of processing big video data on computer clusters.However, the thesis focuses on using Apache Hadoop to provide an empirical evaluation of the proposed algorithms.Apache Hadoop is selected since it has been designed to work on commodity hardware.This thesis has been investigating di↵erent approaches to improve the performance of video processing on Hadoop clusters.These approaches are based on using the Hadoop MapReduce programming model to distribute the processing of big video data, and di↵erent sampling methods to avoid unnecessary computation while processing the video data.Change detection is used to detect the changes based on which the proposed algorithms sample the frames.The proposed algorithms can support video processing on faulty systems as well.The thesis also proposes three novel data placement policies to improve the performance of MapReduce-based video processing.iii O Overhead of processing overlapping frames twice p Degree of parallelism or number of segment r Round number st Stride number T h High threshold value T l Low threshold valueSet of all data frames of video segment j V j bm Set of all binary mask frames V j h Set of all history frames of video segment j V j h 0 Set of sample history frames of video segment j w d Size of DSW w h Size of HSW ↵ Overhead of processing overlapping frames twice ↵ 0 Overhead of processing the sample overlapping frames twice ⌧ The saved execution time because of sampling xviii
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.008 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.002 |
| Science and technology studies | 0.002 | 0.000 |
| Scholarly communication | 0.002 | 0.003 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".