Achieving online big data processing: from offline to online data analytics
Bibliographic record
Abstract
Nowadays, a huge volume of data is increasingly produced by various sources, such as mobile sensing devices, system logs, and user activities over Internet.Accordingly, the information hidden in the massive size of data has a great potential to change and benefit our lives in many fields.To explore the values from data, researchers have made efforts to provide solutions to accommodate the big data applications.These applications not only require scalability on computation and storage but also bring a pressing challenge on data processing speed.If the data processing speed cannot catch up with the data generation speed, the computing results provided by the data analytic systems will be useless.Additionally, in many applications such as real-time recommendation system and web index maintenance, dataset involved in the computation needs to be frequently updated.To catch the change of individual data items and deliver the correct result reactively under the fast evolving dataset, we have to adopt online data analytic systems.Unfortunately, there exists a gap between the state-of-the-art data analytic system and the requirement of the online data analytics.It prevents the current data analytic systems/algorithms from being adopted in the more challenging, yet more realistic, online scenarios.In this thesis, we identify three major requirements to move the conventional offline computing systems to online: (1) First, we need to build a fluent online processing pipeline which consumes input data reactively and with high throughput; (2) We desire a storage layer which serves online query and update to the computing state so that it can deliver the results adaptively against the fast evolving dataset; (3) Finally, while moving from offline to online, the data analytic systems should be backward compatible to its offline counterparts, hence, we need to keep the maximum compatibility with infrastructure, programming model, etc.ii In this thesis, we study the solutions to fulfill these three requirements.(1) We design and implement the micro-batch-based online data processing pipeline with fine-grained resource allocation to avoid the per-record backup and handle the highly-skewed computing demands, e.g.videos with various length.(2) We build a parallel-friendly and high-performance computing state management system based on Locality Sensitive Hashing (LSH).Our system differentiates with the conventional LSH system in that it does not need to reconstruct itself to serve the online update requests and adopt multiple system-oriented optimization strategies to scale to large storage size and the multi-cores environment.(3) We achieve the scalable, efficient and reliable computing state management for online data analytics by introducing Resilient State Table (RST) as a component of Spark.So the system we have designed can seamlessly integrate with the programming model and scheduling policy of the original Spark framework.We evaluate our design in various scenarios.Our design exceeds the performance of the state-of-the-art systems in many application domains, e.g. the content-based video indexing, high dimensional nearest neighbors search, real-time web index maintenance, and machine learning algorithms.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.005 | 0.013 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.002 | 0.003 |
| Scholarly communication | 0.011 | 0.020 |
| Open science | 0.005 | 0.008 |
| Research integrity | 0.002 | 0.007 |
| Insufficient payload (model declined to judge) | 0.003 | 0.005 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".