A Parallel Processing Technique for Extracting and Storing User Specified Data
Bibliographic record
Abstract
Users are often interested in a specific type of data (preferences) available from a large volume of data collected on the system. An efficient and effective system that can only store the user preferred data from the large raw data set helps the users to search for relevant information on time. The motivation behind this paper is to devise such a technique. The technique uses machine learning as well as parallel processing to efficiently filter out the user preferred data from a large raw data set. Firstly, the technique stores the filtered data and discards the remaining data which saves storage space for the user. Secondly, it leads to an enhanced searching experience for the user by reducing the search latency. Running the filtering operation can be CPU intensive which often leads to high latency for extracting user preferred data from the raw data set. To solve this problem, the technique employs parallel processing and machine learning, thus reducing the data filtering latency while making the data searching process faster for the user. A proof-of-concept prototype for this technique has been built on the Apache Spark parallel processing engine. The prototype is subjected to several performance experiments using synthetic datasets. The analysis of experimental results shows the viability of the proposed technique and provides insights into system behavior and performance.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.004 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.005 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.004 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".