Handling Large-Scale Data using Two-Tier Hierarchical Super-Peer P2P Network
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
In the ever-expanding world of IoT, data has not only increased in volume and velocity but has also moved from residing in centralized nodes to distributed nodes across multiple locations. Traditional data clustering technologies, based on centralized operations, cannot be scaled to efficiently manage Big Data, thus creating a need for clustering in distributed environments. To address the problem of modularity, flexibility, and scalability, a dynamic hierarchical two-tier architecture and model for cooperative clustering in distributed super-peer P2P network is presented in this paper. The proposed model is called Distributed Cooperative Clustering in super-peer P2P networks (DCCP2P). It involves a hierarchy of two layers of P2P neighborhoods. In the first layer, peers in each neighborhood are responsible for building local cooperative sub-clusters from the local data. Each node sends a summarized view of local data to its super-peer in a form of sub-cluster's centroids extracted from the local cooperative clustering, minimizing the exchange of information between nodes and their super-peers. In the next layer, sub-clusters are merged at each super-peer and at the root of the hierarchy, where one global clustering can be derived. The distributed cooperative approach finds globally optimized clusters and achieves significant improvement in global clustering solutions without the cost of centralized clustering.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.001 | 0.002 |
| Open science | 0.011 | 0.013 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it