D3.1 State-of-the-art report on data collection and analysis
Bibliographic record
Abstract
Today, many countries aim to increase the share of energy consumed that comes from renewable sources. Unfortunately, the electrical power produced from weather-dependent renewable energy sources (RESs; e.g., wind turbines, solar panels) is produced in varying quantities that do not match the varying energy needs. As more and more such renewable energy becomes available, it becomes an increasingly difficult challenge to maintain an energy system that enables the effective use of all available renewable energy. Consequently, tackling this problem is one of the top goals in the energy domain. The MIRACLE (Micro-Request-Based Aggregation, Forecasting and Scheduling of Energy Demand, Supply and Distribution) project aims to invent and prototype key elements of an energy system that is better able to accommodate large volume of electricity from renewable sources. The approach, taken is based on micro-requests that allow an individual consumer/producer to specify acceptable flexibilities in the amounts of energy consumed and the times when this is done. The introduction of such micro-requests from millions of consumers/producers enables fine-grained scheduling of consumption and production of electricity while maintaining a system-wide balance between demand and supply. In order to appropriately manage very large volumes of micro-requests, a reliable, distributed, and highly scalable computer system infrastructure is needed. This deliverable concerns Work Package 3, “State-of-the-art report on data collection and analysis” in the MIRACLE project. We first introduce MIRACLE’s application scenario along with the consequent requirements to the data management infrastructure. Then we survey the state-of-the-art of relevant, existing work on data collection, data integration, query processing, and query optimization from the perspective of the projects requirements. Specifically, the survey covers the following key topics: (1) virtually and materialized integrated systems, including column stores; (2) data exchange solutions, including ETL tools, EAI servers, and data stream management systems; (3) web-scale data management; (4) management of uncertainty in the context of probabilistic databases, OLAP, and data streams; (5) management of multi-version data; (6) efficient tracking of continuous processes; and (7) query optimization based on early aggregation and materialized views. Moreover, relevant existing computer systems in the energy domain are covered. For all technologies surveyed, the relation to MIRACLE is discussed.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.045 | 0.118 |
| Meta-epidemiology (narrow) | 0.003 | 0.002 |
| Meta-epidemiology (broad) | 0.003 | 0.004 |
| Bibliometrics | 0.015 | 0.019 |
| Science and technology studies | 0.003 | 0.002 |
| Scholarly communication | 0.013 | 0.007 |
| Open science | 0.007 | 0.008 |
| Research integrity | 0.003 | 0.003 |
| Insufficient payload (model declined to judge) | 0.101 | 0.149 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".