The Sentinels EOPF Toolkit: Driving Community Adoption of the Zarr data format for Copernicus Sentinel Data
Bibliographic record
Abstract
The Standard Archive Format for Europe (SAFE) specification has been the established approach to publishing Copernicus Sentinel data products for over a decade. While SAFE has pushed the ecosystem forward through new ways to search and access the data, it is not ideal for processing large volumes of data using cloud computing. Over the last few years, data standards like STAC and cloud-native data formats like Zarr and COGs have revolutionised how scientific communities work with large-scale geospatial data and are becoming a key component of new data spaces, especially for cloud-based systems.The ESA Copernicus Earth Observation Processor Framework (EOPF) will be providing access to “live” sample data from the Copernicus Sentinel missions -1, -2 and -3 in the new Zarr data format. This set of reprocessed data allows users to try out accessing and processing data in the new format and experiencing the benefits thereof with their own workflows.This presentation introduces a community-driven toolkit that facilitates the adoption of the Zarr data format for Copernicus Sentinel data. The creation of this toolkit was driven by several motivating questions: What common challenges do users face and how can we help them overcome them? What resources would make it easier for Sentinel data users to use the new Zarr data format? How can we foster a community of users who will actively contribute to the creation of this toolkit and support each other? The Sentinels EOPF Toolkit team, comprising Development Seed, SparkGeo and thriveGEO, together with a group of champion users (early-adopters), are creating a set of Jupyter Notebooks and plug-ins that showcase the use of Zarr format Sentinel data for applications across multiple domains. In addition, community engagement activities such as a notebook competition and social media outreach will bring Sentinel users together and spark interaction with the new data format in a creative yet supportive environment. Such community and user adoption efforts are necessary in order to overcome adoption and uptake barriers and to build up trust and excitement to try out new technologies and new developments around data spaces.In addition to introducing the Sentinels EOPF Toolkit, this presentation will also highlight lessons learned from working closely with users on barriers they face in adopting the new Zarr format and how to address them.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.042 | 0.064 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.004 | 0.003 |
| Science and technology studies | 0.003 | 0.003 |
| Scholarly communication | 0.007 | 0.013 |
| Open science | 0.005 | 0.023 |
| Research integrity | 0.002 | 0.005 |
| Insufficient payload (model declined to judge) | 0.011 | 0.013 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".