Data Oriented Simulation Framework in a Decentralized and Asymmetric HPC Ecosystem
Bibliographic record
Abstract
The MOSAIC lightweight framework is a SAFRAN middleware targeting data management in an infrastructure including HPC facilities. In this paper we show how using a data oriented paradigm allows us to speed up simulation workflows and reduce the overall footprint of files, taking advantages of storage and computational resources availability within a large network of HPC infrastructure. Rather than defining a process consuming and producing data all along its run, we declare dependencies between existing and expected data and we let the system find the right production graph. This allows a makefile-like behavior of session following a workflow, trying to find actual data, fetching and using existing data matching the requirements or producing only those missing. The dependencies analysis is used to find potentially independent parts of the workflows to be run in parallel or to find which is the best workflow to select to obtain the expected final data. The dependency definition is formalized as a database query, this defines the set of accepted candidates as data source or product. Each data of the system has a set of qualifications, a query would look for existing data matching these qualifications and if found would eventually transfer it to the expected location for use. Definitions of data requirements and their relationships are specified in a profile. As these requirement are set of characteristics and not types, the users can define as many profiles they want. We have then different views of the same data, making it possible to have asymmetric understanding of a shared parts of large simulation processes. We have a lightweight representations of file contents, so-called datasets, which have qualifications describing key contents with predefined keywords. Some of these keywords can be generated automatically, for example when the file of the dataset is compliant to a standard like CGNS and it is easier to find key characteristics of the data. A Dataset refers to files but does not embed files, which means some production has been achieved, or no files, which means the dataset has files somewhere else on the network or is waiting for being produced. These datasets are duplicated in multiple databases across a network with multiple storage and computation resources. The framework is in charge of interpreting the data request all over the network and insure a lazy file transfer if required in a concurrency environment. On an HPC system this dramatically reduces the redundancy of files and the actual wall clock time of complex workflow executions. At the end of this paper, we illustrate the use and the performance of such a data oriented system on actual simulation workflows.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.002 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".