Performance evaluation and optimization of an adaptive scheduling approach for dependent grid jobs with unknown execution time
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Most nowadays scheduling algorithms for grids are based on the assumption that the application (job) execution time is known before job run-time. This assumption significantly simplifies job-resource matchmaking, although it has proven to be inapplicable for most real-world applications. On the other hand, there exists a group of applications for which execution progress can be monitored at run-time. Often, when a correlation between job execution progress and total job execution time exists, progress information can serve as a good basis for the prediction of the remaining execution time. Another important issue in the domain of distributed computing is scheduling of jobs composed of tasks with input dependencies, whereby some tasks require inputs generated by other tasks. Since the overhead due to input dependencies is limited, this type of dependencies forms a potential for execution optimization by means of intelligent scheduling of dependent tasks on distributed resources. In this article a detailed performance evaluation and optimization is provided for an adaptive scheduling algorithm for grids that was proposed earlier. The algorithm operates on jobs with input inter-dependencies, whose sub-tasks are organized into a DAG (Directed Acyclic Graph) and for which no information of total execution time is available. The idea behind the approach is that parallel tasks (parent tasks), generating input for the same underlying set of tasks (dependent tasks), should finish more or less simultaneously. Since the dependent tasks can only be executed after all the required inputs are available, the longest parent task is assigned to the fastest available resource, while shorter tasks can be assigned to slower resources, as long as it does not prolong the execution time of the parent set as a whole. The latter creates a possibility for other tasks requiring fast processing to be executed on faster machines. At first, tasks are assigned randomly. Later, the algorithm reacts on dynamic changes in resource status and variations in task execution time predictions by possibly rescheduling parallel tasks. The algorithm's performance was evaluated using workload originating from an existing modeling and virtual experimentation tool for environmental systems (Tornado). Results have shown that significant system overhead is introduced, in terms of additional computational and network load due to the extended checkpointing and migration mechanisms. However, this overhead is compensated by more effective processing of parallel sub-tasks, which are now occupying only resources they strictly need in order not to delay the execution of the job as a whole. In this paper we measure the overhead introduced by the algorithm on network and computational resources and compare it to the overhead of a traditional static approach. It is clear that the effectiveness of the adaptive approach strongly depends on the degree of parallelism of sub- tasks and on their overall execution time heterogeneity. The boundaries for both parameters are studied. Furthermore, the performance of the algorithm can be improved by postponing migration in cases where the benefit of rescheduling is expected to be sufficiently low. Definition of the boundary for the migration postponement is also addressed.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.006 | 0.009 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it