Intelligent HPC: Slashing Turnaround Time via ML-Driven Resource Prediction
Improving HPC System Performance by Predicting Job Resources via Supervised Machine Learning
This paper introduces a supervised machine learning framework integrated with the Slurm Workload Manager to predict required memory and execution time for HPC jobs. By training on 14 million historical records from the "Beocat" cluster, the authors demonstrate that a Decision Tree Regressor significantly outperforms traditional user-estimated resource allocations.
TL;DR
High-Performance Computing (HPC) efficiency is often throttled by a human factor: users over-estimate job resources to avoid "killed" tasks. This paper presents a supervised learning solution integrated into the Slurm Workload Manager that predicts actual memory and time needs. The result? A staggering reduction in large-job turnaround time from 5 days to 10 hours.
Executive Summary
In the world of supercomputing, the Slurm scheduler is king, but it is only as smart as the data it receives. Most researchers are more concerned with their science (biology, physics, etc.) than with hardware specs, leading them to request "safety margins" of memory and time that waste massive amounts of cluster capacity. This research bridges that gap by using historical log data to train a Decision Tree Regressor that tells Slurm what a job actually needs, effectively doubling or tripling system throughput.
The "Safety Margin" Bottleneck
The core problem is an incentive misalignment. If a user requests 10GB but uses 12GB, their job is killed and hours of work are lost. Consequently, users request 50GB "just in case."
- The Cost: The scheduler sees that 50GB as "occupied," leaving other jobs trapped in the queue even if the hardware is sitting idle.
- The Complexity: Predicting usage is non-linear; it depends on the specific user's habits, the number of CPUs, and the type of task.
Methodology: From Logs to Intelligence
The authors leveraged eight years of history (14 million instances) from the Beocat cluster. They transformed this raw log data into a predictive engine using several regression models.
The Prediction Workflow
- Submission: User submits a job with their (usually inflated) estimates.
- Inference: The ML model intercepts the submission and predicts realistic
wclimit(time) andreq_mem(memory). - Update: The model modifies the job metadata before the scheduler processes it.
- Scheduling: Slurm's backfill algorithm uses the optimized values to fit the job into the smallest possible "hole" in the compute schedule.
Figure 1: The integration of the ML model within the standard job submission cycle.
Among the tested algorithms, the Decision Tree Regressor (DTR) emerged as the winner, achieving an R-squared value nearly 10x higher than standard Linear Regression for runtime prediction.
Experimental Results: A 90% Improvement
The authors tested their model using two testbeds: one for "large" jobs (high resource demand) and one for "smaller" jobs.
Large Job Breakthrough (Testbed-1)
The impact on large jobs was transformative. When using user-requested resources, the total execution flow took 5 days. With ML-predicted resources, it dropped to 10 hours.
- Average Wait Time: Dropped from 45.37 hours to 4.0 hours.
- System Utilization: Increased significantly, as the "ghost" resources previously reserved for overestimated jobs were now available for others.
Figure 2: Job execution timeline comparison. The "Predicted" line nearly matches the "Actual" usage, whereas "Requested" (User input) lags significantly.
The Memory vs. Time Debate
A unique insight from this paper is the analysis of which prediction matters more. The authors found that predicting memory and time simultaneously provides the peak performance gain, though both are roughly equal in their impact on preventing scheduler fragmentation.
Critical Insight & Conclusion
While the current model focuses on traditional regression, the authors identify a clear path forward: Reinforcement Learning (RL). By treating scheduling as a sequential decision-making problem, future systems could "learn to act" and migrate jobs between local clusters and the cloud based on predicted cost-utility.
Takeaway for Practitioners: If you are managing an HPC cluster, your biggest performance gain likely isn't faster CPUs—it's smarter scheduling. By implementing even a basic Decision Tree regressor on job logs, you can reclaim nearly 80% of "wasted" waiting time.
Limitations
The study was conducted in a simulated environment (Slurm Simulator). While highly accurate, real-world "production" noise (hardware failures, network congestion) might slightly degrade the R-squared performance. Additionally, the model currently relies on categorical data like username, which might require retraining as the user base shifts.