Predicting the Unpredictable: Learning-Based Job Wait Time Estimation in HTC Datacenters

TL;DR

Researchers at CC-IN2P3 have developed a Machine Learning framework to solve a major frustration for scientists: not knowing when their computing jobs will start. By shifting from exact time predictions to time-range classification using a "Regression + Classification" hybrid model, they achieved a 77% accuracy rate in placing jobs in the correct (or nearly correct) wait window, even in highly complex Fair-Share scheduling environments.

Problem & Motivation: The Black Box of HTC Scheduling

In Large Scale High Throughput Computing (HTC) centers, such as those powering the LHC at CERN, thousands of users submit millions of single-core jobs. These systems use Fair-Share Schedulers, which prioritize jobs based on historical usage and group pledges.

The problem? Fair-share scheduling is a "moving target." Unlike HPC systems that use backfilling (where gaps in the schedule are filled by smaller jobs), HTC systems stay close to 100% utilization. This makes the wait time highly dependent on:

• Resource Quota Sets (RQS): Hidden limits on specific storage or licenses.
• Group Shares: How much your team has already "spent" this year.
• System Bursts: Sudden influxes from high-priority groups that push everyone else back.

Users are left in the dark, leading to poor workflow management. If you don't know if your job will start in 5 minutes or 12 hours, you can't plan your next move.

Methodology: Mapping the Intuition to Math

The authors didn't just throw raw data at a model. They first identified the "Why" behind the wait.

1. Feature Engineering

They mapped intuitive causes—like "Who is submitting?" and "When?"—to specific features. Surprisingly, they found through Spearman’s Rank Correlation that "What is requested" (CPU time/Memory) had almost zero influence in the HTC context. Instead, the most impactful features were:

• RQS Consumption: Current group-level violations.
• System State: A PCA-reduced representation of global and queue-level pending slots.

2. The Model Architecture

While multiple models were tested (Linear Regression, AdaBoost, Bagging), the Decision Tree Regressor (Depth 7-8) emerged as the winner. It offered the best trade-off between training speed and accuracy.

Table 1: The reduced set of features derived from intuition and correlation analysis.

Experiments & Results: Accuracy vs. Utility

The study discovered a fascinating bias: "Direct Classification" models (trying to put a job directly into a 2-4 hour bucket) performed poorly because the data is dominated by "quick starters" (jobs that start in <30 mins). This makes the model "lazy"—it predicts nearly everything will start immediately.

The Hybrid Solution

The breakthrough came from the "Reg. + Clas." approach:

1. Perform a Regression to get a specific value (e.g., predicted wait = 38 minutes).
2. Map that value to a Classification Range (e.g., Range 2: 30 minutes to 2 hours).

Figure: The Confusion Matrix for the hybrid approach shows a much better distribution along the diagonal compared to binary-like direct classifiers.

Key Results:

• Combined Accuracy: 76.92% (Correct or Adjacent range).
• Robustness: The model stays accurate even as wait times grow beyond 9 hours, which is exactly where users need the prediction the most.

Critical Analysis & Conclusion

This work highlights a fundamental truth in ML for systems: Context is King. By including administrative constraints (shares and quotas) rather than just job hardware requirements, the authors cracked the code of HTC wait times.

Limitations & Future Work

• Imbalanced Data: The "quick starter" bias still exists and requires more sophisticated clustering to solve.
• Model Aging: As infrastructure changes, the model must be retrained. The authors are looking into the minimum training size required to keep the model fresh.

Final Takeaway

For datacenter operators, the message is clear: You don't need a massive "black box" neural network to satisfy users. A well-tuned, feature-rich Decision Tree can turn an unpredictable batch system into a transparent utility, significantly improving the "Quality of Life" for the scientific community.