ML-Driven Simulations: Orchestrating Energy Efficiency in High-Throughput Computing

TL;DR

High-Throughput Computing (HTC) environments like HTCondor often suffer from massive energy waste—up to 65% in some cases—due to "miscreant" tasks that never complete. This paper demonstrates that by applying Deep Learning and Random Forest Regression to historical trace-logs, we can predict task success (99% accuracy), runtime, and memory usage using only submission-time metadata. This allows simulators to generate high-fidelity synthetic workloads and enables real-time schedulers to kill "bad" tasks before they waste power.

Background: The Hidden Cost of "Volunteer" Computing

In systems like Newcastle University's HTCondor cluster, computers are often "borrowed" from office desktops. If a primary user logs back in, the HTC task is evicted. If a task is misconfigured (a "miscreant" task), it may cycle through the system indefinitely, consuming electricity without ever producing a result.

The authors analyzed five years of data (over 640,000 tasks) and found a staggering reality: for every 1 second of useful work, the system consumed 2.8 seconds of actual computing time.

The Problem: The Simulation Paradox

Researchers use "trace-driven simulations" to test new energy-saving policies. However, they face two hurdles:

1. Information Leakage: Traces contain the actual execution time, but a real scheduler doesn't know this at the start. Using this data "cheats" the simulation.
2. Fidelity Loss: Statistical models used to generate synthetic traces (to preserve privacy) often lose the "fine-grained" patterns of real users.

Methodology: High-Fidelity Prediction

The authors treat the trace-log not just as a sequence of events, but as a training set for a multi-faceted ML architecture.

1. Identifying the "Good" and the "Bad"

Using metadata like Owner, Command, and Submission Time, the authors used Deep Learning (Stacked Denoising Autoencoders) combined with oversampling (to handle the imbalance of successful vs. failed tasks).

Figure 1: High-level overview of using ML to extract latent patterns from HTC trace-logs.

2. Estimating Latent Characteristics

Since user-provided estimates for runtime and memory are notoriously inaccurate, the paper utilizes Random Forest Regression. This model maps the submission metadata to the expected Duration and ImageSize (memory footprint), allowing the simulator to operate based on predicted values rather than actual ones.

Experimental Insights & Results

The analysis of the Newcastle data (Table 4 in the paper) shows that tasks removed by users or terminated accounted for 51% of total energy consumption.

The authors specifically highlight that traditional statistical models couldn't capture the "spiky" nature of user behavior (Fig 1 in the paper), whereas the ML models mirrored the actual execution profiles much more closely.

Figure 2: Analysis of execution times showing non-standard distributions that ML captures better than traditional statistics.

Critical Analysis & Future Outlook

The core value of this work is the transition from descriptive trace analysis to predictive system modeling.

Takeaway: By predicting failure early, we can prevent "miscreant" tasks from even starting, potentially cutting system energy usage by half.

Limitations: The study focuses on a specific Windows-based university cluster. Future work should validate these ML models on more modern, containerized Linux environments (like Kubernetes-based HTC) where resource isolation might change the "miscreant" behavior patterns.

Conclusion: This paper proves that Machine Learning isn't just for the tasks running on the cluster—it's the key to orchestrating the cluster itself for a sustainable, energy-aware future.