iSklearn: Proving that Simpler Pipelines + Smarter Innovation = SOTA AutoML

TL;DR

iSklearn is a new AutoML powerhouse that swaps complex ensembles for high-performance, minimalist scikit-learn pipelines. By utilizing the irace configurator—a tool originally famous in the optimization community—it bridges the gap between massive parallelization and intelligent model-based learning. It doesn't just match state-of-the-art tools like Auto-sklearn; in domains like Time Series and CV, its interpretable, "vanilla" pipelines often come out on top.

Context: The Cost of Complexity

In the race for higher accuracy, the AutoML industry has pivoted toward Deep Learning and massive Ensembles. While effective, these "black boxes" are computationally expensive, environmentally draining, and nearly impossible to interpret. The authors of iSklearn ask a critical question: Can we achieve SOTA results using simpler pipelines if we optimize the search process and sampling logic more effectively?

The Core Mechanism: Why irace?

Most AutoML tools use SMAC (Sequential Model-Based Optimization) or HyperBand (Bandit-based).

• SMAC is smart but sequential (hard to parallelize).
• HyperBand is fast and parallel but "blind" (model-free).

irace acts as the middle ground. It uses an Estimation of Distribution Algorithm (EDA) to maintain a population of candidate configurations. It employs a "racing" mechanism: configurations are tested on problem instances, and statistically inferior ones are pruned early. This allows more resources to be "sharpened" on the most promising candidates.

The iSklearn Pipeline Template

The system uses a grammar-based template (see below) that forces the search toward clean, 3-stage architectures:

1. Preprocessing: Standard Scaling.
2. Feature Engineering: Selection (Univariate/Multivariate) and Extraction (PCA/SVD).
3. Prediction: Family of models including Linear Models, SVMs, RFs, and MLPs.

Overcoming the Time Series Hurdle

A standout contribution of iSklearn is its Generalized Sampling Setup. While traditional AutoML uses simple holdout sets (which fails for temporal data), iSklearn introduces:

• Meta-folds: Partitioning data into $k$ meta-folds.
• Bottom-level Cross-Validation: Evaluating candidates on $p$ meta-folds independently to ensure robustness against class imbalance and temporal shifts.

Experimental Showdown: Pipelines vs. Ensembles

The researchers tested iSklearn against Auto-sklearn across Computer Vision (CV), NLP, and Time Series (TS).

Key Findings:

• CV Performance: On Fashion MNIST, iSklearn's simple pipeline achieved 88.12%, crushing Auto-sklearn's 79.88%.
• Interpretability: The sunburst charts below show how iSklearn adapts to different domains. NLP tasks almost exclusively converged on Linear Regression (LR), while CV tasks required a diverse mix of SVMs and MLPs.
• Time Series Superiority: By using the "Triple Meta-fold" (TM) setup, iSklearn outperformed Auto-sklearn across the board, as complex ensembles in Auto-sklearn tended to overfit on temporal data.

Quantitative Results

The table below highlights that iSklearn (a single pipeline) is remarkably competitive with the ensemble logic of Auto-sklearn.

Critical Insight: The "Ablation" Lesson

The authors conducted an ablation study to isolate whether the success came from the concurrence of irace or the minimalist template. By running our minimalist template through SMAC (the engine of Auto-sklearn), they found that the Template itself is a major contributor to performance.

However, the Sampling Setup (Factor $p$ and Cutoff time) proved to be the "secret sauce" for CV and TS tasks. Increasing the configuration budget from 2000 to 5000 experiments showed diminishing returns compared to simply improving the sampling quality.

Conclusion & Future Outlook

iSklearn proves that "less is more" in AutoML. By focusing on a minimalist but structurally sound search space and applying a statistically rigorous configurator like irace, we can build models that are faster, greener, and more interpretable.

Future Work: The team aims to automate the "Data Preparation" stage (which was manual in this study) and explore multi-objective configuration—optimizing for both accuracy and inference speed simultaneously.

Main Takeaway for Practitioners: If you're working with non-tabular data or Time Series, don't assume a complex ensemble is the answer. A perfectly tuned "vanilla" pipeline may be hiding SOTA performance.