Toward Intelligent Assistance: Bridging the Gap in Data Mining Process Design

TL;DR

Building an effective Data Mining (DM) pipeline is often seen as an "art" reserved for experts. This paper challenges that notion by introducing IDEA (Intelligent Discovery Electronic Assistant), an ontology-based system that uses AI planning to automatically generate and rank valid DM workflows. In a head-to-head comparison using the KDDCUP 1998 dataset, IDEA's automated recommendations actually outperformed the human contest winners, achieving a 41% profit increase.

The Problem: The "Default Strategy" Trap

Data mining is not just about picking an algorithm; it is a multi-stage process involving preprocessing, induction, and postprocessing. Despite the availability of toolkits like Weka, most practitioners (even experts) suffer from a "consideration set" bias. They stick to what they know—often decision trees—because they are easy to implement without complex preprocessing.

As noted in the analysis of the KDDCUP-2000 competition, only a small fraction of participants explored the full range of available algorithms. This lack of systematic exploration leads to sub-optimal results, especially in complex tasks like cost-sensitive classification.

The Solution: An Ontology-Based Planner

The authors argue that the constraints of data mining are formalizable. For example, a Naïve Bayes classifier might only accept categorical data. If your dataset is numeric, a "valid" process must include a discretization step.

1. The DM Ontology

The heart of the system is a structured ontology where every operator (e.g., C4.5, Random Sampling, Discretization) is defined by:

• Preconditions: What must be true about the data state (e.g., "numeric attributes exist").
• Effects: How the data state changes (e.g., "attributes are now categorical").
• Heuristics: Estimated impact on speed, accuracy, and comprehensibility.

Fig 1: The prototypical DM process template used by the IDA.

2. Systematic Enumeration & Ranking

By treating DM process composition as an AI planning problem, IDEA can filter out hundreds of thousands of invalid combinations, leaving a manageable set of valid "plans." These plans are then ranked using:

• Compositional Heuristics: Estimating performance based on the combined "scores" of each step.
• Auto-Experimentation: Running the actual code on subsamples of the user's data to observe real performance trends.

Methodology: The Power of Hierarchy

A standout feature of the IDEA prototype is its use of Hierarchical Planning. For specialized domains like target marketing, the system doesn't just start from scratch; it uses established "schemata" (e.g., building a probability estimator and a regression model separately and multiplying their outputs) and then searches for the best specific algorithms to fill those slots.

Table 1: Example of valid DM process plans generated by the IDA.

Experimental Results: Beating the Experts

The authors validated IDEA on 23 UCI datasets and the KDDCUP 1998 "Veterans Association" donation dataset.

Key Findings:

• Ranking Accuracy: For large datasets, the correlation between the IDA's predicted ranking and the actual performance (measured by Spearman's $r_s$) was often above 0.90.
• Profit Optimization: In the KDDCUP 1998 task, most human teams failed to beat a "mail-to-everyone" default strategy. IDEA, by combining logistic regression with neural networks in a non-obvious multi-model pipeline, achieved a 41.23% gain, while the human winner achieved 39.32%.

Table 2: IDEA's performance on KDDCUP 1998 data vs. human benchmarks.

Critical Insight: The Network Effect of Knowledge

Perhaps the most profound takeaway is the concept of Network Externalities. When one specialist adds a new preprocessing technique (like "Dual Scaling") to the ontology, every user of the IDA immediately benefits. The system can now "plan" that technique into pipelines where it fits, effectively sharing expert intuition across the entire organization without direct human consultation.

Conclusion & Future Work

The IDA framework proves that while data mining is complex, the "meta-process" of discovery can be structured and automated. While the current work focuses on linear processes and well-understood operators, it paves the way for a future where IDAs handle recursive feature selection, parameter tuning, and even more complex iterative loops.

Limitations: The system still relies on a manually curated ontology. Future iterations would benefit from automated metadata extraction and the integration of "Automatic Bias Selection" to further refine the search for the perfect model.