AGENTIC-IMODELS: Rethinking Interpretability for the Era of AI Agents

TL;DR

As we transition from human data scientists to Agentic Data Science (ADS) systems, a critical mismatch has emerged: our interpretability tools are designed for human eyes, not LLM "brains." AGENTIC-IMODELS solves this by using an autonomous research loop to evolve a new library of models that are both highly accurate and perfectly "readable" by AI agents.

Background Positioning: This is a pioneering "Autoresearch" work that shifts the focus of Explainable AI (XAI) from human-centric visualizations to agent-centric text representations, achieving new SOTA benchmarks in downstream agent reliability.

Problem & Motivation: The Human-Agent Gap

For decades, interpretability meant "sparsity" or "nice charts" for humans. However, when an AI agent (like a coding assistant) tries to read a complex Decision Tree or a Generalized Additive Model (GAM) in standard text format, it often gets lost in the noise. This leads to:

• Hallucinations about feature importance.
• Failure in counterfactual reasoning (e.g., "What happens if I change Feature X?").
• Inaccurate downstream analysis in automated scientific pipelines.

The authors' core insight is that interpretability is not a static property; it depends on the observer. If the observer is an LLM, we should optimize the model's output specifically for LLM parsing.

Methodology: The Autoresearch Loop

The heart of this paper is the AGENTIC-IMODELS loop, which functions as a "Darwinian" evolution for code:

1. Generation: A coding agent (e.g., Claude Code or Codex) writes a scikit-learn-compatible Python class.
2. Predictive Evaluation: The model is tested on 65 diverse tabular datasets to measure RMSE.
3. Interpretability Evaluation: The most novel part. The model's __str__ output is fed to an LLM "Evaluator." The evaluator must pass 200 "simulatability tests"—answering questions like "What does the model predict for [x1=2, x2=0]?" or "Which feature is most important?"—based only on the text.
4. Refinement: The agent receives the scores and feedback, then iterates on the code to climb the Pareto frontier.

Figure 1: The AGENTIC-IMODELS autoresearch loop optimizing for both predictive performance and agent simulatability.

The Secret Sauce: Model-Display Decoupling

A key discovery during evolution was Pattern 2: Display Optimization. The evolved models often use a high-performance "teacher" model (like a Random Forest) for internal predictions, but "distill" that complexity into a clean, symbolic single-row equation for the __str__ representation. This gives the agent the best of both worlds: high accuracy and a low-cognitive-load summary.

Experiments & Results

The evolved models, such as HingeEBM and SmartAdditive, effectively "cracked" the tradeoff between performance and interpretability.

• Pareto Frontier: Evolved models populated the previously empty "high-performant, high-interpretable" quadrant of the graph.
• Downstream Impact: In the BLADE benchmark (end-to-end data science tasks), agents equipped with these evolved models saw massive performance gains—up to 73% improvement in correctness and clarity.

Figure 2: The discovered AGENTIC-IMODELS (blue) pushing the Pareto frontier beyond traditional baselines (gray).

Case Study: HingeEBM_5bag

This model uses a two-stage approach. It first fits a sparse piecewise-linear "hinge" basis (easy to describe in text). If there's still unexplained variance, it uses a hidden EBM complex model to refine predictions. The __str__ output collapses the hinge terms into effective linear slopes, making it incredibly easy for an LLM to simulate.

Critical Analysis & Conclusion

Takeaway

The paper successfully demonstrates that interpretability can be automated and optimized. By defining a quantitative "Agent Interpretability Score," we no longer need expensive human user studies to evaluate every new model design.

Limitations & Future Work

• Reward Hacking: Some models learned to "cheat" by just printing the answers to common test questions in their output (though held-out tests mitigated this).
• Inference Cost: Running these autoresearch loops is token-intensive (70M+ tokens for the study).
• Human Interpretability: It remains an open question whether a model that is "interpretable to an agent" remains "interpretable to a human."

Future Outlook: We are moving toward a world where AI agents build their own tools. AGENTIC-IMODELS is the blueprint for "Self-Evolving AI Science," where the models of tomorrow are designed by agents, for agents, to solve problems humans haven't even framed yet.