[ConstraintBench] Can LLMs Solve NP-Hard Problems Without a Solver?

Executive Summary

TL;DR: Large Language Models (LLMs) are often touted as universal problem solvers, but when it comes to the rigorous world of Operations Research (OR), they remain "near-sighted." ConstraintBench reveals that while frontier models like GPT-5.2 and Claude 4.6 can find decent solutions, they struggle to even satisfy basic constraints in complex environments, particularly in temporal and high-interaction domains like crew assignment.

Positioning: This work moves beyond "coding" benchmarks (where LLMs just write Python for Gurobi) to test intrinsic constraint reasoning. It is a high-bar diagnostic tool that identifies a fundamental gap in how autoregressive models navigate feasible regions.

Problem & Motivation: The Gap Between Coding and Solving

Traditionally, the AI community has evaluated LLMs on optimization by asking them to write code (e.g., NL4Opt, IndustryOR). If the code runs correctly in a solver, the LLM "wins." However, this ignores a critical question: Does the model actually understand the constraints?

Real-world decision-making often happens in low-latency environments or within agentic loops where a solver might not be available. Current models often hallucinate entities or ignore "long-chain" dependencies (like time windows in routing), suggesting that their "reasoning" is often a surface-level heuristic rather than a systematic search.

Methodology: Solver-Grounded Benchmarking

The core innovation of ConstraintBench is the use of the Gurobi Optimizer not just as a reference, but as a filter.

The Generation Pipeline

1. Seed Generation: Using a combinatorial product of industries (e.g., E-commerce) and scales to create thousands of unique prompts.
2. Agentic Scenario Creation: An LLM agent builds a scenario, but a Gurobi-based verifier checks it. If it's infeasible, the agent iterates until a valid, solvable problem is created.
3. Formal Verification: Every model response is checked by a deterministic verifier that re-calculates the objective from scratch—never trusting the LLM's self-reported numbers.

Figure 1: The 10 domains covered by ConstraintBench, spanning Binary, Integer, and Continuous variables.

Experimental Insights: Feasibility is the Ceiling

The researchers evaluated six frontier models, including GPT-5.2-pro and Claude Opus 4.6. The results were sobering.

• The Feasibility Wall: In the "Crew Assignment" domain, average feasibility was a nearly non-existent 0.8%. Models simply cannot grasp the dense interactions of workload balance and qualification constraints.
• The Feasibility-Optimality Decoupling: In "Facility Location," models were 85% feasible but 0% optimal. They could open facilities and assign customers correctly, but they couldn't find the "sweet spot" that minimizes fixed and variable costs simultaneously.

Figure 2: Aggregate performance across 200 tasks. Note the massive gap between feasibility and joint optimality.

Systematic Failure Modes

The authors categorized failures into four groups:

1. Structural Misunderstanding: Thinking a 4-hour task can fit in a 2-hour window (common in Project Planning).
2. Entity Hallucination: Assigning items to "Bin B-99" when only bins B-1 to B-5 exist.
3. Specific Reasoning Failures: Ignoring time windows in vehicle routing.
4. Near-Misses: Exceeding a budget by a tiny margin.

Figure 3: Distribution of failure modes across the benchmark.

Critical Analysis & Conclusion

Takeaway: ConstraintBench proves that LLMs are currently "heuristic engines." They are excellent at finding "good enough" solutions (89-96% optimality among feasible tasks) but fail at the rigorous "all-or-nothing" nature of formal constraints.

Limitations: The 0.1% optimality threshold is extremely strict. If we relaxed it to 5%, models like GPT-5.2-pro would look much better in domains like Portfolio Optimization. However, in OR, 0.1% can represent millions of dollars in savings, justifying the strictness.

Future Outlook: The Haladir team plans to expand this to 100 domains and use it as a training environment. By using Gurobi's feedback as a reward signal in Reinforcement Learning, we might finally move LLMs from "broadly intelligent" to "precisely optimal."