What the harness actually automates
The paper's question is whether an agent harness can autonomously improve a general-purpose LLM on a target task under a limited compute budget. ImproveAnyTask answers by organizing task adaptation as an iterative optimization loop rather than a single training run. The user supplies only a task-level request, for example "Autonomously optimize the model for MMLU-Pro"; the harness then handles diagnosis, strategy choice, data construction, training configuration, and re-evaluation.
The design borrows its structure from gradient-based parameter optimization. Error signals play the role of gradients, candidate strategies play the role of update directions, and actual training runs play the role of parameter updates. This analogy is the paper's framing device, not a claim that the harness computes gradients; the loop operates on data and training configurations instead of on parameters directly.
Error attribution picks one focal problem
The first stage, Error Attribution, combines metric-level analysis with individual failure cases to identify a single focal problem. Metric-level analysis shows which scores are weak; case-level inspection shows what the model is actually doing wrong. The harness uses both to choose one priority rather than optimizing everything at once.
The ablation makes this concrete. Removing Error Attribution replaces the joint metric-and-case procedure and explicit focal-problem selection with an unstructured summary of evaluation results that guides the next action. On MATH-500, the full harness changed its supervision and then selected reasoning trajectories, whereas the variant without Error Attribution moved from general mathematical training to two adapter-scaling adjustments. The paper presents these recorded paths as context for the score comparisons, not as proof that any single decision caused a particular score change.
Update direction weighs gains against reproduction difficulty
The second stage, Update Direction, has the agent investigate multiple research-backed strategies and compare their reported gains, applicability, and reproduction difficulty. This is the paper's main departure from pipelines that only automate data construction or training execution: a method that reports a large gain is not automatically selected if it depends on unavailable resources or fails to reproduce.
The ablation again isolates the mechanism. Without Update Direction, the agent performs ordinary research and adopts a feasible method, and it may still inspect resources or reconsider its choice, but the required candidate comparison, reproduction-difficulty assessment, resource verification, and outcome-aware reuse are removed. On MATH-500, the variant without Update Direction cycled between final-answer supervision, a large mathematical instruction collection, and long-form reasoning data. The paper notes that a strategy change is not itself a failure; the comparison concerns how each variant used the available budget.
Executable updates check small before training fully
The third stage, Executable Model Updates, translates the selected strategy into training data and a training configuration. Before full post-training, the harness runs small-scale execution checks and short training runs to surface implementation problems. Removing this Execution-Guided Refinement leaves ordinary execution and debugging in the same stage, so repairs remain possible but only after problems are noticed.
The recorded runs show what those checks catch. In the TAU2-Bench variant without Execution-Guided Refinement, the agent encountered a training-data simulator unable to invoke user-side tools and only partially repaired deduplication and schema issues. In LiveCodeBench, the same variant proceeded with unresolved thinking-template and termination checks. The paper is explicit that these records help interpret outcomes but do not establish that any one implementation issue alone caused a particular score change.
Mean gains differ between Base and Instruct models
The headline result spans 11 tasks across five capability domains, evaluated on Qwen3.5-4B-Base and Qwen3.5-4B-Instruct under a 24-hour budget with resources equivalent to eight H20 GPUs. Mean gains were 18.29 percentage points on the Base model and 11.97 points on the Instruct model, with a maximum gain of 41.96 points.
These two numbers are not interchangeable. They come from different model initializations, and the paper reports them as separate means over the same task set. The gap is consistent with the Instruct model starting from a stronger task-specific baseline, leaving less headroom, but the packet does not state that explanation, so the difference should be read as a reported contrast between initializations rather than a demonstrated cause.
The evaluation also uses fixed splits rather than complete public benchmark test sets. Roughly 85% of each task is allocated to testing, rounded down, with TAU2 split by domain and BFCL by category. Optimization uses validation feedback only, and reported scores come from the held-out test split.
Small test sets make single points fragile
Task sizes vary enormously, and that changes how much weight a single result can carry. MMLU-Pro uses 10,227 test examples and SuperGPQA uses 22,549, but GPQA-Diamond uses 168, MATH-500 uses 425, and AIME 2025 uses only 25 test questions with one greedy response each.
The paper states the consequence directly for AIME: one additional correct answer changes a single-run score by four percentage points, or the two-run mean by two points. A 41.96-point maximum gain and a four-point swing on a 25-question set therefore describe very different levels of measurement stability, even though both are expressed in percentage points.
Metrics also differ by task. MMLU-Pro, SuperGPQA, GPQA-Diamond, MATH-500, and AIME report accuracy; IFEval reports strict prompt-level accuracy; IFBench reports prompt-level accuracy; LiveCodeBench v6 reports Pass@1 with one response per problem; Terminal-Bench 2.0 reports task success rate; TAU2-Bench averages Airline and Telecom scores over 42 and 96 tasks; and BFCL-V4 uses a renormalized no-Web score. The BFCL no-Web metric removes the 20% Web Search weight and renormalizes the rest, giving 12.5% each to Non-Live, Live, and Irrelevance, 37.5% to Multi-Turn, and 25% to Memory. The paper warns this subset metric is not directly comparable to the official full-task Overall score.
Validated strategies are reused across rounds
Re-evaluation determines the outcome of each update and informs subsequent trials. Effective strategies and scripts are retained for reuse, while unsuccessful attempts provide evidence for revising future decisions. The extended IFEval Base experiment illustrates the checkpoint ancestry: C1 starts from the Base model with persona-conditioned constraint tuning and 29,900 exported examples, C2 adds rubric-guided responses for 55,998, C3 builds a multi-source mixture for 69,900, and C5 starts from C3 rather than the unsuccessful C4 checkpoint, revising C4's data strategy while keeping the earlier model.
The C1 through C8 labels index candidate checkpoints, not eight accepted updates. The paper also notes that the executable instruction checks validate specified constraints; they do not independently establish factual correctness or the quality of a reasoning trace. Data counts are specific to the model initialization used in that run and are not reused as descriptions of the ablation experiments.
Where the evidence stops
The harness restricts executable updates to supervised post-training; extending it to reinforcement learning and broader post-training objectives is left to future work. The paper also does not systematically measure whether specialization affects general capabilities, citing the substantial evaluation effort and the many combinations of tasks, models, and training strategies. The substantial task-specific gains support applications that need focused model adaptation without repeated manual experimentation, and future work will study how to preserve general capabilities during this process.
Several boundaries qualify the results. Validation feedback is isolated from final testing, but repeated use of validation feedback can still lead to validation-set overfitting, and exact-overlap auditing does not rule out semantic contamination in external data. Higher task scores do not establish safety or fairness. Autonomous code execution and external API calls introduce privacy, security, and resource-use risks. The optimization agent's API temperature was set to 0, yet external resource availability, API behavior, and training randomness can still affect the full search trajectory.
The most decision-relevant reading is a comparison, not a single number. The harness is most clearly supported where a task has a large held-out test set and a metric that is stable under single responses, and least clearly supported where the test set is small enough that one correct answer moves the score by four points. The Base-versus-Instruct gap of 18.29 versus 11.97 percentage points is the paper's own signal that starting point matters as much as the harness, and the unmeasured effect on general capabilities is the boundary a reader should carry into any deployment decision.
Sources used in this answer
ImproveAnyTask: An Autonomous Post-Training Harness for Iterative Model Self-Improvement
Across 11 tasks, ImproveAnyTask achieves mean gains of 18.29 and 11.97 percentage points on the Base and Instruct models, respectively, with a maximum gain of 41.96 points, under a 24-hour budget with resources equivalent to eight H20 GPUs.