Can multi-session solve-review-wiki-verify workflows make olympiad agents more reliable?

Three failure modes a single session cannot fix

The paper starts from a diagnosis rather than a benchmark gap. In a single long agent session, a plausible but false intermediate claim can contaminate everything that follows; more reasoning cannot supply a theorem or rule the model never learned; and an agent can misread a requirement or weaken the goal, proving a nearby statement or testing convenient cases, then appear to succeed. The authors call this last pattern problem misunderstanding and reward hacking.

These three failures need different remedies, which is why HOA uses three feedback channels rather than one stronger model. Semantic review asks whether the artifact fully answers the original problem. The wiki supplies missing facts or methods. The verifier tests a machine-evaluable condition. The paper states explicitly that these signals are complementary rather than interchangeable, so the design question is which gate catches which failure.

The reviewer judges a frozen artifact, not the worker's reasoning

HOA is a flow-level framework built on Humanize: it coordinates multiple agent sessions and tools rather than making one API call or running one command-line agent. In each round the worker produces or revises a candidate artifact. An independent reviewer then evaluates that frozen candidate against the original problem and the acceptance plan, and returns specific findings to the worker for the next round. The reviewer does not see the worker's hidden reasoning or conversation history, and the candidate stays read-only during review.

That separation is the mechanism, not a stylistic choice. Because the worker cannot accept its own output or bypass review, and because the reviewer cannot silently repair the artifact, the loop forces each reported issue to be addressed in a new worker round. The loop ends when the reviewer declares the candidate complete or the run hits its resource limits. The worker may call the verifier for immediate machine feedback, but verification and semantic review remain separate gates.

The wiki returns verified lemmas, not prose hints

The mathematics and chemistry wikis are built differently because the domains need different reusable knowledge. After a mathematical problem is solved and accepted, a curation agent extracts the main theorem and reusable intermediate lemmas from the Lean development; each record carries declaration names, Lean signatures, a short description, the source problem, required imports, source locations and verification status. The chemistry wiki is assembled before evaluation from textbooks, olympiad problems outside the test set and online references, organized into concept-centered pages with recorded sources. No separate wikis were built for physics or coding, because the authors judged that IPhO and IOI knowledge is largely already in model pretraining.

A concrete trajectory shows why the format matters. In the G2 tensor-square decomposition run, the agent retrieved three records that had already passed Comparator and independent review, formalizing a 14-dimensional simple G2 quotient, a 64-dimensional irreducible highest-weight module, and the classification of fourteen irreducible tensor-square types. The agent applied these Lean declarations directly when assembling the final theorem instead of reproving them, and the artifact passed Comparator plus both the Nanoda and default Lean kernels. The paper describes this as the wiki acting as an executable interface rather than an unstructured hint.

The verifier checks the statement, the reviewer checks the intent

For mathematics and formalized physics, the verifier is Comparator. It builds the untrusted Lean submission in a sandbox, confirms that the submitted theorem matches the trusted challenge statement, restricts permitted axioms, and validates the exported proof with the Lean kernel. The axiom restriction is not incidental: the paper notes that a proof can compile while using a disallowed axiom.

The division of labor is explicit. The reviewer is an agent inside the loop that assesses the artifact semantically, flags reward hacking or problem misunderstanding, and decides whether another revision is needed; the verifier is an objective task-specific tool that both worker and reviewer can call. Comparator establishes that the stated theorem is proven, not that the informal problem was translated faithfully. Physics problems are formalized before solving through the Science Mango pipeline, which builds a blueprint separating physical laws, given values, earlier results and the claim to prove, then writes typed Lean declarations with the target equation withheld; a physics reviewer rejects missing laws, omitted figure parameters, tautologies and statements that assume the answer before the statement is frozen.

Most of the gain arrives with the review loop

The ablation on the first 50 PutnamBench problems separates the contributions. A pure API call solves 1 of 50 for Kimi-K3 and 3 of 50 for GPT-5.6 Sol. Adding the Codex command-line harness moves those to 4 and 46. Adding the HOA wiki and verifier gives 21 and 49. Adding the full Humanize loop gives 47 and 50.

The ordering matters for interpretation. For the stronger model, most of the improvement comes from the tool-level harness, and the flow-level loop closes the last few problems. For the weaker open model, the wiki and verifier contribute a large step and the review loop contributes the largest single jump, from 21 to 47. The gap between the two models narrows from 42 problems at the Codex /Goal level to 3 problems with the full loop, which is the paper's evidence that systematic feedback can bridge substantial differences in base-model capability.

Extra hours convert into verified solutions only with review

The Lean-Eval comparison tests whether longer runs help. On the first 100 eligible problems in alphabetical order, with every failure retained in the denominator, pure Codex with /Goal reaches 34 of 100 and plateaus after roughly 30 hours. HOA keeps recovering verified solutions and reaches 63 of 100 over 150 hours of worker time. The authors describe this as another scaling dimension: sustained agent time, converted into verified solutions by the review and revision loop rather than by repeating the same failing method.

The headline leaderboard number is a different measurement and should not be conflated with that 100-problem subset. On the full Lean-Eval set, HOA solves 251 of 304 problems (82.6%) and ranks first, improving GPT-5.6 from 112 of 304. The paper's own framing is that workflow design is a major source of capability rather than a thin layer around the base model.

An open model with HOA matches or beats a closed baseline

With GPT-5.6 Sol, HOA reports perfect scores on the IMO, IPhO and IOI 2026, near-full marks on the IChO 2026 problem set, and all 672 PutnamBench problems solved. With the open Kimi-K3, HOA reaches full scores on the IMO, IPhO and IOI and a gold-level 97.1% on the IChO. The paper states that these results closely match the closed-source GPT-5.6 Sol + /Goal baseline on the IMO and IPhO while exceeding it on the IChO and IOI.

The comparison is between workflows, not just models, so it does not isolate model quality. The limitations section says as much: workflow comparisons may be affected by differences in inference time, token budgets, tool access and stopping behavior, and longer multi-session runs consume more computation than a single-agent baseline. The reported olympiad and PutnamBench results also use closed model snapshots whose training data is not fully documented, so benchmark contamination cannot be ruled out.

The guarantee stops at the formalized statement

The most useful boundary is the verifier's scope. Comparator confirms agreement with trusted Lean statements, but that guarantee depends on the correctness and fidelity of those formalizations; it does not by itself establish that an informal problem was translated correctly. For chemistry, physics and programming, automated checks cover only the properties encoded by the evaluator. The evaluation also spans a finite set of olympiad, programming and formal-mathematics benchmarks, so it does not establish generalization to arbitrary scientific research or to domains without reliable task-specific verification, and the experiments do not isolate every component interaction or fully characterize reviewer false positives and false acceptances.

Given those limits, the paper's priority is specific: the review loop, not the wiki or the verifier alone, is what turns additional agent time into verified solutions, and the largest measured benefit appears where the base model is weakest. On PutnamBench, the full loop moved Kimi-K3 from 21 to 47 of 50 while moving GPT-5.6 Sol only from 49 to 50. A reader deciding whether this workflow is worth its extra computation should therefore ask first how much headroom the base model still has on the target task, and second whether a trusted formal statement exists to verify against.

Sources used in this answer