Does hierarchical procedural understanding improve proactive AI assistance?

Fixed-granularity guidance hides both progress and context

A proactive assistant watches a user's activity and must decide two things at once: whether to speak now, and what to say. The paper argues that existing training and evaluation data support only part of this. Prior datasets either target detection-based proactive understanding — essentially deciding when to interrupt — or supply procedural guidance at a single fixed granularity.

The authors' stated problem with fixed granularity is informational. Guidance at one level gives limited signal about fine-grained progress, so a model struggles to tell whether a unit of work is finished, and it gives limited signal about broader procedural context, so the model cannot shift between coarse and fine guidance when a user's needs change. The paper's proposed fix is to supervise the same task at three nested levels simultaneously.

Phase, step, and action form a nested hierarchy

The paper defines three guidance levels. A phase is a high-level unit that achieves an intermediate goal through multiple steps. A step is a procedural unit that achieves a specific outcome through multiple actions. An action is an observable fine-grained event that can be guided independently.

These levels are not parallel labels but a parent–child structure. In the model's output format, positive numbered IDs express parent–child relationships, units are numbered in temporal order within a clip, and child numbers restart under each parent. Entries are emitted in phase, step, action order, with inactive levels omitted. This is the mechanism the paper claims lets one model track both fine-grained progress and broader procedural context.

The ProactiveCoach suite pairs training data with a benchmark

The suite has three parts: ProactiveCoach-Instruct for training, ProactiveCoachBench for evaluation, and fine-tuned vision-language models combined with an adaptive guidance system.

The underlying dataset contains 11,119 samples built from 7,232 first-person videos across eight source datasets, split into 10,007 training samples and 1,112 benchmark samples, with samples from the same video kept in the same split. Average sample duration is 7.8 minutes and median duration is 8.8 minutes, ranging from 40 seconds to 18 minutes, for 1,452.1 total hours. The data carry 582,084 guidance targets: 46,074 at phase level, 154,507 at step level, and 381,503 at action level. On average a sample holds 4.14 phases, with 3.35 steps per phase and 2.47 actions per step — the action level is where most supervision volume sits.

Three metrics answer three different questions

Evaluation happens at predefined anchor points split into Response Anchors, where guidance is expected, and Silent Anchors, where the model should stay quiet. On ProactiveCoachBench all ground-truth guidance timestamps serve as Response Anchors, while Silent Anchors are randomly sampled timestamps with no ground-truth guidance from the central 30–70% of each phase, step, or action segment. At every anchor, models see the most recent 10 seconds of video plus the complete ground-truth dialogue history before that point; using ground-truth rather than model-generated history prevents earlier mistakes from cascading.

The three metrics are not interchangeable. sF1 matches generated and ground-truth guidance one-to-one on timing and content, sums semantic similarities of matched pairs, and takes the harmonic mean of sPrecision and sRecall — it measures content and coverage. gF1 looks only at whether the model responded or stayed silent at each anchor, ignoring content entirely, and is the geometric mean of response F1 and silence F1. PQS combines both: a silent anchor scores 1 only if the model stays silent, a response anchor scores by relevance, specificity, actionability, and conciseness judged by GPT-5.2, and the average across anchors is multiplied by 100. Because ProactiveCoachBench has equal numbers of Response and Silent Anchors, an always-silent predictor reaches PQS 50 but gF1 0, which is why the paper reads PQS alongside gF1 rather than alone.

Hierarchical supervision adds up to 9.6 points over fixed granularity

The headline comparison is between hierarchical and fixed-granularity supervision on the same backbones. The paper reports that hierarchical supervision improves overall performance across backbones by up to 9.6 percentage points.

The per-backbone table shows the pattern. Qwen3-VL-4B averages 62.69 at phase level, 58.96 at step level, and 53.24 at action level on ProactiveCoachBench; Qwen3-VL-8B reaches 64.07, 59.37, and 52.99; Qwen3.5-4B reaches 62.94, 59.30, and 53.44; Qwen3.5-9B reaches 61.13, 58.81, and 54.77. The corresponding untrained rows for the same backbones sit far lower — for example Qwen3-VL-4B at 26.65, 27.83, and 27.96. Scores decline as guidance gets finer: every fine-tuned backbone scores highest at phase level and lowest at action level, so the action level remains the hardest of the three.

The same fine-tuned models were also evaluated on the public validation split of EgoProactive at step level only, where they score in the 47–49 average range, showing the training transfers to a benchmark the authors did not construct.

A lightweight router handles guidance-level switches

The adaptive guidance system combines a fine-tuned model with a lightweight guidance router. The point of the router is that the guide model itself is not retrained when the requested guidance level changes.

Across four guidance-level transitions, the system outperforms the in-context adaptation baseline by 57.1 percentage points without additional fine-tuning. This is a comparison against a baseline that adapts through prompting alone, so the gain reflects what the router adds on top of in-context adaptation rather than the value of hierarchical training by itself. The paper frames the result as supporting guidance-level changes without additional fine-tuning of the guide model.

The evidence covers benchmarks, not deployed assistants

The motivating scenarios in the paper — smart glasses recognizing what a user is doing, a humanoid robot monitoring task progress — are framed as future aspirations, not systems that were built or tested here. All reported numbers come from ProactiveCoachBench and the EgoProactive validation split, under a fixed protocol: 10 seconds of preceding video, ground-truth dialogue history, and predefined anchors.

Two protocol choices bound how far the numbers travel. Ground-truth dialogue history is supplied rather than generated, so the evaluation isolates guidance decisions from error accumulation over a long interaction. And the anchor-based scoring samples Silent Anchors from the central 30–70% of each segment, so silence behavior at segment boundaries is not what these scores measure. The paper also notes that PQS alone is misleading on a balanced anchor set, which is why gF1 must be read with it.

The paper's own framing is that the datasets and method are a foundation for future research on proactive AI assistance, not a validated product capability.

The action level is where the remaining gap sits

If one number should drive how readers weigh this paper, it is the level-wise split rather than the headline gain. Hierarchical supervision lifts every backbone, but the ordering is consistent: phase-level averages cluster around 61–64, step-level around 59, and action-level around 53–55. The finest level, which also carries the largest share of guidance targets at 381,503 of 582,084, is where models are weakest.

That makes the practical priority clear for anyone building on this work: the value of hierarchical supervision is largest where coarse guidance already works, and the open problem is whether action-level progress can be tracked as reliably. The 57.1-point router gain addresses a different question — switching between levels on request — and should not be read as closing the action-level gap.

Sources used in this answer