One Stroke at a Time: Teaching VLM Agents the Art of Structured Sketching

TL;DR

Researchers have developed a VLM-based agent capable of generating complex vector sketches part-by-part (e.g., drawing a "helmet" then "torso"). By creating the ControlSketch-Part dataset and employing a Multi-turn Process-Reward GRPO strategy, the model moves beyond "black-box" one-shot generation to a transparent, editable, and highly structured creative workflow.

The Problem: Generative "Magic" vs. Designer Control

Current text-to-vector models (like DiffSketcher) operate like a magic wand: you provide a prompt, and the entire sketch appears at once. While visually impressive, this fails on two fronts:

1. Inflexibility: If the "wheels" of a generated car look wrong, you often have to regenerate the whole image.
2. Lack of Intuition: Humans don't sketch by diffused noise; we draw semantically—one part at a time.

The challenge lies in the Data Gap. There are no large-scale datasets that label which SVG paths belong to which semantic part (e.g., "this curve is part of the rabbit's ear").

Methodology: The "Critic" in the Machine

The authors solve the data problem with a VLM-driven annotation pipeline. Instead of manual labeling, they use a high-end VLM (Gemini 3.0 Pro) in a multi-stage process:

• Decomposition: Proposing a list of semantic parts.
• Critique & Refinement: A second VLM pass to fix overlapping or missing parts.
• Diagnostic Visualization: Re-coloring the sketch based on parts to help the VLM "see" its own assignment errors.

Architecture & Training

The agent (based on Qwen-VL) is trained in two stages:

1. Supervised Fine-Tuning (SFT): Learning the "language" of SVG coordinates ($M$ and $C$ commands for Bézier curves).
2. Multi-turn GRPO (Reinforcement Learning): This is the secret sauce. Most RL only rewards the final result. Here, the authors use Process-Rewards, where the agent receives a "score" after every part it draws. This is calculated using DreamSim, a perceptual metric that aligns better with human vision than standard CLIP scores.

The model takes the current canvas, the global prompt, and the specific part instruction to generate the next set of SVG paths.

Experiments: Superior Quality and Control

The model was tested against SketchAgent and general-purpose VLMs like Gemini 3.1 Pro.

Key Findings:

• Visual Fidelity: Unlike the simplistic "doodles" of previous agents, this model produces professional-grade sketches with complex curves.
• Local Editing: Because the model understands "parts," users can prompt it to “replace the wheels” or “remove the wings” without affecting the rest of the sketch.
• Human Preference: In double-blind studies, humans preferred the "Ours (SFT + RL)" results significantly more than holistic generation methods.

Comparison showing how the proposed method maintains structural integrity and semantic relevance across various categories like astronauts, bears, and cars.

Critical Insight: Why Process-Rewards Matter

The core takeaway from this work is the effectiveness of the Process-Reward. In generative agents, "Outcome-based" rewards (only scoring the final image) lead to the "sparse credit" problem—the model doesn't know which specific turn caused a failure. By providing a reward at every step, the authors successfully bridged the gap between training on "perfect" ground-truth data and the "noisy" reality of the model's own intermediate generations.

Conclusion & Limitations

While this is a massive leap for Vector Agentic Workflows, the model still faces challenges with "unfamiliar concepts" and occasional premature stopping (where a part is only half-drawn). Future work involving Chain-of-Thought (CoT) reasoning before each stroke could further enhance the "planning" capabilities of these digital artists.

Takeaway for Practitioners: If you are building generative agents for complex tasks (code, graphics, or layout), don't just reward the final output. Implement a part-level decomposition and reward the process.