[arXiv 2026] LongNAP: Anticipating the Next Click with Long-Context User Modeling

TL;DR

Language models currently see us through a "narrow keyhole" of prompts. Stanford researchers have introduced Next Action Prediction (NAP)—a task where AI looks at your entire history of screenshots and clicks to guess what you’ll do next. They developed LongNAP, an architecture that reasons about your past to predict your future, outperforming standard fine-tuning by a staggering 79%.

Background: Beyond the Prompt

Most AI assistants are reactive; they wait for you to type an instruction. But a "truly proactive" system needs to know that because you just read a rejection email, you’re likely to hop onto Slack to vent to a collaborator.

The challenge? Context. Real-world behavior is scattered across weeks of interactions. You can't fit 1,800 hours of screen time into a standard Transformer context window. Moreover, models often forget "latent" information—those small details (like a meeting time noticed three days ago) that suddenly become relevant now.

Methodology: NAPsack & LongNAP

1. NAPsack: Annotating the "Un-annotatable"

Progress requires data. The team released NAPsack, a tool that passively records screenshots and I/O events, using Vision-Language Models (VLMs) to turn raw pixels into natural language descriptions (e.g., "Clicked the 'Downloads' folder"). They labeled over 360,000 actions from 20 users over a month of continuous phone use.

Figure: NAPsack compresses raw screen intake by identifying meaningful interaction bursts.

2. The LongNAP Architecture

LongNAP doesn't just predict; it thinks to remember. The process happens in two RL-optimized phases:

• Reasoning to Retrieve: The model observes current screen state and generates a "reasoning trace" (e.g., "User is house hunting"). This trace acts as a semantic query to a BM25 library of past behaviors.
• Reasoning to Predict: It takes the retrieved memories, updates its internal logic, and generates a final sequence of predicted actions.

The system is trained using GRPO (Group Relative Policy Optimization), rewarding the model when its predictions match what the user actually did five minutes later.

Figure: The two-phase reasoning process enables LongNAP to pull relevant history from an unbounded memory.

Experimental Battleground

The researchers compared LongNAP against heavyweights like Gemini 3.0 Flash and SFT (Supervised Fine-Tuning) versions of Qwen-2.5-VL.

Key Results:

• Personalization Wins: On individual users, LongNAP smashed the SFT baseline, raising the LLM-judge similarity score from 0.21 to 0.38 (+79%).
• Generalization: Even when tested on "unseen" users, the model outperformed few-shot RAG by 13% over Gemini, showing it learns general human computer-use patterns.
• Confidence Calibration: When the model is highly confident, its accuracy jumps—26% of trajectories are almost perfectly aligned with the ground truth.

Figure: LongNAP significantly outperforms all prompt-based and fine-tuned baselines.

Critical Insight: Why does it work?

The "Reasoning" part of LongNAP is the secret sauce. Ablation studies showed that removing reasoning traces dropped performance by 19%.

Why? Because reasoning allows the model to compress complex histories into "queries." Instead of searching for "Excel," the model searches for "User's typical Monday morning accounting workflow." As training progresses, these traces actually get shorter and more efficient, evolving into a specialized language between the model and its own memory.

Challenges: Privacy and Alignment

Operating a "Next Action Predictor" is a privacy nightmare. The authors acknowledge this "Privacy Paradox"—users want proactive help but shouldn't have to upload their entire screen history to a central server. The paper advocates for decentralization (on-device training) and redaction models to strip PII before it reaches the reasoning engine.

Moreover, there's the Alignment issue: If you habitually procrastinate on YouTube, do you really want an AI that predicts (and thus facilitates) your next procrastination session?

Conclusion

LongNAP shifts the paradigm from AI as a "Passive Responder" to AI as an "Active Collaborator." By formalizing Next Action Prediction and providing the tools to label longitudinal data, this work lays the foundation for assistants that don't just follow orders—they anticipate needs.

For those interested in the implementation, the authors have released powerNAP, an online version that learns from your interactions in real-time.