[VISA] Value-Knowledge Decomposition: Solving the Alignment Tax in Personalized LLMs

TL;DR

Personalizing an LLM usually breaks it. When you fine-tune a model to adopt a specific "brand voice" or "cultural value," it often starts hallucinating or forgets its core knowledge—a phenomenon known as the Alignment Tax. The VISA framework, proposed by researchers from Peking University, solves this by decoupling the two: it keeps the knowledge base frozen and uses a lightweight "Rewriter" module trained via GRPO to inject values purely as a stylistic and logical layer.

The results are striking: VISA achieves higher factual consistency than GPT-4o while precisely hitting target value profiles across 10 distinct psychological dimensions.

The "Value Drift" Paradox

Why is personalization so hard? The authors identify a fundamental vulnerability: Value Drift. In their foundational experiment, they found that even fine-tuning a model on "neutral" data like medicine or law causes its underlying value system (e.g., its stance on authority or autonomy) to shift unpredictably.

Current solutions like prompting are too shallow and brittle, while standard SFT (Supervised Fine-Tuning) is too invasive, overwriting the "knowledge neurons" of the model with "value neurons."

Figure 1: Visualizing how a model's foundational values drift after being fine-tuned on purely knowledge-centric datasets.

Methodology: The VISA Architecture

VISA (Value Injection via Shielded Adaptation) treats value alignment as a dynamic control problem. Instead of changing the model, it changes the output using three modules:

1. Value Detector ($D_{\psi }$): A regression model that maps any response to a vector in the 10-dimensional Schwartz Theory of Basic Values space (Self-Direction, Security, etc.).
2. Instruction Translator ($T_{\varphi }$): Converts natural language (e.g., "be more traditional") into a $\Delta$ shift in the value vector space.
3. Value Rewriter (${\pi }_{h}eta$): The core engine. It takes the original factual response and the target value vector to generate a "value-injected" version.

Reinforcement Learning via GRPO

To train the Rewriter, the authors utilized Group Relative Policy Optimization (GRPO). Unlike traditional RLHF which requires a complex "Critic" model, GRPO uses group statistics to determine the relative quality of different outputs.

The reward function is the secret sauce:

• $R_{val}$: Rewards high cosine similarity to the target value vector.
• $R_{cons}$: Rewards high bidirectional entailment between the original and rewritten text (ensuring facts aren't lost).

Figure 2: The VISA pipeline decoupling knowledge preservation from value alignment.

Experiments & Results: Beating GPT-4o at its Own Game

The researchers tested VISA against heavyweights like GPT-4o and Gemini-3-Pro using a new benchmark, VCR-45K.

• Semantic Reliability: VISA maintained a consistency score of 0.87, whereas Gemini-3-Pro's consistency plummeted to 0.49 when given complex value-alignment instructions.
• Joint Success Rate (JSR): VISA consistently showed a better trade-off between "hitting the value target" and "keeping the facts straight" across model scales from 0.6B to 8B parameters.

Table 1: Comparison of factual consistency and value alignment precision.

Qualitative Case Study

In a direct comparison (Figure 5 in the paper), when asked to rewrite advice about task prioritization to emphasize "Security" and "Conformity," GPT-4o actually hallucinated new information about "sustainable practices" that wasn't in the original text. VISA, however, kept the exact same advice but reframed the internal logic to satisfy the value requirements.

Beyond Static Targets: Adaptive Value Search

One of the most innovative parts of the paper is Chapter 5, where the authors apply VISA to cases where the target value is unknown. By using a bi-level optimization loop, the model can actually "search" for the best value vector that maximizes domain-specific performance (like Math accuracy) while minimizing drift from a baseline.

Critical Insight & Conclusion

The industry has long struggled with the "Alignment Tax." VISA suggests that the solution isn't to build better monolithic models, but to architecturally decompose the problem. By treating "Values" as a modular adaptation layer rather than a core parameter update, we can deploy models that are both factually bulletproof and highly personalized.

Limitations: The reliance on Schwartz Theory is powerful but limited to human-centric psychological values. Future work could extend this to more abstract ethical frameworks or industry-specific compliance rules.

Final Takeaway: Modular, externalist alignment (Plug-and-Play) is proving to be more stable and efficient than internal parameter fine-tuning for high-precision tasks.