Can curvature-aware influence weighting improve activation steering in language models?

Why averaging activations dilutes a concept

Activation steering assumes a high-level feature such as refusal or toxicity is linearly encoded in activation space, so control reduces to finding a direction and adding it to hidden states with a strength. The standard way to build that direction is a difference in means: average activations over concept examples, average over contrastive non-concept examples, and subtract.

The paper argues this average is a poor target. Averaging over a contrastive dataset pulls in unrelated concepts and noise, and the resulting transport is dominated by a few tokens. The encoded direction then reflects token-level regularities rather than the theme the researcher intended to steer toward. The proposed alternative is to steer toward the examples that most express the concept thematically instead of toward an expectation over all of them.

Influence functions rank examples by curvature

To decide which examples express a concept most strongly, the authors use influence functions. An influence score estimates how much a single data point contributes to the model's representation of a concept. Crucially, the score includes the curvature of the model's loss landscape through a preconditioner, rather than comparing raw activation similarity alone.

The paper's own framing is that this curvature lets influence capture concept-relevant relationships beyond superficial token-level similarity. The authors also test that claim directly: influence-based and activation-based methods rank the same data points significantly differently, which they present as evidence that influence functions see concept-relevant information the activation proxy misses.

The concept direction itself is built from contrastive prompt-completion triples. Each triple pairs one prompt with a concept completion and a non-concept completion, and the per-triple measurement is the length-normalised log-probability difference between the two completions. Each triple's gradient is rescaled to unit norm before averaging, so no single triple dominates through gradient magnitude alone.

From influence scores to weighted transport mass

Influence scores are computed per transformer block over MLP parameters, so a data point can be highly influential at one depth and negligible at another. The authors exploit this by weighting points per layer, effectively steering with a different dataset at each depth rather than one global dataset.

Raw layer-wise scores are heavily tailed, so they are z-scored to give scale invariance across layers, then passed through a softmax with temperature tau. That temperature tunes the weighting between a point mass on the single most influential example as tau approaches zero and a uniform distribution as tau grows, which the authors describe as robustness to influence-score outliers.

The weighted scores then enter optimal transport. The method follows per-dimension affine maps rather than Sinkhorn or linear-programming solvers, keeping inference cheap. Under a convex cost the one-dimensional coupling is monotone, so the transport plan needs only a sort, with complexity O(n log n). The closed-form variant, IF-Act, solves the affine map by least squares under the quantile coupling; the open-form variant, IF-LinEAS, learns the affine parameters jointly by backpropagation. Only the target distribution is weighted in the general formulation, though for concept suppression the authors weight the source distribution instead.

Toxicity suppression gains are largest and most consistent

On Jigsaw toxicity suppression at steering strength lambda equal to 1, the influence-weighted methods produced the lowest toxicity scores in the reported tables. For Gemma-2-2B with all layers steered, LLM-judge toxicity fell from 0.344 unsteered to 0.031 for IF-Act and 0.018 for IF-LinEAS, against 0.083 for Act and 0.095 for LinEAS. The RoBERTa judge showed the same ordering, with IF-Act at 0.001 and IF-LinEAS at 0.009 versus 0.044 for Act and 0.045 for LinEAS.

For Llama-3.2-3B with all layers steered, LLM-judge toxicity fell from 0.268 unsteered to 0.177 for IF-Act and 0.052 for IF-LinEAS, against 0.257 for Act and 0.101 for LinEAS. The paper's discussion states that curvature information substantially helps suppression of the broader concept of toxicity, and the per-layer influence weights support a mechanism: for toxicity the weights concentrate clearly in the middle layers, whereas for OneSec objects and TruthfulQA they stay close to uniform across layers.

IF-Act steers hard but damages model quality

The two influence-weighted variants sit at very different points on the steering-quality trade-off. IF-Act often achieves high scores on truthfulness and OneSec concepts, but the paper reports that this frequently comes at a high cost to MMLU and perplexity. In the OneSec table for Gemma-2-2B with all layers steered, IF-Act reached a concept score of 0.721 against 0.708 for Act, but fluency perplexity rose to 32.27 and WikiText perplexity to 1010.35, and MMLU fell to 0.267 from 0.549 unsteered. For Llama-3.2-3B, IF-Act's WikiText perplexity reached 1.85e5.

IF-LinEAS behaves differently. On the same OneSec setting it reached a concept score of 0.584 versus 0.544 for LinEAS, with fluency perplexity 8.49 versus 8.38, WikiText perplexity 61.55 versus 55.36, and MMLU 0.489 versus 0.499. The authors conclude that IF-LinEAS often improves on LinEAS while largely preserving model quality, giving a more favourable balance between steering performance and capability preservation.

The perplexity pattern differs by objective. Under suppression, perplexity stays roughly constant as steering strength increases; under induction, it rises steadily. The paper attributes the suppression arm's stability largely to its smaller overall change relative to the more volatile induction objective.

Excluding first and last layers trades steering for stability

The layer-exclusion ablation separates headline steering strength from model stability. Not steering on the first and last layers hurt the headline induction and suppression numbers, but it significantly helped stability, giving lower perplexity and higher MMLU scores.

The authors hypothesise that leaving the beginning unsteered gives the model a better chance to understand the text, and that leaving the end unsteered reduces volatility in output tokens, which may aid perplexity and help the model select the right multiple-choice answer. They explicitly leave testing this hypothesis to future work, so it should be read as an interpretation rather than a demonstrated mechanism.

The effect is not uniform across models. In the per-concept OneSec plots, Gemma adapts well to excluding the first and last layers, while Llama largely collapses and mostly fails to induce any concept meaningfully. Object-level results also vary widely within a model: concepts such as church, book, flower and football steer well, while others steer poorly, and Llama's balloon concept fails entirely.

Truthfulness gains depend on the model and pair count

On TruthfulQA the picture is more mixed than for toxicity. With all layers steered and 1024 contrastive pairs, Gemma-2-2B reached MC1 0.255 for IF-Act and 0.203 for IF-LinEAS, against 0.256 for Act and 0.214 for LinEAS, with unsteered MC1 at 0.209. MC2 was 0.482 for IF-Act and 0.356 for IF-LinEAS, against 0.400 for Act and 0.353 for LinEAS.

For Llama-3.2-3B with all layers steered and 1024 pairs, MC1 was 0.264 for IF-Act and 0.253 for IF-LinEAS, against 0.284 for Act and 0.256 for LinEAS, with unsteered MC1 at 0.245. MC2 was 0.474 for IF-Act and 0.401 for IF-LinEAS, against 0.447 for Act and 0.402 for LinEAS.

The paper also varies the number of contrastive pairs between 32 and 1024. The reported delta column shows the change in MC2 from 32 to 1024 pairs: for Gemma IF-Act the change is +0.023 and for IF-LinEAS it is -0.013; for Llama IF-Act it is +0.006 and for IF-LinEAS it is -0.007. More contrastive pairs therefore did not uniformly improve truthfulness, and the direction of the change depended on the method and model.

Where influence weighting pays off and where it does not

The paper's own priority ordering is clear. Curvature-aware influence weighting helps most for the broad concept of toxicity, helps for object-based concepts and truthfulness in most settings, and is best deployed through the open-form IF-LinEAS variant when capability preservation matters. IF-Act is the stronger steerer on truthfulness and OneSec in several settings, but it lands at a less favourable point on the steering-quality trade-off because of its perplexity and MMLU cost.

Two boundaries qualify that reading. First, the authors note that steering strength is currently applied uniformly across all layers even though influence scores concentrate in the middle layers, and they propose using the per-layer influence breakdown to decide where and how much to steer as future work. Second, they suggest that other data-attribution or relevance signals could provide complementary or similarly effective weightings, and that comparing such signals systematically remains open.

For a reader deciding what to take from this paper, the practical split is between objective and variant rather than a single headline number. If the goal is toxicity suppression, influence weighting delivered the lowest toxicity scores in the reported tables for both models. If the goal is concept induction or truthfulness while keeping MMLU and perplexity close to unsteered levels, IF-LinEAS is the variant the paper's evidence supports, and the first and last layers are the ones whose exclusion most clearly buys stability at some cost to steering strength.

Sources used in this answer