Introspection Fine-Tuning

Overview

Over about four days, as the capstone project for CAMBRIA, my group set out to replicate Introspection Fine-Tuning (IFT): Training Small LLMs to Introspect on 1B-8B Llama models.

The two tasks used in this paper were sentence localization: which of N sentences was steered, and strength comparison: which of N sentences received the strongest steer. We successfully reproduced their main results: small Instruct models (~2-8B params) introspect reliably above chance; introspective ability seems to be a function, at least in part, of model scale; IFT, "supervised fine-tuning on sentence-localization examples constructed from the model's own perturbed forward passes," drastically improves performance on both sentence localization and generalizes to the held-out task of strength comparison.

However, we found their claim of "negligible degradation on standard capability benchmarks" to be very fragile under the IFT training regime. It's heavily predicated on early-stoppage in training and a particular learning-rate initialization and schedule. Many trails we ran actually saw massive hits to performance on MMLU. This fragility made it seem to me that (1) the capability preservation they found isn't something inherent to this training regime, rather a byproduct of careful optimization, and that (2) there are probably better finetuning methods out there.

This isn't a huge deal necessarily, but there is a non-trivial capability decrease either way. As a result, we tried to see if there are different finetuning methods that don't face this same tradeoff.

We tried including an implicit regularization term in the objective. Specifically, a KL-divergence "anchor" similar to the term you'd find in a DPO training objective function. This seemed to improve capability maintenance through training, without inhibiting the fine-tuned model's introspection gains.

We also explored a different training objective, which was loosely based on optimizing a natural-language report of the "introspection," as opposed to optimizing for single digit answers to "which sentence was perturbed?" This we've called delta-curve supervised finetuning, DC-SFT.

All of the below numbers are from our tests on Llama-3.2-1B-Instruct. These are rough preliminary findings with large error bars, so everything here is naturally to be taken with a (pretty big) grain of salt.

Interesting findings from the replication

As noted above, the headline findings replicated well, but the capability claims were fragile. The most interesting piece of this, though, was that performing IFT on Gaussian noise injections (instead of concept vectors) results not just in at-chance localization performance but also in complete capability degradation. Localization sat at 10%, while MMLU performance collapsed from 49% to 28%. This makes it seem to me like there might be something inherent in the IFT training objective that itself is counterbalancing things the model has already learned, even if it's not being replaced with a new capability of introspective.

Adding the KL anchor

I'd argue that capability preservation should be an inherent property of the objective, not a byproduct of careful optimization. This is especially true if we want to accelerate frontier safety efforts and move past the "safety-capability tradeoff."

So here, we first simply took the original IFT objective and added the penalty term. We specifically wanted to compare this to instances in which capabilities dropped under plain IFT, so we trained under a regime that collapsed MMLU performance (6 epochs, sustained high LR).

We did a sweep across different regularization strengths, ranging λ = 0.0 to 2.0. Across all strengths, localization performance remained about the same (hovering ~70%) while capability retention increased monotonically with higher coefficients. We saw a ~21.1 point loss in MMLU performance when λ = 0.0 and just a ~1.5 point loss when λ = 2.0. This makes sense, but paired with equivalent localization performance indicates that the introspection itself has no direct bearing on capability.

I think that this might point to the fact that this introspective ability already lives somewhere in the weights of these models and adding this regularizer forces the adapter weights to just surface this, rather than trying to construct it from scratch. There's intuitive reason to think that the latter (constructing from scratch) might have much steeper gradients at first on the non-regularized loss landscape, while the regularized, since it is penalized for straying from the original model, has to learn something much more efficient and "like" the reference model.

DC-SFT

IFT trains the model to emit a single digit and takes cross-entropy over the ten digit logits at the final position. This seems to discard a lot of what a model could say about its own state, and a single numeric label can't express degrees of confidence or many other generally introspection-relevant information.

DC-SFT replaces the target with a broader natural-language report, something like "Sentence: 4, layer: 7/16, injected concept: apple." This is then optimized via normal token-level cross-entropy.

Around that sit a few other changes: injection coefficients sampled continuously (log-uniform 2–20) rather than from a fixed set, four injection layers held out of training entirely, and a per-example curriculum weight derived from how much the injection actually moved the residual stream, which also relabels injections too weak to be readable as negatives.

DC-SFT loses only on the metric the other directly optimizes, and even then, of the six layers the eval measures, the paper trained on all six and we only trained on two, one of which is the last layer (a layer that saw no meaningful injections—an issue discussed below). This seems to me as evidence for greater generalization under DC-SFT. DC-SFT also performs much better on anything requiring a graded sense of magnitude.

There also doesn't seem to be an inherent capability tradeoff with DC-SFT, and adding the KL anchor on top recovers (the little) capability decrease: 49.2 base → 46.4 DC-SFT → 48.6 anchored DC-SFT.

Mini J-lens analysis

We also fitted some Jacobian lenses. We validated our general lens-fitting pipeline by comparing it to published/public lenses (Qwen 3-1.7B and Llama-3.1-8B lenses). The lenses we fit for these models were nearly identical to these published versions.

Something interesting we discovered with this was that the two objectives seem to build their answer differently across layers. The IFT objective had a sharp increase in the logits associated with the correct sentence number at layer 12 (chance before then). DC-SFT had a similar jump at layer 12, but then more-so continued to ramp up through layer 14. Adding the KL anchor didn't change either shape, so it seems to indicate something about what kind of internal object is built/recovered through training: a discrete classifier vs. a graded magnitude estimator.

Issues, confounds, and final thoughts

Reviewing the codebase afterward turned up a fairly important confound to recognize. The steering vectors were injected with fixed norms, with their coefficients sampled uniformly across layers. However, because the residual stream is constantly being added to as you progress through the network, its norm grows with depth. As such, a steering coefficient that creates a large perturbation at layer 3 is a tiny blip at layer 14. The plausible confound is that the models might have just been learning that they haven't been steered as a function of layer depth as opposed to having been injected with something foreign.

(There's some additional pulling apart that we need to do here, too, namely that an injection that's small relative to the magnitude of the residual stream is, in a way, not really steering the model. The injected concept is so faint that it probably isn't having any meaningful effect on the model's internal representations. That said, ensuing results seem to indicate the model is learning something unintended as a function of layer depth.)

We observed the model abstained at extremely higher rates in later layers—24/24 trials in layers 9 and 12. The steering coefficients need to be scaled by the residual norm at the injection layer.

Another issue I found was that negative targets ("I do not detect an injected thought") are only about ⅓ the length of positive targets ("Sentence: 4, layer: 7/16, injected concept: apple."). Further, every negative example pushes towards the same ~six tokens. This biases the optimization to cheaply navigate towards always abstaining. Abstention isn't the worst thing, in fact, it's good that a model doesn't try to confabulate as much as other introspection studies might, but this isn't quite what we intended.

Also, just for any of these observations to really hold, we'd need to make these tests much more robust (more concepts, more seeds, more/bigger models, etc.).

To me, what we've taught the model here (even once the above issues are fixed), isn't introspection in any proper higher-order or metacognitive sense. It's a somewhat obtuse perturbation detection. If it is really looking back on its weights / is causally dependent on reflectively observing this "delta" in the residual stream curve, that's still cool, though, and could be an indication of "privileged access." (But I don't think we've sufficiently shown this, either.)

Project status

All code, adapters, and trained weights are private for now—want to clean some stuff up and fix these mistakes before making any of this public.