Teachers that write weights
Research · Weight-space programming

Large teachers can program small models in closed form

A 27-billion-parameter teacher answers a few hundred unlabelled questions. One line of linear algebra writes those answers into a few hundred columns of a small model's weights. No gradients, no labels, no training loop. The small model then answers new questions better.

The teacher's soft answers flow into a single late layer of the student. Only the highlighted columns change; everything else stays exactly as it was.
TL;DR

We show that a large language model can program a small model's weights directly, through an explicit, closed-form channel. The teacher's answers are translated into a target for the student's internal state and written with one ridge-regression solve into a few hundred columns of a late MLP layer. Edits were selected on validation data, frozen, and scored once on held-out data. They improved 9 of 10 benchmarks spanning general knowledge, reasoning and medicine, by +3.6 points on average and up to +11.6, while changing at most 0.12% of the weights.

+3.6
points mean held-out accuracy gain over ten benchmarks
+11.6
points on PathVQA, the largest single gain
≤0.12%
of the student's 853 million weights changed
0
gradient steps, and no labels used to fit the edit

01Weights are programs. Who gets to write them?

Small open models are everywhere that large ones cannot go: on a laptop, inside a hospital network, on a phone. They are also much less capable.

In our study, Qwen3.5-0.8B answers 44% of questions correctly across ten multiple-choice benchmarks. Its 27-billion-parameter sibling answers 77%. The usual way to close such a gap is fine-tuning or distillation. Both work, but they rewrite millions to billions of weights through a long, opaque optimisation. The result is hard to audit, hard to bound in its side effects and hard to combine with other updates.

Earlier this month, Takehi, Teoh, Zhang and Khattab at MIT CSAIL proposed a striking alternative, which they call weight-space programming. Their idea is to treat a network's weights as a program to be read and written. In one of their demonstrations, a frontier model inspected the same Qwen3.5-0.8B, edited 0.016% of its weights, and lifted MATH-500 from 7.9% to 48.1%. That result inspired this work.

If a large model can write a small model's weights, can it do so through a channel we can write down, inspect and test, so that every change is explained by one equation?

That is the question we set out to answer. We replaced the opaque editor with an explicit, closed-form channel. The teacher contributes only what it knows, its answers. The student's own geometry decides where those answers go. Linear algebra does the writing.

The rest of this post follows one thread. We first explain the recipe and let you try it. We then show that it works on held-out data. Finally we ask four questions about the resulting programs: how small are they, why do they work, could gradient descent have written them, and can several of them share one model?

02The recipe: from a teacher's answers to a weight update

The method has four steps. Click through them, or let them play.

Figure 1. Four steps, no training. The bars in steps 1 and 2 are an illustrative example. Every edit in the study followed exactly these four steps.

The core is step 3. A transformer's MLP layers behave like key–value memories: the input to the down-projection acts as a key, and the matrix maps it to a value that is added to the residual stream. Model-editing methods such as ROME and MEMIT used this view to insert single facts. We use it to insert a skill, namely the teacher's way of answering a whole task.

Let \(K\) stack the student's keys on the teacher's questions, and let \(R\) stack the changes in the residual stream that would move the student's answers toward the teacher's. Let \(C\) be the covariance of keys on everything we want to leave alone. The edit is then a single protected ridge regression:

\[ \Delta W^{\top} = \big(K^{\top}K + \lambda n\, C\big)^{-1} K^{\top} R, \qquad W_{:,S} \leftarrow W_{:,S} + \alpha\, \Delta W. \]

The first term asks the update to produce the teacher's answers. The second asks it to stay silent on everything else. The solution is the best compromise, and it is computed exactly rather than approximated by gradient descent. It touches only a chosen set \(S\) of columns, typically 32 to 512 out of 3,584. We call each such update an edit or, in the spirit of weight-space programming, a task program.

Try it: a protected solve in two dimensions

The equation is easier to trust once you have pushed it around. The playground below shrinks the idea to a 2-D layer. The blue points are the teacher's questions and the rings are where the teacher wants them to go. The grey cloud is everything else the model does. Drag the rings, then switch protection on and off.

Closed-form edit, live
Every frame solves the equation above exactly.
Teacher targets reached
Collateral drift on everything else
task inputsteacher targetprotected inputsdrift
Figure 2. The protection term finds the quiet directions. Without protection, the solve takes the cheapest route to the targets and drags the whole grey cloud along. With protection, it routes the change through the direction in which the protected inputs barely vary, so the targets are reached while almost nothing else moves.

03Does it work on questions it has never seen?

An edit is only interesting if it generalises, so we treated evaluation as strictly as we could. Every configuration choice was made on a validation split: the layer, the number of columns, the temperature and the dose. A validation gate kept an edit only if accuracy rose and the model's behaviour on unrelated text barely changed. The winning edit was then frozen and fingerprinted, and scored exactly once on a held-out test split.

We tested ten benchmarks in two suites. The general suite has MMLU, SuperGPQA, ARC-Challenge, LogiQA 2.0 and GPQA Diamond. The medical suite has MedQA, MedMCQA, PubMedQA and two visual question-answering datasets with images, SLAKE and PathVQA.

Closing part of the gap, without training
Mean accuracy over ten benchmarks, held-out test sets
Figure 3. Held-out gains on both suites. Top: mean test accuracy of the 0.8B student before and after editing, and of its teacher. Bottom: change in test accuracy (percentage points) of the frozen edit for each benchmark, with 95% bootstrap confidence intervals. Starred bars stay significant after correction over all 30 tests, and none of the 30 was a significant loss. Hover over a bar for details. The medical-specialist teacher was evaluated on the medical suite.

With Qwen3.5-27B teaching Qwen3.5-0.8B, the frozen edits raised accuracy on nine of ten benchmarks (Figure 3): +3.2 points on the general suite and +4.0 points on the medical suite. ARC-Challenge rose by +5.7, LogiQA 2.0 by +5.8, and PathVQA, a pathology-image benchmark, by +11.6. The effect replicated with a five-times larger student and with a different teacher, the medical specialist MedGemma-27B. Every one of the four teacher–student pairs improved its suite average.

04A needle in a haystack of 853 million weights

The first question is how small a program is. The edits are small. Zoom in to see how small: pick a benchmark and go from the whole model, to one matrix, to the columns that were actually rewritten. These are the real column indices of the frozen edits.

Figure 4. Programs are tiny. In the whole-model view, each square stands for 25,000 weights, laid out in model order. Lit squares are the weights the edit changed. In the matrix view, each vertical line is one of the 3,584 columns of the last layer's MLP down-projection; edited columns are bright, and taller lines mark larger updates. Across all edits, 0.004–0.12% of the weights changed.

05Why it works: the teacher's knowledge, written next to the readout

The second question is where the gain comes from. A gain after editing could have two sources. It could be the teacher's knowledge, or it could be a generic nudge that, say, makes the model prefer option B. We separated the two by refitting every frozen edit with everything held fixed except the targets.

Same solve, different targets
Mean held-out change, 0.8B student, ten benchmarks
Which mistakes get fixed?
Student errors corrected by the edit, out of 100
24.3%
when the teacher was right
10.1%
when the teacher was wrong
Figure 5. The teacher's signal is what matters. Top: the same solve with the teacher's targets shuffled across questions hurt accuracy, and random updates of the same size did nothing. Bottom: the edit fixed the student's errors 2.4 times as often when the teacher knew the answer. This held on all ten benchmarks.

This is the signature of knowledge transfer. If the edit simply shifted the student's preferences, it would fix errors at the same rate whether or not the teacher knew the answer. Instead the corrections follow the teacher. The edits also move the student's whole answer distribution toward the teacher's, and they halve calibration error, so the student's confidence becomes more honest.

Where the edit is written matters just as much as what is written. When we wrote the same kind of edit at every depth of the network, a clear pattern appeared.

Edits work best next to the readout
Mean validation change (pp) when the edit is written at each layer, with three edit widths. Qwen3.5-4B student; MedQA, PubMedQA and SLAKE.
Figure 6. The last layer wins at every width. Close to the output, the step from a desired change in the answer to a change in the residual stream is almost exactly linear. There the closed form predicts what the edit will do with a cosine of about 0.99 between predicted and realised change. Deeper in the network, later layers reshape the change before it reaches the output. Every selected program was also strikingly simple: 90% of its energy lay in just 3 to 18 directions.

06Closed form beats gradient descent on the same weights

The third question is whether the closed form matters. A natural objection is that any method would do well if it were allowed to change those columns. So we gave gradient descent exactly the same opportunity. It got the same layers, the same columns, the same teacher targets, a learning-rate sweep and access to the same validation set for choosing its checkpoint.

Same weights, two ways of writing them
Mean held-out change (pp) per teacher–student–suite comparison
What else changes?
Median drift on unrelated text (KL from base model, log scale), and peak GPU memory to fit, 0.8B student
Figure 7. Exact beats iterative. Top: the closed-form edit gained more than gradient distillation on the same weights in five of six comparisons, and on 21 of 30 benchmarks. Bottom: it also changed the model's behaviour on unrelated text 6× less than gradient descent on the same columns, and 19× less than LoRA distillation of every MLP layer. It needs barely more memory than inference: 1.8 GB, against 1.6 GB for the model alone.

This shows that the small size of the gains reflects the small size of the update, not a weak way of finding it. LoRA distillation, which rewrites hundreds of millions of weights, can gain more. But it does so with no closed-form description and much larger side effects. Within the same tiny budget, the closed-form solve is the better writer.

07Many programs, one model

The last question is whether programs can be combined. A practical system needs to carry many skills at once. So we added the five general-suite programs into a single 0.8B model. Naively, this fails badly, because each edit tramples the others. The closed form, however, offers a direct remedy. We add the other tasks' unlabelled inputs to the protection covariance \(C\), and each program is then written in directions the other tasks do not use. Toggle it below.

Figure 8. Protection lets programs coexist. Five unprotected edits summed into one model collapse accuracy by 12.2 points. With cross-task protection, the same five programs live in one model with a net +3.0 point gain. That is close to the +3.2 of five separately edited models and to the +3.4 of one update fitted to all five tasks together.

08What this opens up

Weight-space programming suggested that weights can be written rather than trained. Our results add the properties needed before the idea can be trusted where mistakes matter. The channel is explicit: every edit is one equation of the teacher's targets. The gains are tested out of sample under a frozen protocol. The side effects are measured and bounded by construction. Together they suggest a different way of thinking about model adaptation.

Scope
The edits are fitted to multiple-choice readouts and depend on the answer format they were fitted on. Part of their effect corrects the student's answer priors. Gains are moderate on average and largest when the student has room to grow. The paper reports all of this in detail.

Until recently, weights were something we could only train. These results show that a useful part of them can be written by one equation, from a teacher's answers, and then checked like any other program.

Common questions

Is this just distillation?

It uses distillation's signal, the teacher's soft answers, but not its machinery. There is no loss to minimise iteratively and no backward pass through the student. The update has one analytic expression, and it touches a few hundred columns of one or two layers.

How is it different from model editing such as ROME or MEMIT?

Those methods insert individual facts from hand-specified subject–object pairs. Here the target is a teacher's behaviour on a whole task, learned from unlabelled inputs, and the edits are selected and evaluated as task-level programs on held-out benchmarks.

Could the gains come from tuning on the test set?

No. Every choice was made on a separate validation split, and the chosen edit was frozen and fingerprinted before the test split was touched. Each edit was scored on test exactly once, and significance was corrected over all 30 tests.

How expensive is it?

One forward pass of the teacher and the student per input, plus a small linear solve. Refitting an edit takes minutes on one GPU, at a peak memory barely above running the model.

Citation

This work was motivated by, and builds on, Sparks of Weight-Space Programming by Takehi, Teoh, Zhang and Khattab (MIT CSAIL). If you find it useful, please cite:

@article{imam2026teachers, title = {Large teacher models can program the weights of small models in closed form}, author = {Imam, Raza}, journal = {Preprint}, year = {2026} }