One Way Not to Light a Light Bulb

October 14, 20267 min read

Research

The last post ended with a costume. Tell a small model you are a wise elder and it changes its manners, not its mind. A five-word instruction, "think step by step", beat nine elaborate descriptions of who to be. That left an obvious next question. A costume comes off at the end of the evening. What if you raised the model in the role instead? Let it answer thousands of problems as a critic, train it on those answers, and then take the prompt away. Does the critic stay a critic with nobody telling it to be one? And if it does, is it any better at thinking? This is that experiment. The short version: the role stays, and the thinking doesn't improve. The longer version has a bug in the training data, a second run to fix it, and a result I'm happy to call a failure.

---

The setup

One small model, Qwen3.5-0.8B, on one RTX 4060. Six roles, each a system prompt of about ninety words:

Role	What the prompt asks for
inventor	lay out several different methods, then pick the best
critic	check every step, verify the answer a second way, hunt for your own errors
doer	the shortest sure path, no detours, stop when done
debater	build the strongest case against your answer, then decide
gardener	ninety words about tomatoes, a library and tea. The pebble from last time
step	"Think step by step before answering."

The recipe for each role:

1. The base model answers 2,000 maths problems from GSM8K with the role's prompt.
2. A small adapter (LoRA, a thin layer of extra weights on top of the frozen model) is trained on those answers with the prompt removed. The model learns to produce the role's answers when nobody asks for the role.
3. The raised model sits a test of 300 new maths questions, plus science questions from ARC and the XSTest safety set, with no prompt at all.

Every role is trained three times with different random seeds, so one lucky run can't carry the result. Every role trains on the same problems and the same number of examples, so the only difference is how the answers were written. Before the run I wrote down one question and froze it: does the raised critic beat the raised gardener on maths? The gardener is the control. It has a prompt of the same length, with nothing in it about thinking. If the critic wins, the checking helped. If they tie, the prompt length did whatever happened. My prediction, in my own words at the time: "I think we don't see much difference cause the sample size is too small to see a difference... I am a pessimist for this working."

Run one: the critic wins, sort of

The full run survived a power cut and an out-of-memory crash. Both times it resumed where it stopped. The frozen question came back: critic minus gardener: +0.06 [+0.02, +0.10] Six points, on 300 questions, averaged over three seeds, with the confidence interval clear of zero. A small pilot of 50 questions had said +0.12 with an interval touching zero. So the larger run shrank the effect and made it real. My reading at the time: "I think they did differ significantly once we have trained them." Then the obvious question: real compared with what? The gardener is a control, but the untrained model is the thing we want to improve. So I compared every raised role with the base model, which gets 56% with no prompt:

Raised role	minus base
critic	−0.04
step	−0.03
inventor	−0.07
gardener	−0.10
debater	−0.12
doer	−0.42

Every role made the model worse. The critic won the frozen comparison because the gardener fell further, not because the critic climbed. My note in the journal: "These were supposed to have higher intelligence than the base, but if we trained it and it had worse, that's not good." The doer is a special case. It learned to answer with a bare number in a box and no working, and on maths that is a reliable way to be wrong.

The role did stick

Before the bad news, the good news, which is the half of the question the costumes failed. The critic prompt asks the model to check its work. After training, with no prompt, the raised critic used checking words ("verify", "wait", "mistake") in 100% of its answers. The base model does it in 27%, and the raised gardener in 18%. A blind judge agreed. I gave Codex 700 answers with the model names stripped and asked it to say which model wrote each one. Chance is 100 correct. It got 223. It spotted the debater 81 times out of 100 and the critic 54 times. It could not tell the base model, the step role and the gardener apart, which is itself a finding: those three write the same way. So raising works on behaviour. The model keeps the role when you take the prompt away. What it doesn't do is make the model smarter.

The bug in the recipe

Here's where reading the rows paid off again. The training data was every answer the model wrote in role, right or wrong. Nobody filtered it. So each role was partly trained on its own mistakes, and the share of mistakes differed by role:

Role	training answers that were correct
critic	67%
step	66%
inventor	62%
gardener	61%
debater	53%
doer	23%

Put that next to the table above and the order is almost the same. The roles that trained on more wrong answers got worse at maths. That's a confound, and it means the first run can't say whether the role or the errors did the damage.

Run two: correct answers only

The fix is simple to say. Train only on answers the answer key marks correct. To keep it fair, every role trains on the same problems, so I kept the 275 problems that every role got right. The doer had to go, because it got so few right that the shared set shrank to 83. Four passes over the 275 problems gave 68 training steps, close to the first run's 63, so the amount of training stayed the same and only the quality of the data changed. My prediction: "Yes, with the correct answer it should beat the base model."

Eight and a half hours later:

Raised role	run one, minus base	run two, minus base
step	−0.03	+0.02 [−0.03, +0.08]
critic	−0.04	−0.02 [−0.08, +0.03]
gardener	−0.10	−0.03 [−0.08, +0.02]
inventor	−0.07	−0.04 [−0.09, +0.01]
debater	−0.12	−0.09 [−0.14, −0.04]

And the frozen question, critic minus gardener: 0.00 [−0.03, +0.04]. Clean data closed most of the damage. The gardener went from −0.10 to −0.03. It also closed the critic's lead, because that lead was mostly the gardener's damage. Step by step is the only role above the base model, and its interval includes zero. Only the debater is clearly worse. My reading, written before anyone explained it to me: "It's not better, but it's not worse. The truth of the matter is the change is too small to be measured."

What I'll claim, and what I won't

I'll claim that training a small model on its own role-prompted answers installs the role. The critic keeps checking with no prompt, and a blind judge can pick out the critic and the debater far above chance. I'll also claim that training on unfiltered answers made every role worse at maths, roughly in order of how many wrong answers it trained on. Filtering out the wrong answers removed most of that damage, and it also removed the critic's advantage over the control. I won't claim that raising can't make a model smarter. This was a 0.8-billion-parameter model, 275 problems, 68 training steps and one graphics card. A real effect smaller than about five points would hide inside these intervals. In my words: "I think we didn't train enough for the change to be substantive. We are limited by hardware and capabilities." What's left is a habit without a benefit. The critic checks its work every time and gets no more answers right for it. Edison is supposed to have said he found ten thousand ways not to make a light bulb. This is one of mine. The harness, the training runs and the analysis were built and run by AI agents, and a second AI judged the answers blind. My part was the predictions, written before each run, the questions, and refusing to read a number until five of its rows had been checked by hand. The code and the journal, predictions included, are public in rsi-path.