The Benchmark Was Fair. My Explanation Wasn't.

I started this because a number on a model card looked suspicious. MiniCPM5-2B was reporting an AIME 2025 score of 86.5 against 29.6 for Qwen3.5-2B. That is a large gap between two models in the same size class, and the easiest part of a comparison to get wrong is usually the baseline. There was a concrete reason to check this one. The models’ chat templates did different things when the harness didn’t explicitly set enable_thinking. Qwen3.5-2B closed the reasoning slot before generation began. MiniCPM5 left the model a choice. Qwen3.5-4B, in the same family, opened the slot. You could call the same function on all three and end up giving them different starting conditions. I built a template auditor that renders those cases rather than guessing from the template text. It confirmed the mismatch. I then made the mistake I was supposed to be investigating: I saw a plausible explanation and started expecting the result to support it.

What I actually ran

The evaluation harness runs both models on the same 30 AIME 2025 problems, with the reasoning flag set explicitly. The shared settings were temperature 0.6, top_p 0.95, and a 32,768-token output cap. A fourth stage used Qwen’s recommended thinking-mode top_k and presence penalty to check whether the shared sampler was costing it points. The run took place in September on one RTX 4060 with 8 GB of memory. There were 259 attempts across four configurations. The score is avg@n: calculate accuracy within each problem, then average across problems. That matters because I changed the attempt count during the first stage. Dividing all correct attempts by all attempts would give the early problems more weight.

Configuration	Attempts	avg@n	95% interval
Qwen3.5-2B, thinking off	79	17.5%	7.5–29.2%
Qwen3.5-2B, thinking on	60	11.7%	3.3–21.7%
MiniCPM5-2B, thinking on	60	71.7%	56.7–85.0%
Qwen3.5-2B, its recommended thinking sampler	60	15.0%	5.0–26.7%

The intervals in the full report use 20,000 bootstrap resamples over problems. They are wide, as you would expect from 30 problems. These are results for this harness and run, not universal scores for the models.

My explanation didn’t survive

Turning thinking on did not rescue Qwen’s score. The point estimate fell from 17.5% to 11.7%. The intervals overlap, so I’m not claiming that turning it on reliably made the model worse. But the improvement I had expected wasn’t there. The template mismatch was real. My explanation of the scorecard was not supported by the run. The sampler was the next thing to check. Qwen’s documentation warns about loops and recommends a presence penalty. I had used one sampler for both models, which was a reasonable attempt at keeping the comparison controlled, but it also meant I hadn’t used the settings Qwen’s authors recommended for this model. That deserved another stage. With the recommended thinking sampler, Qwen scored 15.0%, up 3.3 percentage points from the shared sampler. The paired interval for that change touched zero. It didn’t explain the gap either. MiniCPM5’s absolute score also came in below its model card. So did Qwen’s. I didn’t reproduce the card’s exact scores, and I don’t know the cause of that offset. What I could say was narrower: neither of the two suspected mechanisms removed MiniCPM5’s advantage in this experiment. The relative comparison held up better than I had expected.

The answer was there. The ending wasn’t.

Reading the outputs was more useful than staring at the score. Qwen with thinking on finished only seven of its sixty attempts under the shared sampler. All seven were correct. The other fifty-three looped or reached the token cap without producing a correct answer. Under its recommended thinking sampler, nine attempts finished, and all nine were correct too. That is sixteen correct answers from sixteen attempts that ended normally, and zero correct answers from the other 104. This doesn’t establish 100% reasoning accuracy: the problems it manages to finish may be the easier ones. It does show why a single accuracy column was hiding a useful distinction. Across all four stages, attempts that terminated were correct 66 times out of 95, or 69.5%. Attempts that looped or were capped were correct eight times out of 164, or 4.9%. Those are pooled attempt-level rates; they are a separate measurement from the per-problem avg@n scores above. The repetition penalty reduced Qwen’s detected loops from 45 to 33. Unfortunately, cut-offs went from eight to eighteen. Mean output length increased by roughly 5,300 tokens. On this run, much of the apparent improvement in looping became longer unfinished output. Calling that a fix would have been generous.

The parts I would do differently

The lab notes are longer than I would like, because I managed to find several ways to make the experiment less tidy. The first stage started with four attempts per problem and moved to two partway through. I added early loop stopping during that stage too. The grader initially accepted an incomplete final \boxed{} expression, so it had to be fixed and the saved outputs regraded. One grade changed from wrong to correct. Four outputs remain flagged as uncertain because I had saved only the last 400 characters, which was not enough to regrade everything properly. The repository preserves the original result file. I should have saved complete answers from the start. I should also have checked the vendor settings before the first full run rather than learning about them from the looping results. Early stopping assumes that a detected reasoning loop would not eventually recover. That is plausible, but I have not measured its recovery rate by rerunning those attempts without the stop. Power cuts made the persistence code matter more than I had expected. Initially I saved results after an entire batch completed. Later I saved and flushed each attempt as it finished. A subsequent interruption then cost the answer in flight instead of the whole batch. That was a small engineering change with a very obvious benefit. I also learned not to add up per-answer latency in a batched run and call it GPU time. Several answers were running concurrently. The stage timing comes from the wall-clock log, excluding the periods when the machine was off. The stages used different concurrency because MiniCPM5’s full-length sequences occupied more memory, so even those timings are the cost of this experiment on this machine rather than a clean speed comparison.

Checking the numbers

The four saved JSONL files, harness, settings, and limitations are public. I recomputed the counts and avg@n scores from those saved files on 4 October. That was a check of the recorded results, not another model run. I still think the template defaults should be explicit. But the result I needed to publish was that the defect I found didn’t explain the benchmark I had set out to challenge. I would have preferred a cleaner experiment. I would also have preferred to be right. The files are there so somebody else can check what happened. This article was drafted with AI assistance from the recorded code, results, and lab notes. The September runs are the underlying work; the wording is a new account of it, not a claim that this prose was written without assistance.