I’m interested in making models better at reasoning about conceptual issues, like alignment, bargaining solutions, ethics, etc. A natural question for how to make models better at conceptual reasoning is where existing models get their conceptual reasoning capabilities from. In particular, frontier models are improving at conceptual reasoning tasks. So clearly, the AI companies are doing something that helps. Unfortunately, it’s not so clear what they’re doing in general and it’s also not so clear what they’re doing that helps.

Enter Olmo 3, a family of LLMs trained by the Allen Institute for AI (AI2). AI2 is very open about how they train Olmo. In particular, they publish papers, model weights, training checkpoints, and the training data. As far as I know, Olmo 3 is the best model that is open in this sense. (It’s pretty far from the current frontier, though. Also, their training pipeline leans heavily on the use of distillation, i.e., training on outputs from stronger models.) Additionally, the training of Olmo proceeds in a few distinct stages, so in principle it allows attributing capability gains/losses to specific kinds of training or training data.

In this post, I measure the performance of different post-trained checkpoints in the Olmo 3 family and try to draw some conclusions about what kinds of training improve conceptual reasoning performance. Unfortunately, the results are relatively murky.

Note: All the experimental work here is done by coding agents. Relative to a research paper, I’ve tried less hard to ask my coding agents to double-check things and I gave them less careful specifications of what I want, etc.

The datasets

I use two of our datasets for measuring conceptual reasoning capabilities – LMCA and what I’ll here call “DTBench”. (I’m using the capabilities component of DTBench.) LMCA measures ability to assess (informal) arguments in conceptual domains. Specifically I use a version where I present the model with two argumentative critiques of the same text and ask it which critique is better. At this capability level I think LMCA is a great measure of the kind of conceptual reasoning capabilities that I care about. DTBench is a dataset of multiple-choice decision theory questions related to Newcomb’s problem. While these questions are about a confusing, conceptual domain, the questions themselves mostly aren’t very conceptual in that they mostly have an uncontroversial, non-fuzzy, ground-truth answer. So it’s much less clear how useful of a measure it is for what I’m interested in. It’s cheap enough to run, though. Also, because it explicitly measures a pretty narrow domain, it’s more clearly out of distribution and therefore interesting for studying generalization. Both datasets cover a broad capabilities range, so a 1–2% improvement in scores is substantial. Scores generally come with confidence intervals broader than this, but paired tests sometimes allow drawing conclusions about which of two models is better nonetheless.

Results

The following two figures show the model scores at the different training checkpoints. So, for instance, it shows that after just (pretraining and) Think SFT post-training, Olmo 3 7b achieves an accuracy 0.558 on LMCA (i.e., it correctly guesses which of two critiques is better 55.8% of the time). The figure also gives the parse rate and the fraction of questions for which we have no results even after retrying.

Since consecutive pipeline stages were evaluated on the same questions, we test each stage transition with a paired test: for every question we compute the change in success rate (fraction correct among five sampled answers – ignored responses that don’t parse) between the two stages, and the asterisks report a two-sided test of whether these per-question changes average to zero (* p<.05, ** p<.01, *** p<.001).

Some relevant facts and speculations about the training data

Here are the fractions of prompts that contain conceptual keywords in the different training datasets:

Think-SFT-7B: 1.11%

Think-DPO-7B: 4.80%

Think-RL-7B: 2.62%

Think-SFT-32B: 1.61%

Think-DPO-32B: 5.00%

Think-RL-32B: 2.56%

Instruct-SFT: 2.05%

Instruct-DPO: 4.58%

Instruct-RL: 2.32%

RL-Zero-Math-7B: 0.06%

RL-Zero-Code-7B: 0.20%

RL-Zero-IF-7B: 3.62%

RL-Zero-General-7B: 4.50%

RL-Zero-Mix-7B: 1.55%

(From hand-auditing, most of these are false positives, i.e., training on most of these doesn’t provide any conceptual training signal.)

Interestingly, some of the RLVR datasets have high fractions of prompts containing conceptual keywords. My understanding is that the RLVR datasets are much less conceptual nonetheless. For instance, the instruction-following RL dataset rewards mostly non-conceptual aspects of the model response. In general, it’s relatively hard to reward responses to conceptual prompts (whereas it’s easy to pose conceptual questions to strong models and train your weaker model on the strong model’s output). This is supported by very cursory hand-auditing.

Interpretation

First, I should say that this is reading tea leaves to some extent, because these models don’t differ that much in performance.

TLDR: The larger models are better; reasoning models are better than instruct models; most other post-training effects are small. It’s unclear how much intuitive notions of relevance of data track usefulness for improving LMCA/DTBench scores. For DTBench, some of the post-training shows some returns on scale (beyond very minimal “elicitation returns”). For LMCA, the situation is relatively murky.

(For context, these scores are all worse than Haiku 4.5. The best 32b scores on LMCA are about as good as Qwen 3.5 4b.)

In more detail:

• As one would have expected, the bigger 32b models are generally better than the 7b models. There are some differences in the training datasets and methods, though, between corresponding models of the two different sizes.
• The Think branches generally seem better than the Instruct branches. This is perhaps expected, because both LMCA and the DTBench are tasks that one would intuitively think benefit from thinking for longer. Indeed, current frontier models get returns from thinking on datasets like LMCA. That said, in fall 2025 we found that turning on reasoning did not help the models of the day and reasoning versions of the same model didn’t outperform instruct versions of that model.
• Among the 7b models, the Think SFT/DPO models seem generally better than the RL-Zero models. That said, some of the RL-Zero models perform surprisingly well – sitting between the Think and the Instruct SFT models – considering that they mostly train on domain-specific data (say, coding) with basically no coverage of anything conceptual. The differences mostly aren’t statistically significant. One plausible explanation for this (given the strong Think > Instruct differential) is that eliciting longer reasoning is more important than domain relevance of the training data, and so the Instruct SFT models are worse despite having more relevant training data.
  • It’s still very surprising that the RL Zero code model posts basically the same LMCA score as the best Think branch model, given that coding doesn’t seem relevant to LMCA.
  • Note that the RL-Zero-Math model only gets a high DTBench score because it only produces parsable answers on relatively easy questions. See the results below, which are based on matching the questions.
• Maybe the DPO training generally helps (improvements in 7 out of 8, three statistically significant, only one very small regression). On the other hand, the most relevant comparison – 7b/32b Think branch on LMCA – shows no improvement. So it’s unclear.
• RLVR results seem somewhat all-over-the place. (E.g., note that RLVR on the 32b Think model first decreases LMCA and increases DTBench performance and then on further training increases LMCA performance and decreases DTBench performance. Probably just noise…)
• How useful is Think-SFT training? Of course, it induces reasoning, which we suppose is helpful. (Also correct formatting.) But is it just some super basic form of elicitation? (Of course, an extremely fuzzy question…) E.g., is Think-SFT basically just telling the base model “try to get the answer right, reason for many thousands of tokens if you like”. I suspect it does a little more, for the following reasons. But I’m not sure.

Conclusion and possible next steps

Unfortunately, I think the most interesting question – does training on broad chat datasets as per the Think-SFT/DPO training stage make the models better at conceptual reasoning? – isn’t fully resolved.

As usual, a big bottleneck is that our evaluation is low-powered. If LMCA was much larger or we had three more datasets like LMCA, we’d probably have a much better sense of whether/how much the different post-training stages help. An unusual affordance in this case is that in principle we could use frontier models to make a large synthetic extension of LMCA. So if I wanted to spend, say, a week to get a better sense of the relative ability of different Olmo checkpoints, I’d consider making this dataset.

Olmo 3 post-training isn’t all that expensive. They give $2.75M as an estimate of total training cost for the 32b Think model. About 85% of that is pretraining. So replicating the entire post-training run would cost in the hundreds of thousands. But most of their post-training is not a candidate for making models better at conceptual reasoning anyway. (Also, if we care about capability rather than size of the base model, we could fine-tune, say, a 4b Qwen 3.5 base model instead of the 32b Olmo base to save further money relative to the Olmo experiments.) So, even with a limited budget, a lot of the things that one could learn from Olmo 3 may be better learned by simply post-training a model oneself.

More results

Evaluating base models

Of course, it’d be useful to have results for the base model, but it’s not clear how to elicit the base model in a way that’s “fair”. I have tried the following two things, mostly to investigate how much work is done by Think-SFT.

• Giving it somewhat related (but not quality-selected) few-shot examples from the Think-SFT training mix in such a way that parse rates are acceptable.
  • For the 7b model, I tried three versions of this with 3, 8, 16 examples and got LMCA scores of 0.518, 0.53, 0.52. This is much worse than the models in the Think-SFT branch.
• Giving it five LMCA examples with Fable-written example reasoning.
  • For the 7b model, this yielded an accuracy of 0.555. This is roughly as good as the Think-SFT and Think-DPO models.
    • But then Think-SFT and Think-DPO are further improved by few-shot prompting them with the gold examples as well. Think-SFT gets 0.581 (+0.023, paired p=0.032) and Think-DPO gets 0.570 (+0.017, paired p=0.17). 
  • For the 32b model, this yielded an accuracy of .574, which is worse than (although within error bars of) Think-SFT, Think-DPO, and the RLVRed models. It’s roughly level with the first checkpoint within the Think-SFT stage (see below) (the model resulting from the first 9.3% of training).

SFT-/DPO-trained models versus RL-Zero models

Thinking versus Instruct models

Rewards through Think-SFT

Olmo3 publishes checkpoints through the Think-SFT stage, so we can see whether Think-SFT sees returns from scale. Note that the 32b model is trained by training two models at different learning rates and “souping” them (i.e., averaging the weights) in the end.

(On DTBench, the first Think-SFT 7b checkpoint has a parse rate of 35% and is missing results for 2.5% of questions. All other checkpoint results are complete.)

Although the confidence intervals of the individual checkpoints all overlap, the paired test says the 32b improvement over the course of Think-SFT training is significant: comparing the first published checkpoint (step 1000, ~9% of training) against the released model, the per-question changes come out as +.022 (p=.005) for the 5e-5 run and +.017 (p=.057) for the 1e-4 run on LMCA.

Does Nemotron Cascade 8b improve at conceptual reasoning through post-training?

I compared two checkpoints: the one right before the “RLHF” phase and the one right after. (The only post-training stage before RLHF is SFT. Later training stages are RL for math, coding, etc. Since assessing base models is so murky, I didn’t bother.) Like Olmo 3’s DPO stage, Cascade’s RLHF has pretty broad coverage. (It’s not just aligning the model or making sure it doesn’t use curse words or whatever.) As a result, the Cascade paper reports broad improvements from the RLHF stage. Note that they train against a reward model rather than using the DPO technique.

Anyway:

So, it helps with DTBench and is neutral or slightly harmful for LMCA.