Notes on Midtraining
Creating better specialized LLMs with midtraining and continual pretraining...
The training process for large language models (LLMs) has historically been composed of two high-level stages: pretraining and post-training. Pretraining exposes the LLM to a large volume of diverse textual data, while post-training focuses on refining and aligning model behavior via techniques like supervised fine-tuning (SFT), reinforcement learning (RL), and more. Recently, LLM training pipelines have become increasingly complex due to the addition of intermediate training stages between pretraining and post-training. In particular, three new stages are commonly encountered in the literature:
Continued Pretraining (CPT): an extended period of pretraining that begins from an existing pretrained checkpoint, usually with the goal of specializing the model toward a particular domain or data distribution.
Midtraining: an intermediate stage between general pretraining and post-training that continues the pretraining process on a more curated data distribution, often by annealing the data mixture toward higher-quality, domain-specific, reasoning, or instruction-like data.
Long-Context Training: a dedicated training period that extends the model to longer sequence lengths after initially training on shorter sequences to improve training efficiency.
In this overview, we will focus on the first two concepts outlined above, covering a wide variety of industry tech reports and research papers that propose practical strategies and best practices for CPT and midtraining. Notably, these techniques are highly similar in implementation—both typically continue LLM pretraining on large-scale textual data using next-token prediction or related objectives. As a result, the boundary between CPT and midtraining is somewhat blurry, and the terminology is not always used consistently across papers. However, these are usually framed as two different concepts in practice:
CPT commonly refers to taking an existing pretrained model—oftentimes from another research group or organization—and continuing to train it on data that specializes the model toward a domain or application of interest.
Midtraining more often refers to a planned stage within the original LLM training pipeline that bridges general pretraining and post-training. Rather than pretraining on a fixed distribution until completion, the data mixture is progressively adapted over time to emphasize different data distributions.
As we will see, CPT and midtraining are powerful tools for building domain-specialized LLMs. However, using these techniques effectively requires rigorous empirical analysis and evaluation to inform proper choices around data quality, mixture design, timing, and duration. To understand how to properly tune these training stages, this post will analyze recent training efforts in detail and extract relevant best practices for successfully configuring midtraining and CPT.
Preliminaries. There are several background concepts that will help to better understand the papers included in this post. Links to further reading on each of these topics have been provided below for reference:
Characterizing Datasets and Building Better Models with Continued Pre-Training [1]
Although LLMs are proficient at a wide variety of tasks, these models can fall short in solving tasks that are not meaningfully represented in their training distribution. If we know during training that our model will be deployed in a particular domain, we can often improve performance by performing CPT on domain-relevant data. In [1], the authors present a practical recipe for efficiently tuning CPT by using shorter and cheaper proxy experiments to identify the best datasets, learning rates, training durations, and data mixtures.
“Running LR sweeps at the 1B-10B token regime is a fast way to identify which datasets enhance model performance. This allows us to remove ineffectual datasets and eventually mix the beneficial datasets and train them for longer periods, making CPT an efficient tool for identifying useful datasets.” - from [1]
The central observation in [1] is that short CPT runs can accurately predict the effectiveness of much longer runs. Rather than immediately running large-scale CPT experiments with various datasets, we can first characterize each dataset using a smaller proxy run. More specifically, we can determine which datasets are worth including in CPT by:
Performing a short CPT run.
Measuring whether training on this data improves downstream evals.
Prior to the full-scale CPT experiments, the authors perform short CPT runs—around 1B tokens instead of 50B+ tokens—on each candidate dataset individually. From these experiments, we can easily see which datasets yield consistent improvements in downstream performance, as well as whether certain data sources have minimal (or even negative) impact. Such an approach allows useful datasets to be identified before investing compute into CPT. In [1], FLAN and OpenWebMath are the only datasets that consistently improve performance.
Once the appropriate datasets have been identified, properly tuning the learning rate and training duration for CPT is essential. A hyperparameter sweep is first performed over the learning rate using a fixed training duration. Then, the best learning rate is used to sweep over possible training durations. Optimal learning rates and training durations differ across datasets; see above. OpenWebMath continues improving model performance over longer training durations, while FLAN reaches peak performance much earlier and begins to degrade model performance at longer durations. There is no universal CPT schedule—tuning is essential because each dataset may exhibit different learning dynamics.
Finally, the paper demonstrates that even stronger results can be achieved by mixing together the beneficial datasets. A simple heuristic is proposed for determining the mixing ratio: weight each dataset according to the amount of training that produced its best individual performance. Intuitively, datasets that remain beneficial for longer training durations receive proportionally more weight in the final mixture. After constructing this mixed dataset, the learning rate and training duration sweeps are repeated to identify the best overall training recipe.
While this approach produces the strongest results in the paper (see above), many tradeoffs still exist when selecting the final CPT setup. For example, training for longer durations may continue to improve in-domain performance, but gradually overspecializes the model and causes forgetting on more general benchmarks. To better balance these competing objectives, authors in [1] merge an earlier CPT checkpoint with a later checkpoint trained for longer, recovering much of the general capability while retaining most of the domain-specific gains.
Domain upsampling. As mentioned previously, annealing the data mixture (i.e., changing how frequently each data source is sampled) during pretraining is a commonly used technique related to midtraining and CPT. In [2], the authors show that inexpensive proxy experiments can be used to efficiently optimize the data mixture used during the final stage of pretraining. Rather than repeating an entire pretraining run, they modify only the final 10–20% of training, allowing many candidate data mixtures to be evaluated at a fraction of the cost.
As shown above, tuning the data mixture during the final stage of pretraining can substantially boost performance. To determine which datasets are most valuable, the authors repeatedly re-run the final training phase while ablating individual data sources and measuring the change in benchmark performance. From these experiments, we can easily estimate the impact of each data source on model quality and refine the final data mixture by upsampling important sources.
Composer 2 Technical Report [3]
Another practical example of CPT is the Composer 2 model from Cursor [3]. This model uses a two-part training framework with CPT and RL to further specialize the Kimi-K2.5 model for realistic coding tasks. Most LLMs possess strong coding capabilities after pretraining, but CPT can further refine these capabilities by emphasizing high-quality data that matches the domain in which the model will be used (i.e., coding sessions in Cursor). Unlike general pretraining, the CPT phase adopts a code-dominated data mixture. By training on this data, we can expand the model’s knowledge, adapt it to the data that it is likely to encounter in the real world, and create a much stronger base model for further RL training.
“The continued pretraining stage aims to improve the language model’s base knowledge, specifically in the domain of coding… Recent models use a staged training approach, progressively filtering towards higher quality data.” — from [3]
Composer 2 undergoes three training stages prior to RL:
A primary CPT phase that uses 32K-token sequences and consumes the majority of training compute.
A shorter long-context extension phase that increases sequence length to 256K tokens.
A final short SFT phase on targeted coding tasks.
These phases are then followed by a downstream RL training phase that uses the same harness as the deployed model in Cursor. This phase focuses on end-to-end coding tasks that reflect realistic software engineering environments and tools.
As shown below, code-specific CPT yields consistent model improvements that continue at scale. A key motivation in [3] is simply scaling continued training on real-world coding tasks. Such performance benefits are especially notable given that the base model is already trained extensively on code—large-scale CPT on top of a pretrained model still yields benefits when creating a specialized coding model.
We see in [3] that CPT influences the effectiveness of downstream RL training. As shown above, improvements from CPT directly translate into better downstream RL performance—CPT is beneficial for downstream RL. The figure above compares three models with different compute budgets (i.e., small, medium, and large) for CPT, then runs identical SFT and RL training phases for each model.
One additional benefit of CPT is that this additional training stage can be used to build customizations or new capabilities into an existing model. For example, authors in [3] also add Multi-Token Prediction (MTP) layers to Composer 2 to enable accelerated inference with speculative decoding. Rather than including a separate training stage for these MTP layers, we can just introduce them during CPT, allowing for joint optimization with the model itself. In this way, CPT allows us to both specialize the model on domain-specific data and make any necessary modifications to the model before later training stages.
Practical Examples of Domain-Specialized CPT
The Composer 2 report [3] is a practical example of how CPT can be used as a starting point for domain-specialized training. We just need a sufficiently large dataset for CPT that aligns well with our downstream application. There are also many examples of CPT being used for specialized training beyond Composer 2. We will now reference several notable domain-specific training pipelines that use CPT and analyze high-level trends in how these pipelines are usually set up. Many of these papers focus on coding, though other domains (e.g., math) are also covered.
