Supervised Fine-Tuning vs Reinforcement Fine-Tuning

Imitation versus optimisation

SFT is imitation. Every gradient step pulls the model toward a response someone has already produced, which makes training stable and predictable and makes the data the product. The method cannot ask for more than the examples contain. It is closely related to knowledge distillation, where the examples come from a stronger model, and to imitation learning generally.

RFT is optimisation against a score. OpenAI's documentation describes the loop: the system "samples several responses per prompt, scores them with the grader, and applies policy-gradient updates based on those rewards". Nothing in that loop requires a reference answer, only a way to tell better from worse. The DeepSeek-R1 paper (January 2025; later published in Nature) argues that this removes the dependence on human-annotated reasoning traces and reports emergent self-reflection, verification and strategy adaptation. Its pure-RL model, R1-Zero, rose from 15.6% to 71.0% on AIME 2024 during training.

The grader is the hard part

RFT moves the difficulty from collecting answers to specifying a reward. A grader that can be satisfied by something other than the intended behaviour will be. OpenAI warns that a model may learn to "reward hack your grader" and that a signal a lucky guess can satisfy is too noisy to train on. DeepSeek chose rule-based accuracy and format rewards for R1-Zero rather than a learned reward model, stating that neural reward models "may suffer from reward hacking" at scale. Tasks with checkable outcomes, such as mathematics, code with tests and structured extraction, suit RFT for this reason. Subjective tasks need rubric graders such as an LLM judge, with the added risk that the judge can be gamed.

Why pipelines use both

Reinforcement learning needs some reward to learn from, so a model that never succeeds gets no signal. SFT is the usual fix. DeepSeek-R1's pipeline has four stages: an SFT cold start on thousands of long chain-of-thought examples, reasoning-focused RL, a second SFT round on about 800,000 samples generated by the RL model, then a final RL stage. The cold start also corrected R1-Zero's readability problems.

A May 2026 write-up by Kim and Ahmad on training a small recursive model reports the same dependency in sharper form: Qwen3.5-4B scored zero on pass@16 before an SFT cold start from a larger teacher, and GRPO then lifted the training reward from 0.3 to 0.6. That is a single task reported in a blog post, and should be weighed accordingly.

Small models and cost

RFT is not automatically the better route to a capable small model. In DeepSeek's comparison at 32B parameters, distilling the large RL-trained model into the small one by SFT (72.6% on AIME 2024) beat applying RL directly (47.0%). RFT also costs more per training prompt, since each needs several sampled completions and grader calls, although published like-for-like cost ratios are scarce. Both methods can be applied to all weights or through adapters such as LoRA; the choice of signal and the choice of which parameters to update are independent.