Post-Training: SFT, LoRA, DPO and GRPO in Plain Terms

By Gopi Krishna Tummala

Pre-training gives a model knowledge; post-training gives it behavior. A map of the main steps (supervised fine-tuning, LoRA, preference tuning, and reinforcement learning with checkable rewards), following one model as it becomes a SQL assistant.

Part 4 of the MLOps map. Previous: Training Frameworks. Next: LLM Serving with vLLM.

Your company wants an assistant that turns questions like “what was revenue by region last quarter?” into correct SQL for its own data warehouse. You start with an open base model. It knows a lot of SQL from pre-training on the internet, but ask it a question and it may continue your sentence instead of answering, invent table names, or write a query that runs and returns the wrong number.

Pre-training gave the model knowledge. Post-training gives it behavior: answer in the right format, use your schema, prefer queries that are correct and readable, and stop when done. This post follows the SQL assistant through the four steps most teams use: supervised fine-tuning, parameter-efficient fine-tuning with LoRA, preference tuning, and reinforcement learning with rewards you can check.

The idea in one picture

flowchart LR
  B["Base model<br/>Llama, Mistral, Qwen<br/>knows SQL, doesn't<br/>follow instructions"] --> S["Supervised fine-tuning<br/>10K to 1M prompt and<br/>ideal-answer pairs"]
  S --> P["Preference tuning<br/>DPO: better vs worse<br/>style, safety, correctness"]
  P --> R["RL with checkable rewards<br/>GRPO: run the query,<br/>compare results"]
  R --> M["Ship, or merge<br/>specialists with<br/>TIES or DARE"]
  L["LoRA / QLoRA / DoRA<br/>train 0.1 to 1% of params,<br/>keep the base frozen"] -.-> S
  L -.-> P
  L -.-> R

Each step changes behavior a little more. LoRA isn’t a separate step; it’s how most teams afford to run the others.

Supervised fine-tuning: show it good answers

Supervised fine-tuning (SFT) is the same next-token training as pre-training, but on examples of exactly the behavior you want: a question plus the schema as the prompt, and a correct SQL query as the answer. The loss is only counted on the answer tokens, so the model learns to respond, not to repeat questions.

Two things matter more than anything else here. The first is quality over quantity. The LIMA paper fine-tuned a 65B model on just 1,000 carefully chosen examples, and human raters judged its answers as good as or better than GPT-4’s in 43% of comparisons. A few thousand verified question and query pairs from your own warehouse will usually beat a hundred thousand scraped ones. The second is the chat template. Every model family wraps turns in special tokens, and training with the wrong template is one of the most common fine-tuning bugs: the model leaks stray tokens into answers or never learns when to stop. Use the tokenizer’s own apply_chat_template for both training and serving.

LoRA: change a little, cheaply

Updating every weight of a large model needs enormous memory (see Training Frameworks for the arithmetic). LoRA, low-rank adaptation, freezes the original weights and learns a small correction for each weight matrix as the product of two thin matrices:

$W = W_0 + BA$

flowchart LR
  X["x<br/>input"] --> W0["W0, d x k<br/>frozen weights"]
  X --> A["A, r x k<br/>random init"]
  A --> B["B, d x r<br/>zero init"]
  B --> S["scale by alpha / r"]
  W0 --> Add(("+"))
  S --> Add
  Add --> H["h = W0 x + BA x"]
  classDef frozen fill:#6366f1,color:#fff,stroke:#4f46e5
  classDef train fill:#10b981,color:#fff,stroke:#059669
  class W0 frozen
  class A,B,S train

The frozen path (blue) is the original layer. The trainable path (green) is a thin detour through rank r; because B starts at zero, the detour adds nothing until training moves it.

If $W_0$ is $d \times k$ , then $B$ is $d \times r$ and $A$ is $r \times k$ , with a small rank $r$ like 16. For a 4096×4096 layer, that’s about 131,000 trainable numbers instead of 16.8 million. $B$ starts at zero, so the model begins exactly where the base model was and drifts only as far as training pushes it. The adapter is a few hundred megabytes, so you can keep one per task and swap them.

QLoRA goes further by storing the frozen base model in 4-bit precision while training the adapters in higher precision. That’s what puts a 70B model on a single 80 GB GPU: about 35 GB for the 4-bit base, a few GB for adapters and their optimizer state, and the rest for activations.

flowchart TD
  subgraph GPU["One 80 GB H100"]
    QW["Base model in NF4, 4-bit<br/>about 35 GB for 70B, frozen"]
    AB["LoRA A and B in BF16<br/>about 0.5 GB at r=16"]
    DQ["Dequantize to BF16<br/>on the fly, per layer"]
    Gr["BF16 gradients<br/>about 1 GB"]
    Ac["Activations<br/>about 8 GB"]
  end
  subgraph CPU["CPU RAM"]
    Opt["Adam state for adapters<br/>about 2 GB, paged out<br/>only under memory pressure"]
  end
  QW -->|"dequantize"| DQ
  DQ --> Gr
  DQ --> Ac
  AB --> Gr
  Gr -->|"page on demand"| Opt

Approximate sizes. QLoRA’s tricks: 4-bit NormalFloat for the frozen base, quantizing the quantization constants too, and paged optimizer state that spills to CPU memory during spikes instead of crashing.

DoRA is a variant that learns the size and direction of each weight change separately and usually beats plain LoRA at the same rank.

For the SQL assistant, a LoRA adapter trained on your question and query pairs is often all the training you need to start.

Preference tuning: better, not just correct

After SFT, the model writes plausible SQL, but some of it is subtly worse than it should be: a query that’s correct but scans every partition, or one that silently drops rows with a careless join. It’s easier to say which of two answers is better than to write the perfect one, so the next step learns from comparisons.

The classic approach, RLHF, trains a separate reward model on those comparisons and then optimizes the language model against it with reinforcement learning. DPO (direct preference optimization) skips both: it trains directly on pairs of a preferred and a rejected answer, raising the probability of the preferred one relative to a frozen copy of the starting model. It’s much simpler and more stable, which is why it became the default.

DPO has one knob worth understanding, usually called β. Think of it as a leash to the starting model. A small β lets the model move far from where it started, which means bigger behavior changes but also a higher risk of overfitting to whatever superficially separates the preferred answers, such as length or caution. A large β keeps it close. If a preference-tuned model starts refusing harmless requests, the usual fix is better preference data and a tighter leash, not a looser one.

Reinforcement learning with rewards you can check

SQL has a property most tasks don’t: you can check answers automatically. Run the generated query against a test copy of the warehouse and compare its result with the right one. That makes reinforcement learning practical without a learned reward model.

The method most teams use now, GRPO, comes from the DeepSeekMath paper and became widely known through DeepSeek-R1. For each question, the model writes several answers, say eight queries. Each gets a reward (1 if the result matches, 0 if not). Each answer’s advantage is how much better it did than the group’s average, scaled by the spread of rewards in the group:

$\hat{A}_i = \frac{r_i - \text{mean}(r)}{\text{std}(r)}$

The model is then pushed toward its better-than-average answers and away from the worse ones. The group itself is the baseline, so no separate critic network is needed. A question that all eight attempts get wrong, or all get right, teaches nothing, which is the right behavior: the useful signal comes from questions the model is sometimes right about.

The risk is reward hacking. If the reward only checks that the query runs, the model learns to write queries that run. Reward what you actually care about, the correct result, and keep a held-out set of questions people check by hand.

The three ways to tune on preferences and rewards, side by side:

flowchart TD
  subgraph RLHF["RLHF, classic"]
    R1["Human preference pairs"] --> R2["Separate reward model"] --> R3["PPO loop with a critic<br/>unstable, memory hungry"] --> R4["Aligned model"]
  end
  subgraph DPO["DPO, 2023"]
    D1["Same preference pairs"] --> D2["Closed-form loss<br/>no reward model"] --> D3["Aligned model"]
  end
  subgraph GRPO["GRPO, DeepSeekMath 2024 and R1 2025"]
    G1["Batch of prompts"] --> G2["G answers per prompt"] --> G3["Score with a verifier<br/>rules or an LLM judge"] --> G4["Normalize within the group"] --> G5["Policy gradient<br/>no critic"] --> G6["Reasoning model"]
  end

Each newer method removes a moving part: DPO drops the reward model and RL loop, GRPO drops the critic by using the group as its own baseline.

Keeping what the model already knew

Fine-tuning narrowly can cost general ability. After heavy SQL training, the assistant might get worse at explaining its queries in plain English, and preference tuning toward caution can make it refuse ordinary requests. Three practical defenses:

flowchart LR
  Base["Base model<br/>capable, unaligned"] -->|"SFT only"| Under["Under-aligned<br/>wrong format,<br/>harmful outputs"]
  Base -->|"heavy DPO, low beta,<br/>refusal-heavy data"| Over["Over-aligned<br/>refuses benign requests"]
  Base -->|"SFT, no replay"| Forget["Capability collapse<br/>forgets math after<br/>medical SFT"]
  Base -->|"SFT + DPO + replay<br/>+ the right beta"| Target["Capable and aligned"]
  classDef good fill:#10b981,color:#fff,stroke:#059669
  classDef bad fill:#ef4444,color:#fff,stroke:#dc2626
  class Target good
  class Under,Over,Forget bad

Three common ways to land in the wrong place, and the recipe that usually avoids them.

Mix in general data. A small share of general instruction data in each training stage reminds the model of everything else it should still do. InstructGPT applied the same idea during its RL phase by mixing in pre-training updates.
Prefer a lower LoRA rank first. A smaller adapter makes a smaller change. Raise the rank only if the task needs it.
Merge carefully. You can combine fine-tunes, for example a SQL adapter and an explanation adapter, with merging methods like TIES or DARE that reduce interference between them. Merge the actual weight changes (for LoRA, the product $BA$ ), not the A and B matrices separately; tools like mergekit and PEFT’s adapter merging handle that for you.

What I’d remember

Pre-training gives knowledge; post-training gives behavior. Each step nudges behavior, and data quality decides how far it gets.
LoRA makes post-training affordable. Small adapters on a frozen base, and QLoRA puts a 70B model on one GPU.
Preferences are easier to give than perfect answers. DPO learns directly from pairs; β is the leash to the starting model.
If you can check answers, use reinforcement learning. GRPO compares a group of attempts and needs no critic.
Protect general ability. Mix in general data, keep changes small, and evaluate on more than the target task.

The whole module on one page:

mindmap
  root((Post-training))
    SFT
      Loss on answer tokens only
      Chat template must match
      Quality over quantity
    PEFT
      LoRA trains r times d plus k
      QLoRA NF4 base
      DoRA magnitude and direction
    Preference
      DPO with no reward model
      Beta is the leash
      Reference log-probs
    RL
      PPO with a critic
      GRPO group baseline
      Verifiable rewards
    Keep capabilities
      Replay data
      Merge with base
      TIES and DARE

Where to go next

Nathan Lambert’s free RLHF Book, the clearest longer treatment of everything here.
Papers: LoRA, QLoRA, DPO, DeepSeekMath (GRPO).
The Multimodal Atlas, for the same steps applied to vision-language and image models.
Qwen3-VL and Mixture of Experts, for post-training one specific VLM, and why to be careful fine-tuning a mixture-of-experts router.
LLM Serving with vLLM, the next part of this series.

Previous: Training Frameworks. Next: LLM Serving with vLLM.

Questions to sit with

For your own use case, which step would add the most: better SFT data, preference pairs, or a checkable reward?
Why does initializing $B$ to zero matter for the first step of LoRA training?
What would reward hacking look like for the SQL assistant, and how would you catch it?