TRL

Each method pairs a Trainer class with a Config dataclass. Configs extend transformers.TrainingArguments, so all of its arguments work in any trainer config.

Many more trainers (OnlineDPO, ORPO, CPO, GKD, …) live in trl.experimental with unstable APIs: https://huggingface.co/docs/trl/experimental_overview

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",  # model ID or a PreTrainedModel instance
    args=SFTConfig(output_dir="Qwen2.5-0.5B-SFT"),
    train_dataset=load_dataset("trl-lib/Capybara", split="train"),
)
trainer.train()

Pass model as a string and route loading kwargs through modelinitkwargs (e.g. {"dtype": "bfloat16", "attnimplementation": "kernels-community/flash-attn2"}) instead of calling frompretrained yourself. The tokenizer/processor is inferred from the model; pass processingclass only when it differs. For LoRA, pass peftconfig=LoraConfig(...).

Dataset formats

Conversational: {"messages": [{"role": ..., "content": ...}]} (language modeling) or {"prompt": [...], "completion": [...]}. The chat template is applied automatically — never apply it yourself. Extra columns are allowed; GRPO forwards them to reward functions. Reference: https://huggingface.co/docs/trl/dataset_formats

SFT: the fields that matter

SFTConfig(
    max_length=1024,        # truncation length; None disables truncation
    packing=True,           # pack sequences into max_length blocks: fewer pad tokens, higher throughput
    padding_free=True,      # flatten batch, no padding; requires FlashAttention; implied by packing
    use_liger_kernel=True,  # fused Liger kernels, reduces peak memory
    assistant_only_loss=True,  # loss only on assistant turns (conversational datasets)
)

GRPO: online RL

def reward_len(completions, **kwargs):
    return [-abs(20 - len(c[0]["content"])) for c in completions]

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=reward_len,  # or a list; rewards are summed
    args=GRPOConfig(output_dir="Qwen2.5-0.5B-GRPO", max_completion_length=512),
    train_dataset=load_dataset("trl-lib/DeepMath-103K", split="train"),
)

Reward functions are called with keyword arguments prompts, completions, completionids, trainerstate, plus every extra dataset column — accept **kwargs for the ones you ignore. Return list[float], one reward per completion. With conversational data, completions is a list of message lists, not strings.

The generation batch is perdevicetrainbatchsize × numprocesses × stepspergeneration (or set generationbatchsize directly) and must be divisible by numgenerations (default 8). Generation is the usual bottleneck — enable vLLM with usevllm=True: vllmmode="colocate" shares the training GPUs (size with vllmgpumemoryutilization); vllmmode="server" uses a separate trl vllm-serve --model <model_id>.

AsyncGRPOTrainer (trl.experimental.async_grpo) implements the same algorithm with generation decoupled from training: a background worker streams completions from a vLLM server while the training loop consumes them, so the two overlap instead of alternating.

CLI

Flags mirror the config fields: trl sft --modelnameorpath Qwen/Qwen2.5-0.5B --datasetname trl-lib/Capybara. YAML via --config; distributed presets via --accelerate_config zero3 (Python scripts: accelerate launch train.py).