CPU Offloading

References

• Stable docs: @docs/training/cpu-offloading.md
• Structured metadata: @skills/nemo-mbridge-perf-cpu-offloading/card.yaml

What It Is

Two independent mechanisms to move data from GPU to CPU memory:

Quick Decision

Enablement

Optimizer CPU offloading (recommended for large models)

cfg.optimizer.optimizer_cpu_offload = True
cfg.optimizer.optimizer_offload_fraction = 1.0
cfg.optimizer.overlap_cpu_optimizer_d2h_h2d = True

CLI overrides:

optimizer.optimizer_cpu_offload=True \
optimizer.optimizer_offload_fraction=0.5 \
optimizer.overlap_cpu_optimizer_d2h_h2d=True

Activation CPU offloading (small/medium models only)

cfg.model.cpu_offloading = True
cfg.model.cpu_offloading_num_layers = 16
cfg.model.cpu_offloading_activations = True
cfg.model.cpu_offloading_weights = False

cfg.model.pipeline_model_parallel_size = 1
cfg.model.recompute_granularity = None
cfg.model.cuda_graph_impl = "none"

Config Parameter Reference

Optimizer offloading

Activation offloading

Compatibility And Constraints

Activation offloading

• pipelinemodelparallel_size must be 1
• recompute_granularity must be None
• Cannot combine with finegrainedactivation_offloading
• Cannot combine with CUDA graphs
• cpuoffloadingnumlayers must be in [0, numlayers-1)

Optimizer offloading

• Requires usedistributedoptimizer = True (default in most recipes)
• No PP, recompute, or CUDA graph restrictions
• optimizeroffloadfraction must be in [0.0, 1.0]

Practical: large MoE models

Activation offloading is blocked for Qwen3-30B-A3B and similar large MoE models. The PP=1 constraint means each GPU holds all 48 layers; model weights + optimizer states alone (~70 GB) exceed H100 80 GB capacity.

Minimal Runnable Command

uv run python scripts/training/run_recipe.py \
  --recipe qwen3_30b_a3b_pretrain_config \
  optimizer.optimizer_cpu_offload=True \
  optimizer.optimizer_offload_fraction=0.5 \
  train.train_iters=20 \
  train.global_batch_size=8 \
  train.micro_batch_size=1

Verification

Unit tests

uv run python -m pytest \
  tests/unit_tests/models/test_gpt_full_te_layer_autocast_spec.py -k "cpu_offload" \
  tests/unit_tests/peft/test_utils.py -k "cpu_offload" -q

Success criteria

• Config validation passes for the selected offloading mode
• Training completes without OOM or NCCL errors
• Loss matches the non-offloaded baseline (max delta < 0.001)
• Memory usage drops proportionally to offload fraction

Code Anchors

MCore activation offload constraints

```1296:1310:3rdparty/Megatron-LM/megatron/core/transformer/transformerconfig.py if self.cpuoffloading and ( self.cpuoffloadingnumlayers < 0 or self.cpuoffloadingnumlayers >= self.num_layers ): raise ValueError(...)

if self.cpuoffloading and self.pipelinemodelparallelsize > 1: raise ValueError( "Currently there is no support for Pipeline parallelism with CPU offloading" )

if self.cpuoffloading and self.recomputegranularity is not None: raise ValueError( "CPU offloading does not work when activation recomputation is enabled" )

### MCore CUDA graph incompatibility

```1943:1944:3rdparty/Megatron-LM/megatron/core/transformer/transformer_config.py
            if self.cpu_offloading:
                raise ValueError("CUDA graphs not supported with CPU offloading.")

MCore fine-grained offloading mutual exclusion

```1427:1430:3rdparty/Megatron-LM/megatron/core/transformer/transformerconfig.py if self.finegrainedactivationoffloading: assert ( not self.cpuoffloading ), "finegrainedactivationoffloading cannot be enabled with cpu_offloading."

### MCore HybridDeviceOptimizer instantiation

```480:518:3rdparty/Megatron-LM/megatron/core/optimizer/__init__.py
        if config.optimizer_cpu_offload:
            # ... setup cpu/gpu optimizer classes ...
            optimizer = HybridDeviceOptimizer(
                param_groups,
                offload_fraction=config.optimizer_offload_fraction,
                cpu_optimizer_cls=cpu_optimizer_cls,
                gpu_optimizer_cls=gpu_optimizer_cls,
                overlap_cpu_optimizer_d2h_h2d=config.overlap_cpu_optimizer_d2h_h2d,
                pin_cpu_grads=config.pin_cpu_grads,
                pin_cpu_params=config.pin_cpu_params,
            )

Bridge CUDA graph guard

```232:234:src/megatron/bridge/models/gptfulltelayerautocastspec.py assert not config.cpuoffloading and config.recompute_granularity is None, "Cudagraphs not supported"

### Bridge activation offloading in PEFT

```621:631:src/megatron/bridge/peft/utils.py
        if self.config.cpu_offloading and self.config.cpu_offloading_activations:
            x.activation_offloading = True
        x, _ = self.linear_in(x)
        x = self.activation(x)
        if self.config.cpu_offloading and self.config.cpu_offloading_activations:
            x.activation_offloading = True
        x, _ = self.linear_out(x)

Failure Diagnosis

Known Limitations

• Activation offloading requires PP=1, making it impractical for large models

(30B+ MoE) that need pipeline parallelism.

• Optimizer offloading throughput penalty scales linearly (~1.9x at 25%,

~4.2x at 100% for Qwen3-30B-A3B).

• D2H/H2D overlap provides only ~7% speedup because CPU Adam compute is

the dominant bottleneck.

• finegrainedactivation_offloading is a separate module-level approach

that works with PP > 1 but cannot be combined with layer-level cpu_offloading.