Gradient Checkpointing

Also known as: Activation Checkpointing

Gradient checkpointing saves memory by not storing all intermediate activations: during backward, selected layers are recomputed from saved checkpoints. You trade extra compute for the ability to train longer sequences or larger micro-batches.

When to use it

Cost

Expect slower steptime (often ~20–30%+ depending on what you recompute). Prefer checkpointing transformer blocks rather than every tiny op. It does not change the mathematical gradient—only the memory/time tradeoff.

Enable it when you are VRAM-bound; leave it off for short-context runs where throughput matters more than memory headroom.

← Back to AI Glossary