Gradient Descent
Also known as: GD
Gradient descent minimizes a scalar loss by repeatedly stepping parameters opposite the gradient: $\theta \leftarrow \theta - \eta \nabla_\theta L$. The learning rate $\eta$ controls step size. Gradients come from backpropagation in neural nets.
Why it matters
- Core training loop behind almost all deep models
- Full-batch GD is rarely used at LLM scale; SGD / mini-batches dominate
- Learning-rate schedules and adaptive methods build on this update rule
How it shows up
Plain GD uses the full dataset each step. Practice uses mini-batches, momentum, and adaptive optimizers such as Adam. Too-large $\eta$ diverges; too-small wastes compute. Plateauing loss often means LR, data, or capacity issues—not “gradient descent failed.”
Related: gradient accumulation and batch size control the effective step statistics when memory is tight.