Gradient Descent

Also known as: GD

Gradient descent minimizes a scalar loss by repeatedly stepping parameters opposite the gradient: $\theta \leftarrow \theta - \eta \nabla_\theta L$. The learning rate $\eta$ controls step size. Gradients come from backpropagation in neural nets.

Why it matters

  • Core training loop behind almost all deep models
  • Full-batch GD is rarely used at LLM scale; SGD / mini-batches dominate
  • Learning-rate schedules and adaptive methods build on this update rule

How it shows up

Plain GD uses the full dataset each step. Practice uses mini-batches, momentum, and adaptive optimizers such as Adam. Too-large $\eta$ diverges; too-small wastes compute. Plateauing loss often means LR, data, or capacity issues—not “gradient descent failed.”

Related: gradient accumulation and batch size control the effective step statistics when memory is tight.

← Back to AI Glossary