Loss Function

Also known as: Objective Function

A loss function quantifies prediction error during training. Optimization (gradient descent / Adam-style) minimizes expected loss on the training distribution.

LLM default

Token-level cross-entropy (negative log-likelihood) is standard for next-token prediction. Average CE relates to perplexity ($\mathrm{PPL} = e^{\overline{\mathrm{CE}}}$). Preference methods (DPO) use different pairwise objectives on top of an LM.

Loss ≠ product goal

Downstream quality still needs separate task evaluation—accuracy, groundedness, latency, preference wins. A falling train loss can hide overfitting or format collapse. Log loss curves, but ship on metrics tied to user outcomes.

When comparing runs, match tokenization and packing: loss per token is only comparable under the same data and tokenizer.

← Back to AI Glossary