Activation Function

An activation function introduces nonlinearity between layers so networks can approximate complex functions. Examples: ReLU, GELU, SiLU/Swish, tanh, sigmoid, softmax (at outputs).

Transformers often use GELU or SwiGLU variants in MLP blocks.

← Back to AI Glossary