Feed-Forward Network

Also known as: FFN, MLP Block

In Transformers, the feed-forward network (position-wise MLP) follows attention in each block, expanding to a wider hidden size then projecting back.

Much of an LLM’s parameter count lives in FFNs; MoE variants sparsely activate expert FFNs per token.

← Back to AI Glossary