Concise definitions for AI, machine learning, and LLM engineering. Each term has its own indexable page — browse below or search.
150 terms
-
Activation Function
Activation Function: An activation function applies an elementwise nonlinearity after a linear transform so stacked layers can approximate non linear maps.… Read full definition →
-
Adam
(AdamW)
Adam is an adaptive optimizer: it keeps exponential moving averages of gradients (first moment) and squared gradients (second moment) per parameter, then… Read full definition →
-
Adapter
Adapter: An adapter is a small trainable module inserted into a frozen base model so you can specialize without updating (or storing) a full copy of the… Read full definition →
-
Agent
(AI Agent, LLM Agent)
Agent: An LLM agent uses a language model as planner/controller: it proposes actions, calls tools (search, code, APIs, browsers), observes results, and… Read full definition →
-
Alignment
Alignment is the effort to make model behavior match human intentions and product values—typically helpful, honest, and harmless in the intended context.… Read full definition →
-
Approximate Nearest Neighbor
(ANN)
Approximate nearest neighbor (ANN) search finds mostly the nearest vectors much faster than exact scan. Indexes such as HNSW, IVF, and LSH (via FAISS or a… Read full definition →
-
Attention
(Soft Attention, Attention Mechanism)
Attention lets a model decide, at each step, which parts of a memory of vectors matter most—and then form a weighted average of those vectors. In the original… Read full definition →
-
Autoregressive
(Autoregressive Model, AR)
Autoregressive: An autoregressive model builds a sequence one step at a time , and each new token is predicted only from what came before it—not from the… Read full definition →
-
BM25
BM25 is a classic ranking function for sparse retrieval over bag of words indexes. It saturates term frequency, down weights very common terms (IDF), and… Read full definition →
-
Backpropagation
(Backprop)
Backpropagation computes gradients of the loss w.r.t. each parameter by applying the chain rule from output to input through the computation graph. Those… Read full definition →
-
Base Model
Base Model: A base model is a pretrained language model before instruction/chat alignment—strong at continuation and completion, weaker at following assistant… Read full definition →
-
Batch Size
Batch size is how many examples (or tokens) contribute to one optimizer step. Larger batches yield stabler gradient estimates and better GPU utilization;… Read full definition →
-
Beam Search
Beam search is a decoding algorithm that keeps the top B partial hypotheses at each step, expanding every beam by possible next tokens and pruning back to B by… Read full definition →
-
Benchmark
Benchmark: A benchmark is a standardized dataset plus metric used to compare models—e.g. MMLU, HumanEval, GSM8K, BEIR for retrieval. Shared suites make… Read full definition →
-
Byte-Pair Encoding
(BPE)
Byte-Pair Encoding: Byte Pair Encoding (BPE) builds a subword vocabulary by iteratively merging the most frequent adjacent symbol pairs, starting from… Read full definition →
-
Catastrophic Forgetting
Catastrophic forgetting is when training on a new distribution erases previously learned skills. Domain continued pretraining or aggressive fine tuning can… Read full definition →
-
Causal Language Modeling (Causal LM)
Causal language modeling (causal LM): predict the next token from left context only. The training objective behind GPT, LLaMA, Gemma, and other decoder-only… Read full definition →
-
Chain of Thought Prompting
(CoT)
Chain of Thought Prompting: Definition: Chain of thought prompting is a method where the prompt includes intermediate reasoning steps, encouraging the model to… Read full definition →
-
Chat Template
Chat Template: A chat template is the model specific formatting that wraps roles (system / user / assistant / tool) into the exact token sequence an instruct… Read full definition →
-
Chunking
Chunking splits long documents into smaller passages for embedding and retrieval. Chunk size and overlap trade context completeness against retrieval… Read full definition →
-
Citation
Citation: In LLM apps, citations are pointers from generated claims back to source passages, chunk IDs, or URLs. They help users verify answers and help you… Read full definition →
-
Context Window
(Context Length)
Context Window: The context window is the maximum number of tokens a model can attend to in one forward pass—prompt plus generation, depending on API… Read full definition →
-
Continued Pretraining
(CPT, Domain-Adaptive Pretraining)
Continued pretraining (CPT) resumes next-token training on a new domain corpus after the original pretraining run. Different from SFT, which uses labeled chat… Read full definition →
-
Contrastive Learning
Contrastive learning trains encoders so related pairs (augmentations, dual views, text–image matches) land close in embedding space and unrelated pairs stay… Read full definition →
-
Convolutional Neural Network
(CNN)
Convolutional Neural Network: A CNN applies shared local filters (convolutions) across space or time, building hierarchical feature maps with translation… Read full definition →
-
Cosine Similarity
Cosine similarity is a metric used to measure how similar two vectors are by computing the cosine of the angle between them. It is widely used in machine… Read full definition →
-
Cross-Entropy
Cross-Entropy: Cross entropy measures how well a predicted probability distribution matches a target distribution. As a loss, classification compares softmax… Read full definition →
-
DPO
(Direct Preference Optimization)
DPO: Direct Preference Optimization (DPO) aligns a model to preference pairs (chosen vs rejected responses) without training a separate reward model or running… Read full definition →
-
Decoder
(Target Decoder)
Decoder: In an encoder–decoder system, the decoder generates the target sequence one step at a time (for example French words), conditioned on what the encoder… Read full definition →
-
Decoder-Only
Decoder-Only: Decoder only models (GPT style) stack Transformer blocks with a causal mask so each token attends only leftward. The same stack handles… Read full definition →
-
Decoding
Decoding is the algorithm that turns model logits into a concrete token sequence at inference time: greedy argmax, sampling (with temperature, top k , top p ),… Read full definition →
-
Deep Learning
Deep learning is machine learning with multi layer neural networks that learn hierarchical representations from data, reducing hand built features for many… Read full definition →
-
Dense Retrieval
Dense retrieval encodes queries and documents into vectors with an embedding model, then finds nearest neighbors—usually via ANN indexes. Dual encoders keep… Read full definition →
-
Dependent Variable
(Response Variable, Output Variable, Target Variable)
Dependent Variable: A dependent variable is the quantity a model or experiment aims to explain or predict. Its value is treated as a function of one or more… Read full definition →
-
Detection Head
(Object Detection Head, Box Head)
Detection Head: A detection head is the output module of a classical object detector (YOLO, RetinaNet, Faster R CNN). It predicts box geometry and class scores… Read full definition →
-
Domain Adaptation
Domain adaptation specializes a model to a target distribution—legal, medical, code, or your company’s docs—that differs from pretraining data. The goal is… Read full definition →
-
Dropout
Dropout randomly zeroes activations during training so the network cannot rely on brittle co adaptations. At evaluation time, dropout is disabled (or inverted… Read full definition →
-
Early Stopping
Early stopping ends training when a validation metric stops improving for a patience window, then reloads the best checkpoint. It is a simple, strong… Read full definition →
-
Embedding
Embedding: An embedding is a learned representation of data as a dense continuous vector. It maps high dimensional, discrete, or symbolic inputs (words,… Read full definition →
-
Embedding Model
Embedding Model: An embedding model maps text (or other modalities) to fixed size vectors optimized for similarity tasks. In a RAG stack it is usually separate… Read full definition →
-
Encoder
(Source Encoder)
Encoder: In an encoder–decoder system, the encoder reads the source sequence (for example the English sentence in translation) and builds representations of… Read full definition →
-
Encoder–Decoder
(Seq2Seq Architecture, Encoder-Decoder)
Encoder–Decoder: An encoder–decoder model splits sequence work in two: 1. The encoder reads the source and builds representations (often bidirectional). 2. The… Read full definition →
-
Epoch
Epoch: An epoch is one full pass over the training dataset. LLM continued pretraining is often measured in epochs over your domain corpus or in optimizer… Read full definition →
-
Evaluation
(Eval)
Evaluation measures whether a model meets task goals: accuracy, factuality, latency, safety, and user preference. Automated benchmarks plus human or rubric… Read full definition →
-
FAISS
FAISS (Facebook AI Similarity Search) is a library of exact and approximate vector search indexes on CPU/GPU. It is a common building block under custom RAG… Read full definition →
-
Feature Engineering
Feature engineering constructs input variables that help classical ML models learn—scaling, encodings, interactions, domain aggregates, and leak free time… Read full definition →
-
Feed-Forward Network
(FFN, MLP Block)
Feed-Forward Network: In Transformers, the feed forward network (position wise MLP) follows attention in each block. It expands each token’s hidden state to a… Read full definition →
-
Few-Shot
(Few-Shot Learning, N-Shot)
Few-Shot: Few shot prompting includes a handful of input–output demonstrations in the prompt so the model infers the pattern. It sits between zero shot (no… Read full definition →
-
Fine-Tuning
Fine-Tuning: Fine tuning adapts a pretrained model to a downstream task or domain by continuing training on task specific data. It can mean full weight updates… Read full definition →
-
Flash Attention
Flash Attention: FlashAttention is an IO aware exact attention algorithm that tiles computation to reduce HBM reads/writes. It speeds up training and inference… Read full definition →
-
GGUF
GGUF is a file format commonly used with llama.cpp style runtimes to ship quantized LLM weights for local CPU/GPU inference. It packages tensors plus metadata… Read full definition →
-
Gradient Accumulation
Gradient accumulation runs several forward/backward passes, summing gradients, then applies one optimizer update. It approximates a larger batch when the true… Read full definition →
-
Gradient Checkpointing
(Activation Checkpointing)
Gradient checkpointing saves memory by not storing all intermediate activations: during backward, selected layers are recomputed from saved checkpoints. You… Read full definition →
-
Gradient Descent
(GD)
Gradient descent minimizes a scalar loss by repeatedly stepping parameters opposite the gradient: \theta \leftarrow \theta \eta \nabla \theta L . The learning… Read full definition →
-
Grounding
Grounding means tying model outputs to verifiable sources—retrieved documents, tools, or user provided context—rather than unconstrained parametric recall.… Read full definition →
-
Guardrails
Guardrails are policies and filters around an LLM—input screening, output validation, topic allowlists, PII redaction, and tool permission checks. They… Read full definition →
-
HNSW
(Hierarchical Navigable Small World)
HNSW is a popular graph based ANN index: multi layer proximity graphs enable roughly logarithmic time search with strong recall. It is widely used inside… Read full definition →
-
Hallucination
Hallucination: The generation of output by a model that is not grounded in the input data or real world facts. Types of Hallucinations: 1. Factual… Read full definition →
-
Hard Prompt
Hard Prompt: A hard prompt is ordinary natural language text (and special tokens) fed to a model—the human readable prompt string. Unlike a soft prompt, every… Read full definition →
-
Hybrid Search
Hybrid search combines sparse and dense retrieval—typically BM25 plus dense vectors—via score fusion or reciprocal rank fusion (RRF). You get lexical precision… Read full definition →
-
In-Context Learning
(ICL)
In-Context Learning: In context learning is the ability of LLMs to adapt to a task from examples or instructions inside the prompt, without updating weights.… Read full definition →
-
Independent Variable
(Predictor, Feature, Input Variable, Explanatory Variable)
Independent Variable: An independent variable is an input used to explain or predict a dependent variable. In ML it is usually called a feature or predictor :… Read full definition →
-
Inference
Inference is running a trained model to produce outputs—as opposed to training. For LLMs this means autoregressive decoding with sampling or search strategies… Read full definition →
-
Instruct Model
(Chat Model)
Instruct Model: An instruct (chat) model is a base LM further tuned to follow instructions and multi turn chat formats. It expects templated roles (chat… Read full definition →
-
Instruction Tuning
Instruction tuning fine tunes a base LM on diverse (instruction, response) pairs so it follows natural language commands instead of only continuing raw text.… Read full definition →
-
KV Cache
(Key-Value Cache)
KV Cache: During autoregressive decoding, attention keys and values for past tokens are reused. The KV cache stores them so the model does not recompute the… Read full definition →
-
Knowledge Distillation
(Distillation)
Knowledge distillation trains a smaller student to mimic a larger teacher’s outputs or hidden states, compressing capability into a cheaper model. Classic… Read full definition →
-
LLM Chaining
LLM chaining connects multiple model calls so each step’s output feeds the next. Used to split research, drafting, and formatting into controlled stages. Read full definition →
-
LM Head
(Final Vocab Projection, Vocabulary Projection, Output Projection, Unembedding)
LM Head: The LM head (final vocab projection) is the last linear layer of a causal language model. It maps each position’s hidden vector to a vector of logits,… Read full definition →
-
LSTM
(Long Short-Term Memory)
LSTM cells use gates (input, forget, output) to control information flow through a memory cell, mitigating vanishing gradients on long sequences versus vanilla… Read full definition →
-
Latent Space
Latent Space: A latent space is the continuous representation space where a model encodes inputs. Geometry in this space—clusters, directions,… Read full definition →
-
Layer Normalization
(LayerNorm)
Layer normalization re centers and re scales activations across features for each example, stabilizing deep transformer training. Unlike batch norm, it does… Read full definition →
-
Learning Rate
(LR)
Learning Rate: The learning rate scales how far parameters move per optimizer step. Too high → unstable or divergent training; too low → painfully slow… Read full definition →
-
LoRA
(Low-Rank Adaptation)
LoRA (Low Rank Adaptation) freezes a pretrained model and injects small trainable low rank matrices into selected layers (often attention and MLP projections).… Read full definition →
-
Logit
Logit: A logit is a raw, unnormalized score output by a model before softmax (or similar) converts scores to probabilities. In LMs, the LM head emits one logit… Read full definition →
-
Loss Function
(Objective Function)
Loss Function: A loss function quantifies prediction error during training. Optimization (gradient descent / Adam style) minimizes expected loss on the… Read full definition →
-
MLP
(Multi-Layer Perceptron, Multilayer Perceptron)
MLP: An MLP ( multi layer perceptron ) is a stack of fully connected layers: each layer is a linear map (matrix × vector + bias) followed by a nonlinear… Read full definition →
-
Machine Learning
(ML)
Machine learning (ML) builds systems that improve at a task from data rather than purely hand coded rules. You choose a model class, a loss (or reward), and an… Read full definition →
-
Masked Model
(Masked Language Model, MLM)
Masked Model: A masked model (masked language model, MLM) recovers hidden tokens using left and right context. BERT-style encoders; contrast with causal LMs. Read full definition →
-
Massive Multitask Language Understanding
Massive Multitask Language Understanding: It is a benchmark in machine learning and natural language processing (NLP) used to evaluate the general language… Read full definition →
-
Matmul
(Matrix Multiplication, Matrix Multiply, GEMM)
Matmul: A matmul ( matrix multiplication , GEMM) multiplies two matrices (or a matrix and a vector): entry (i,j) of AB is the dot product of row i of A with… Read full definition →
-
Matrix Embedding (Embedding Matrix)
Matrix Embedding (Embedding Matrix): Matrix embedding vs embedding matrix: stacked token vectors (seq_len × dim) after lookup, or the |V| × dim weight table… Read full definition →
-
Mini-Batch
Mini-Batch: A mini batch is a small subset of training examples used to estimate the gradient in one optimizer step. Full batch GD is rare for deep nets; pure… Read full definition →
-
Mixed Precision
(AMP, FP16 Training, BF16)
Mixed precision training runs most matmuls and activations in FP16 or BF16 while keeping a higher precision master copy of weights (typically FP32) for stable… Read full definition →
-
Mixture of Experts
(MoE)
Mixture of Experts (MoE) replaces (or augments) a dense feed forward network with many expert FFNs plus a router that sends each token to a small subset (often… Read full definition →
-
Multi-Head Attention
(MHA)
Multi-Head Attention: Multi head attention (MHA) runs several attention operations in parallel (“heads”), each with its own Q,K,V projections. Head outputs are… Read full definition →
-
Multicollinearity
Multicollinearity is a statistical issue in regression analysis where two or more independent (predictor) variables are highly correlated with each other. What… Read full definition →
-
Multimodal
(Multi-Modal)
Multimodal models accept or produce more than one modality — text, images, audio, video. Typical designs encode each modality into a shared or aligned space,… Read full definition →
-
Neural Network
(Neural Net, NN)
Neural Network: A neural network is a parameterized function built from layered transformations — usually linear maps plus nonlinear activations — trained to… Read full definition →
-
Next-Token Prediction
(Autoregressive Language Modeling)
Next-Token Prediction: Next token prediction trains a model to estimate P(x t \mid x {<t}) for each position. The training loss is usually average cross… Read full definition →
-
Normalization
Normalization rescales data or activations so magnitudes stay comparable — e.g., z score / min max on features, or LayerNorm / RMSNorm inside networks. Where… Read full definition →
-
Open-Source Model
(Open Source Model, Open-Source Models, Open-Weight Model, OSS LLM)
Open-Source Model: An open source model (often open weight ) is a language or multimodal model whose weights are publicly downloadable under a stated license,… Read full definition →
-
Optimizer
Optimizer: An optimizer turns gradients into parameter updates. Basics: gradient descent and SGD (with optional momentum). Adaptive methods (Adam, AdamW) track… Read full definition →
-
Overfitting
Overfitting means the model fits training idiosyncrasies and generalizes poorly. Classic signal: train loss keeps falling while validation / held out metrics… Read full definition →
-
PEFT
(Parameter-Efficient Fine-Tuning)
PEFT: Parameter efficient fine tuning (PEFT) updates only a small subset of parameters — or adds small modules — while freezing most of the pretrained network.… Read full definition →
-
Parameter
(Weights, Model Weights)
Parameters are the learned weights (and biases) of a model — the numbers updated by the optimizer. LLM size is often quoted as parameter count (7B, 70B),… Read full definition →
-
Perplexity
(PPL)
Perplexity (PPL) is \exp(\text{average negative log likelihood}) over tokens — equivalently \exp(\text{cross entropy}) in nats. Lower means the causal LM is… Read full definition →
-
Policy
Policy: In reinforcement learning, a policy \pi(a \mid s) maps states to a distribution over actions. Learning improves expected return under that policy. In… Read full definition →
-
Positional Encoding
Positional Encoding: Self attention is permutation invariant: without position signals, the model cannot tell word order. Positional encoding injects order… Read full definition →
-
Pretraining
Pretraining is the large scale initial training phase where a model learns general representations from massive unlabeled (or weakly labeled) data — e.g., next… Read full definition →
-
Prompt
(User Prompt)
Prompt: A prompt is the text (token sequence) you give a language model as input before it generates a continuation. In the simplest case it is a short prefix;… Read full definition →
-
Prompt Engineering
Prompt engineering is the craft of writing and structuring inputs so a frozen pretrained model does what you want—without updating weights. It is product logic… Read full definition →
-
Prompt Injection
Prompt injection is an attack where untrusted content — a web page, email, uploaded doc, or tool result — contains instructions that try to override the system… Read full definition →
-
Prompt Tuning
Prompt tuning adapts a frozen LLM by training a small soft-prompt embedding prepended to inputs. Only those prompt parameters are updated. Definition from the… Read full definition →
-
QLoRA
(Quantized LoRA)
QLoRA fine tunes LoRA adapters on a base model stored in low bit quantization (commonly 4 bit, e.g. NF4). The frozen base stays memory cheap; adapters and… Read full definition →
-
Quantization
Quantization reduces the numeric precision of model weights (and sometimes activations)—e.g., float16/bfloat16 down to int8, int4, or other low bit formats—to… Read full definition →
-
RAG
(Retrieval-Augmented Generation)
RAG: Retrieval augmented generation (RAG) retrieves relevant documents at query time and conditions the LLM on that context, instead of relying only on… Read full definition →
-
RLHF
(Reinforcement Learning from Human Feedback)
RLHF trains a reward model from human preference comparisons, then optimizes the generative policy (the LLM) with reinforcement learning (often PPO) to… Read full definition →
-
RNN
(Recurrent Neural Network)
RNN: A recurrent neural network (RNN) is the classic sequence model: it processes tokens step by step and passes a hidden state through time. Vanilla RNNs… Read full definition →
-
ReLU
(Rectified Linear Unit)
ReLU (rectified linear unit) is f(x) = \max(0, x) . It is cheap, induces sparsity, and historically unlocked deeper nets versus saturating sigmoids/tanhs.… Read full definition →
-
Regularization
Regularization reduces overfitting by constraining capacity or injecting noise so the model prefers simpler or more robust solutions. Common tools L2 / weight… Read full definition →
-
Reinforcement Learning
(RL)
Reinforcement learning (RL) trains an agent to maximize expected reward through interaction with an environment. A policy maps states to actions; learning uses… Read full definition →
-
Representation Learning
Representation learning discovers features — embeddings, hidden states, or codes — that make downstream tasks easier than working from raw inputs. How it’s… Read full definition →
-
Reranking
(Reranker)
Reranking rescores an initial candidate set of retrieved passages with a stronger model (often a cross encoder) so the LLM sees the best evidence first. Two… Read full definition →
-
Residual Connection
(Skip Connection)
Residual Connection: A residual (skip) connection adds a block’s input to its output: y = x + F(x) . The network learns a residual F instead of a full… Read full definition →
-
Sampling
Sampling draws the next token from the model’s probability distribution (after optional shaping), introducing randomness into generation. Contrast with greedy… Read full definition →
-
Scaling Laws
Scaling laws are empirical relationships between model size (parameters), dataset size, compute, and loss / downstream performance. They guide how to allocate… Read full definition →
-
Self-Attention
Self-Attention: Self attention lets each position in a sequence compute a weighted combination of all positions (including itself). Queries, keys, and values… Read full definition →
-
Self-Supervised Learning
(SSL)
Self-Supervised Learning: Self supervised learning (SSL) creates training signals from the input itself — predict masked spans, next tokens, or agreement… Read full definition →
-
Semantic Search
Semantic search retrieves documents by meaning: a query and each passage are mapped to dense embeddings, then nearest neighbors are ranked by similarity (often… Read full definition →
-
Sequence Model
(Sequential Model, Sequence Model (RNN-style))
Sequence Model: A sequence model processes ordered inputs one step at a time (or in a stack that respects order) and keeps a running summary of what it has… Read full definition →
-
Sequence-to-Sequence
(Seq2Seq)
Sequence-to-Sequence: Sequence to sequence (seq2seq) models map an input sequence to an output sequence of possibly different length—translation,… Read full definition →
-
Soft Prompt
Soft prompt (soft prompting): a learned embedding vector sequence prepended to the input — not readable tokens. Optimized in prompt tuning while the base model… Read full definition →
-
Softmax
Softmax Softmax is an activation function used in multi class classification to convert a vector of real valued scores (logits) into a probability distribution… Read full definition →
-
Sparse Retrieval
Sparse retrieval scores documents with high dimensional sparse signals—typically term frequencies and document statistics (BM25, TF–IDF). Each document is a… Read full definition →
-
Speculative Decoding
Speculative decoding speeds up LLM inference without changing the target model’s distribution (when implemented correctly). A small draft model (or draft… Read full definition →
-
Stochastic Gradient Descent
(SGD)
Stochastic gradient descent estimates the gradient on a subset of data (a mini batch) rather than the full dataset each step. That noise scales training to… Read full definition →
-
Supervised Fine-Tuning
(SFT)
Supervised Fine-Tuning: Supervised fine tuning (SFT) updates a model on labeled input–output pairs—often instruction/response chats. It teaches format ,… Read full definition →
-
Supervised Learning
Supervised learning trains on labeled examples (x, y) to predict y from x —classification, regression, or sequence labeling. The loss compares predictions to… Read full definition →
-
Synthetic Data
Synthetic data is generated by models or simulators for training or eval—e.g., LLM written instruction pairs, rewritten documents, or simulated tool traces. It… Read full definition →
-
System Prompt
System Prompt: A system prompt is a high priority instruction channel (in chat APIs) that sets role, style, tools, and safety policy for the assistant across… Read full definition →
-
Teacher Forcing
(Teacher-Forcing)
Teacher forcing is a training trick for models that generate one token at a time: at each step, feed the decoder the correct previous token from the reference,… Read full definition →
-
Temperature
Temperature is a scalar value (usually between 0 and 2) used during the sampling process from a probability distribution to control the level of randomness in… Read full definition →
-
Tensor
Tensor: In machine learning (ML), a tensor is a generalization of scalars, vectors, and matrices to higher dimensions and is a core data structure used to… Read full definition →
-
Token
Token: In Natural Language Processing (NLP), a token is a basic unit of text used for processing and analysis. It typically represents a word, subword,… Read full definition →
-
Tokenizer
Tokenizer: A tokenizer converts raw text into a sequence of token IDs the model was trained with (and back). Subword schemes (BPE, WordPiece, Unigram) dominate… Read full definition →
-
Tool Calling
(Function Calling)
Tool calling (function calling) lets a model emit structured requests to external functions—calendars, databases, calculators, retrieval—which the runtime… Read full definition →
-
Top-k
Top-k: Top k sampling restricts next token choices to the k highest probability tokens after the softmax, then samples among them (usually after temperature… Read full definition →
-
Top-p
(Nucleus Sampling)
Top-p: Top p (nucleus) sampling keeps the smallest set of tokens whose cumulative probability mass exceeds p (e.g., 0.9), then samples from that set—usually… Read full definition →
-
Transfer Learning
Transfer learning reuses knowledge from a model trained on one task or domain to improve learning on another—typically via pretrained weights plus fine tuning.… Read full definition →
-
Transformer
(Transformer Architecture)
Transformer: The Transformer is a neural network architecture introduced in Attention Is All You Need (Vaswani et al., 2017). It replaces recurrence with self… Read full definition →
-
Underfitting
Underfitting means the model is too weak or under trained to capture the signal—both train and validation metrics stay poor. Unlike overfitting, the gap… Read full definition →
-
Unsupervised Learning
Unsupervised learning finds structure in unlabeled data—clustering, density estimation, dimensionality reduction—without explicit target labels y . Classical… Read full definition →
-
Vector Database
(Vector DB, Vector Store)
Vector Database: A vector database stores high dimensional embeddings and supports similarity search—often via approximate nearest neighbor indexes such as… Read full definition →
-
Vision-Language Model
(VLM, Vision Language Model)
Vision-Language Model: A vision language model (VLM) takes an image (and usually a text prompt) and generates text. Typical stack: a vision encoder turns the… Read full definition →
-
Warmup
Warmup: Learning rate warmup starts training with a smaller learning rate and ramps up over the first N steps before the main decay schedule (linear, cosine,… Read full definition →
-
Weight Decay
Weight decay penalizes large weights (L2 style), encouraging simpler solutions and reducing overfitting. Conceptually it pulls parameters toward zero each… Read full definition →
-
Word Embedding
(Static Embedding)
Word Embedding: A word embedding maps a word (or subword) to a real valued vector so that semantic similarity becomes geometric closeness in the vector space.… Read full definition →
-
Zero-Shot
Zero-Shot: In the context of machine learning (especially few shot learning) and prompt engineering for large language models (LLMs) , zero shot , 1 shot (one… Read full definition →