How to read the math in arXiv AI papers: a practical guide

Viacheslav Dubrov

2026-11-03 · 54 min · Model Engineering

Reading the mathematics in an AI paper is a useful skill for engineers who need to understand a model, reproduce a method or assess an experimental claim. This guide explains how to decode the notation, check an equation on small numbers and connect it to the surrounding paper and implementation. It assumes you can read code, but does not assume that research notation is already familiar.

An equation compresses several decisions into one line: the objects being manipulated, their dimensions, which values are sampled, and what is learned or held fixed. Recovering those details is a good first step. Explaining why a method works and implementing it faithfully take further work.

Start with the attention walkthrough: it connects one equation to array shapes, small numbers and runnable Python. The notation reference explains the expressions you will meet in other papers. The rest of this series applies the same method to equations from specific papers: architecture equations such as RoPE, MLA and LoRA; training objectives such as DPO and GRPO; and generative models and metrics such as the ELBO, diffusion and pass@k. Those parts come out on 4, 5 and 6 November; until then, the walkthroughs this part refers to are not linked. For a longer treatment of the foundations, Tianhua Chen’s The Little Book of Generative AI Foundations develops the linear algebra and probability behind generative models.

Attention, read one symbol at a time

Attention produces an output vector by taking a weighted average of value vectors. It obtains the weights by comparing a query vector for the output position with a key vector for each available input position. The matrices $Q$, $K$ and $V$ collect those query, key and value vectors. Scaled dot-product attention combines these steps in equation (1) of §3.2.1 (§ means “section”) in Attention Is All You Need:

The definitions are in the text, not in the equation. The paragraph before equation (1) says that queries and keys have dimension $d_k$ and values have dimension $d_v$. The method takes the dot product of each query with all keys, divides each by $\sqrt{d_k}$ and applies a softmax “to obtain the weights on the values”. Queries are “packed together into a matrix $Q$”, and keys and values into $K$ and $V$. So each row of these matrices is one token’s vector, and the softmax produces one set of weights per query. The learned matrices that turn token vectors into $Q$, $K$ and $V$ appear later, in §3.2.2.

Shapes and roles. An attention head performs one such calculation with its own query, key and value vectors. Write down their shapes before reading the operators. For one head with $n$ queries and $m$ keys:

Papers give a shape by naming the set the matrix belongs to: $Q\in\mathbb R^{n\times d_k}$ means a matrix of real numbers with $n$ rows and $d_k$ columns, an array of shape [n, dk].

Read from the inside out.

1. $QK^\top$ multiplies an $n\times d_k$ matrix by a $d_k\times m$ matrix. The shared $d_k$ is summed away, so the result is an $n\times m$ table of scores. Entry $(i,j)$ is the dot product of query $i$ and key $j$: multiply corresponding entries, then add the products. For fixed vector lengths, this score is larger when their directions are more similar.

2. Dividing by $\sqrt{d_k}$ stops the scores from growing with the vector width. Footnote 4 of the paper shows why unscaled dot products grow. Suppose the components of a query and a key are independent of one another (knowing some of them tells you nothing about the others), each with mean 0 and variance 1 (the average squared distance from the mean). Then:

   $$\begin{aligned} \text{one product }q_ik_i:&\quad\text{variance }1\\ q^\top k=\textstyle\sum_{i=1}^{d_k}q_ik_i:&\quad\text{variance }d_k,\ \text{standard deviation }\sqrt{d_k}\\ q^\top k/\sqrt{d_k}:&\quad\text{standard deviation }1 \end{aligned}$$

   The lines use two rules about independent numbers. First, the product of two independent numbers that each have mean 0 and variance 1 also has variance 1. Second, when independent numbers are added, their variances add, so the $d_k$ products give variance $d_k$. The standard deviation is the square root of the variance, $\sqrt{d_k}$. Dividing every score by $\sqrt{d_k}$ shrinks the spread by the same factor, back to 1.

   Why the two rules hold

   Variance when the mean is 0. Variance is the average squared distance from the mean. When the mean is 0, the squared distance from the mean is just the square of the number, so the variance is the average of the square. A component $q_i$ with mean 0 and variance 1 therefore has $q_i^2$ averaging 1.

   The average of a product. For independent numbers, the average of a product is the product of the averages. Take a number $a$ that is 1 or 3 with equal chance, so it averages 2, and a number $b$ that is 1 or 5, also with equal chance, so it averages 3. Independent means that $b$ takes each of its values equally often whatever $a$ turned out to be, so all four pairs are equally likely. When $a=1$, the products are 1 and 5, which average $3=1\times3$. When $a=3$, they are 3 and 15, which average $9=3\times3$. Either way the average is $a\times3$, so over both values of $a$ it is $(3+9)/2=6$, which is $2\times3$. Without independence the rule fails: if $b$ were 5 whenever $a$ is 3, and 1 whenever $a$ is 1, only the products 1 and 15 would occur, and they average 8. Wikipedia states the rule for any independent random variables, with “expected value” for the average: independence and expectation.

   Rule 1: one product has variance 1. Take one product $q_ik_i$. Its mean is the product of the averages, $0\times0=0$, so its variance is the average of $(q_ik_i)^2=q_i^2k_i^2$. The squares are independent because $q_i$ and $k_i$ are, and each averages 1, so the variance is $1\times1=1$. The general formula is the variance of a product of independent variables.

   Rule 2: the variances of independent terms add. Call the products $x_i=q_ik_i$, so the dot product is the sum $x_1+\cdots+x_{d_k}$. The terms are independent because each uses different components, and each has mean 0 and variance 1. The sum has mean 0 too, so its variance is the average of its square. Multiplying the sum by itself gives one product $x_ix_j$ for every pair of terms, a $d_k\times d_k$ grid:

   

   The diagonal cells are the squares $x_i^2$, and each averages 1. Every other cell multiplies two different, independent terms, so its average is the product of their averages, $0\times0=0$: its positive and negative values cancel. Only the $d_k$ diagonal cells are left, so the variance of the sum is $d_k$ and its standard deviation is $\sqrt{d_k}$. The spread grows with the square root of the width because most of the grid cancels out. Wikipedia gives the general statement as the variance of a sum of uncorrelated variables; independent variables are always uncorrelated, so the rule covers them.

   With $d_k=64$, raw scores typically spread over about $\pm8$; scaled, over about $\pm1$, at any width.

   Widely spread scores saturate softmax (step 3), and its gradients vanish. If one score in a row is far above the others, softmax gives that key a weight near 1 and every other key a weight near 0. A small change to any score then barely moves the weights, so the gradient that reaches the queries and keys is close to zero, and training has almost nothing to adjust them with. The paper offers this as the likely explanation of an earlier result: without the scaling, dot-product attention did worse than additive attention, an older form that scores a query and a key with a small network, at large $d_k$ (Britz et al., 2017). The paper does not measure the gradients itself.

3. Softmax turns each row of scores into weights. Let $z=(z_1,\ldots,z_m)$ be one row of the scaled scores. The exponential $e^a$, also written $\exp(a)$, uses the constant $e\approx2.718$. Then $\operatorname{softmax}(z)_j=e^{z_j}/\sum_{j'}e^{z_{j'}}$, where $j'$ is just a second index name, so the sum runs over every key in the row. The weights are positive and each row sums to 1.

4. Multiplying the $n\times m$ weights by the $m\times d_v$ values sums away the key axis. Row $i$ of the $n\times d_v$ output is a weighted average of the value rows, weighted by how well each key matched query $i$.

Count the squares in the figure: each product removes the axis its two grids share.

Check it with two numbers. Take one query and two keys whose scaled scores are $(0,\log3)$, chosen so that the exponentials come out as 1 and 3, and values $v_1=(2,0)$ and $v_2=(0,4)$. The figure works through it one operation per column:

Attention blended both values; it did not pick the better-matching token.

Compare it with code. The NumPy function below implements one head. k.T is the transpose. axis=1 is the key axis, so each query’s row is normalized separately. The row maximum is subtracted before np.exp: adding the same number to every score in a row cancels in the softmax ratio, so the result is unchanged, but large scores no longer overflow. The optional mask sets blocked scores to $-\infty$, which makes their weights exactly zero. A decoder uses such a mask to stop each token from attending to later tokens (§3.2.3 of the paper).

Save the block as attention_example.py and run uv run --with numpy python attention_example.py. The checks pass silently.

import numpy as np

def attention(q, k, v, allowed=None):
    """One head. q: [n, dk], k: [m, dk], v: [m, dv], allowed: bool [n, m]."""
    scores = q @ k.T / np.sqrt(q.shape[1])  # [n, m]
    if allowed is not None:
        if not allowed.any(axis=1).all():
            raise ValueError("every query needs at least one allowed key")
        scores = np.where(allowed, scores, -np.inf)
    scores = scores - scores.max(axis=1, keepdims=True)
    weights = np.exp(scores)
    weights = weights / weights.sum(axis=1, keepdims=True)  # rows sum to 1
    return weights @ v, weights  # [n, dv], [n, m]

# The two-key example from the text.
out, w = attention(
    np.array([[1.0]]),
    np.array([[0.0], [np.log(3.0)]]),
    np.array([[2.0, 0.0], [0.0, 4.0]]),
)
np.testing.assert_allclose(w, [[0.25, 0.75]])
np.testing.assert_allclose(out, [[0.5, 3.0]])

# A causal mask: query i may attend only to keys 0..i.
x = np.random.default_rng(0).normal(size=(3, 4))
out, w = attention(x, x, x, allowed=np.tril(np.ones((3, 3), dtype=bool)))
assert out.shape == (3, 4)
np.testing.assert_allclose(w.sum(axis=1), 1.0)
assert np.all(w[np.triu_indices(3, k=1)] == 0)

A query whose keys are all blocked is a case the equation leaves open: every weight in its row would be $0/0$. The function raises an error instead of returning NaN.

From one head to several. The base Transformer runs eight heads in parallel, each with its own learned matrices for producing $Q$, $K$ and $V$, and each with $d_k=d_v=64$. It concatenates their outputs and multiplies the result by another learned matrix, $W^O$ (Attention Is All You Need, §3.2.2).

An implementation can also process a batch: several input sequences together. For example, PyTorch’s scaled dot-product attention accepts a query tensor Q with shape [batch, heads, n, d_k]. The first two dimensions select a sequence and a head; Q[b, h] is the [n, d_k] matrix for that pair. The attention formula applies separately to each pair. The NumPy function here handles only one such pair: its .T and axis=1 operations assume two-dimensional inputs.

Six questions for any equation

What kind of statement an equation makes

Attention is a computation: it says how to get an output from inputs. Papers also write equations that do other jobs:

A constraint states a property the result must have. For attention weights, that property is: every weight is nonnegative and each row sums to 1. Softmax is a computation that produces weights with that property. The requirement alone does not tell you to use softmax.

$=$ asserts that two expressions are equal. $:=$ means “is defined as”: it gives a name to an expression. A definition can name a metric, a loss or a set of allowed values, so seeing $:=$ does not tell you which kind of equation in the table above you are reading.

For example, Evaluating Large Language Models Trained on Code, the paper that introduced the Codex code-generation model, uses $:=$ to define its estimator for pass@k (equation 1). This metric measures the chance that at least one of $k$ generated programs passes the tests. In the DeepSeek-V4 technical report, equation (2) combines two statements: $\mathcal M:=\{\ldots\}$ defines a set of allowed matrices, and $B_l\in\mathcal M$ requires the model’s mixing matrix $B_l$ to belong to that set. The definition uses $:=$; the constraint uses $\in$, meaning “belongs to”. The pass@k and mHC sections work through those examples.

Finding the new equation, its definitions and its code

A paper’s methods section describes how its model or training algorithm works. The heading may be “Method”, “Architecture” or something more specific. Authors often explain an existing method first, then show what they changed. To identify the claimed contribution, start with the introduction’s summary of what is new. For example, DeepSeekMath, §1.1 lists Group Relative Policy Optimization (GRPO), its method for training models from scored answers. Section and equation numbers here refer to the linked paper.

Then compare the proposed method with the earlier method it changes. Two papers illustrate this:

• An architecture change. DeepSeek-V2 describes standard multi-head attention in §2.1.1 and introduces multi-head latent attention (MLA) in §2.1.2. Standard attention stores every earlier token’s keys and values. MLA stores a compressed vector from which they can be reconstructed. Comparing those equations reveals what the model stores differently; the MLA walkthrough derives the memory saving.
• A training change. DeepSeekMath, §4.1.1 prints the objective of Proximal Policy Optimization (PPO) as equation (1) and GRPO as equation (3). Both use an advantage: a number that says whether an answer was better or worse than expected. PPO estimates it with a second learned network, the value model, which predicts the reward still to come from a partial answer. GRPO instead compares the answer’s reward with rewards for other answers to the same question. The GRPO walkthrough compares these terms using the whole-answer notation from DeepSeek-R1.

Find the definitions around the equation. Check the preceding paragraph, any “where …” sentence after it, footnotes and appendices. In Attention Is All You Need, the paragraph before equation (1) defines the dimensions, and footnote 4 explains the scale factor. DeepSeek-V3 defines most of its routing symbols right after equation (15), and the bias $b_i$ just before equation (16). DeepSeek-V2 develops MLA across equations (9)–(19), then collects the computation in Appendix C.

Check the algorithm and code. An algorithm box is a numbered pseudocode listing that gives the order of operations. DeepSeekMath’s Algorithm 1 shows when to generate answers and when to update the model. Algorithm 1 in Denoising Diffusion Probabilistic Models (DDPM) specifies how to sample the training image, noise level and noise. Implementations settle further details: by default, np.std divides the sum of squared deviations by $n$, while torch.std divides by $n-1$. The GRPO example shows how that choice changes the advantage values computed from the same rewards.

arXiv versions can move sections and repeat equations under new numbers, so cite the version. In DeepSeek-R1 v1, the GRPO objective is §2.2.1, equations (1)–(3). In v2, posted in January 2026, it is §2.1, and the same equations reappear as (11)–(13) in Supplementary A.3. A link without a version opens the latest one, so every arXiv link in this series includes the version it was checked against.

Checking your reading with an LLM

A chat model can draft the symbol table, the index of every sum and a small-number check, if you give it the definitions and ask it to flag what is missing. Text copied from a PDF flattens subscripts, superscripts and fractions into one line; the “HTML (experimental)” and “TeX Source” links on the arXiv abstract page give clean text to paste. Paste the equation and the paragraphs around it with a prompt like this:

Below is equation (N) from <paper title, arXiv ID and version>, followed by
the paragraphs that define its symbols.

1. Make a table with one row per symbol: shape or type, meaning, and whether
   it is input data, a trained weight, a frozen model, a fixed setting or a
   random sample. Quote the sentence that defines it.
2. List every symbol the pasted text does not define. Do not guess them.
3. For every sum, mean, softmax, min or expectation, name the index or the
   distribution it runs over.
4. Work through the equation with the smallest numbers that use every term.
5. Say which kind of statement it is: a computation, definition, objective,
   update rule, constraint, bound, estimator or empirical fit.

Without step 2, nothing stops the model from filling a missing definition with the letter’s usual meaning, such as reading $\sigma$ in DPO as a standard deviation instead of the sigmoid. The quotes from step 1 make the table checkable: search the PDF for each one, and treat a row whose quote you cannot find as a guess.

The same procedure as agent skills. paper-math-skills packages this section and the notation reference below as two skills written for Claude Code; other agents that load SKILL.md files can use them too. read-paper-math fetches the pinned arXiv version, quotes the sentence that defines each symbol, reads the equation from the inside out and runs a small-number check. check-paper-math reviews such a reading against the paper: it searches for every quote, re-runs the numbers and flags claims stronger or weaker than the paper’s. paper-math-readings has the skills’ readings for Parts 2–4, one for each of the 15 sections with a Source: line.

Notation reference

If a Greek letter is unfamiliar, open this article’s Greek-letter reference (Appendix A) for its name and common meanings. For more practice with the mathematics, Stanford CS229 provides probability and linear algebra reviews with exercises; Dive into Deep Learning’s preliminaries explain the foundations with code.

Subscripts, marks and relation signs

Letters are local variable names: each paper defines its own. Subscripts are usually indices. $x_i$ may be coordinate $i$ of a vector or example $i$ in a dataset, and $h_t$ is the model’s vector for token $t$, called its hidden state.

Superscripts in model papers are often labels. In $h_t^{(\ell)}$, the superscript in parentheses is usually a layer number, not a power; in DeepSeek-V3’s $\operatorname{FFN}^{(s)}_i$ and $\operatorname{FFN}^{(r)}_i$ it marks shared and routed experts, the ones every token uses and the ones a router picks (see the MoE routing example). In DeepSeek-V2, $W^{DKV}$ and $k_t^C$ carry labels, not exponents; the MLA example defines those labels. $W^\top$ is a transpose and $x^2$ is a square.

$x_{1:T}$ means positions 1 through $T$, including $T$, unlike a Python slice; papers usually count from 1. $x_{<t}$ means every position before $t$. $X_{t,:}$ selects row $t$, as in NumPy slicing.

Marks on a letter usually change its meaning in one of these ways:

Some symbols other than Greek letters also have several jobs:

Five signs say how two sides relate. $:=$, sometimes $\triangleq$, defines a new name. $\approx$ marks an approximation, such as the sampled average in Expectations and where the samples come from. $\propto$ means “proportional to”, with a normalizing constant left out: $p_i\propto e^{z_i}$ becomes a distribution only after dividing by $\sum_j e^{z_j}$. $\ll$ means “much smaller than”, as in DeepSeek-V2’s $d_c\ll d_hn_h$: the compressed width is much smaller than the total width of all heads. $\leftarrow$ is an assignment, as in an update rule.

Sums, counts, intervals and logarithms

The large $\sum$ adds terms and $\prod$ multiplies them. The index below each operator starts at 1 here and runs through the upper limit $N$:

The index belongs to the operator, as a loop variable belongs to its loop. In $y_i=\sum_{j=1}^{d}W_{ij}x_j$, the sum runs over $j$, while $i$ identifies an output entry. The result is a vector with one entry per $i$: the matrix-vector product $Wx$ written out. The indices on the left-hand side tell you the shape of the output.

The factor in front of a sum decides what gets equal weight. A batch with a 2-token answer and an 8-token answer, where $\ell_{it}$ is the loss on token $t$ of answer $i$ (in GRPO, $\ell$ is a term to maximize instead), can be averaged in two ways:

In the first, each answer gets weight $\tfrac12$, shared among its tokens:

In the second, every token has weight $1/10$. GRPO samples $G$ answers $o_1,\ldots,o_G$ to each question, and $\vert o_i\vert$ is the number of tokens in answer $i$. The two averages then read as follows, for DeepSeekMath’s GRPO and for DAPO, a later variant of GRPO:

With $G=2$, these are $L_{\text{answers}}$ and $L_{\text{tokens}}$ above. DAPO switched to the per-token average (§3.3) because the per-answer average gave the tokens of long answers too little weight: the model learned too little from long good answers and did not penalize long, repetitive bad ones enough.

A sum can also count. Papers do this with an indicator: a bold one followed by a condition, $\mathbf 1[C]$, which equals 1 when the condition $C$ is true and 0 when it is false. Some papers put the condition in parentheses or print a double-struck one instead; the meaning is the same. Adding up indicators therefore counts how many times the condition holds. Accuracy is the usual example. If $\hat y_i$ is the predicted class and $y_i$ the correct class for example $i$, then $\sum_i\mathbf 1[\hat y_i=y_i]$ is the number of correct predictions. Predictions (cat, dog, dog) against correct labels (cat, cat, dog) give the indicators 1, 0 and 1, so the count is 2 and the accuracy 2/3. In a mixture-of-experts (MoE) layer, a router sends each token to a few of many feed-forward sub-networks, called experts. In DeepSeek-V3’s balance loss (equation 18), the indicator is 1 when an expert’s score for a token is among the $K_r$ highest scores for that token, where $K_r$ is how many experts each token is routed to. Its sum over the tokens of a sequence counts how often that happened. The score here leaves out the bias that routing adds, so the count can differ from the experts the router actually picks (MoE routing).

The binomial coefficient $\binom nk$, read ”$n$ choose $k$”, counts the ways to choose $k$ distinct items from $n$ without regard to order. For integers $0\leq k\leq n$, it equals $n!/(k!\,(n-k)!)$. The exclamation mark means factorial: $n!=1\times2\times\cdots\times n$, with $0!=1$. For example, $\binom52=5!/(2!\,3!)=10$: five items have ten distinct pairs. DeepSeekMoE uses binomial coefficients to count expert combinations, and the pass@k estimator uses them to count sets of samples.

Intervals use brackets for included ends: $[a,b]$ includes both, $(a,b)$ neither. In Gated DeltaNet, $\alpha_t\in(0,1)$ means strictly between 0 and 1. Elsewhere, $(a,b)$ is a pair.

The natural logarithm reverses exponentiation: $\log(e^a)=a$. Logarithms turn products into sums and ratios into differences: $\log(ab)=\log a+\log b$ and $\log(a/b)=\log a-\log b$, but $\log(a+b)$ does not split. That is why papers work with log-probabilities. The log of a product of token probabilities is a sum, and the log of a ratio of two models’ probabilities is a difference, the form used in the DPO and GRPO objectives. ML papers mean the natural log unless they say otherwise; the results are in nats, and base-2 logs give bits. For a probability $p\leq1$, $-\log p\geq0$, and it grows without limit as $p$ approaches 0, so a confident wrong prediction is expensive.

Log-sum-exp means: exponentiate each score $z_i$, add the results, then take the logarithm. Computing $\log\sum_i e^{z_i}$ directly can overflow when a score is large. Instead, find the largest score and call it $m$. The notation $m=\max_i z_i$ means “compare all entries $z_i$ and take the largest one”; in NumPy, m = z.max().

Subtract $m$ from every score before exponentiating, then add it back after taking the logarithm:

For $z=(1000,1001)$, the largest score is $m=1001$. The shifted scores are $(-1,0)$, so the calculation becomes $1001+\log(e^{-1}+1)\approx1001.313$. Neither exponential is larger than 1. The attention code uses the same subtraction before softmax; there the common factor cancels in the division, so nothing is added back.

Conditional probability and the probability of text

$p_\theta(y\mid x)$ is the probability that the model with parameters $\theta$ assigns to output $y$ given input $x$. The subscript names the parameters; some papers write $p(y\mid x;\theta)$, where the semicolon also marks parameters rather than a condition. With a dot in place of the output, $p_\theta(\cdot\mid x)$ is the whole distribution over outputs rather than one number. RL papers write the same object as $\pi_\theta(o\mid q)$, the probability that the model writes answer $o$ to question $q$, and call the model a policy.

Swapping the event and the condition gives a different number. Suppose 100 model responses: 60 are correct, 50 are valid JSON, and 40 are both. Then

Let $A$ mean “the response is correct” and $B$ mean “the response is valid JSON”. Conditional probability restricts the count to responses satisfying the condition: $P(A\mid B)=P(A\cap B)/P(B)$, provided $P(B)>0$. The symbol $\cap$ means “both events happen”. Here the fraction is $0.4/0.5=0.8$, the same as 40 correct responses among 50 valid-JSON responses.

Two events are independent when knowing that one happened does not change the probability of the other. That requires $P(A\cap B)=P(A)P(B)$. In this sample, $P(\text{both})=0.4$, while $P(\text{correct})P(\text{valid JSON})=0.6\times0.5=0.3$, so the observed proportions do not satisfy independence.

Bayes’ theorem, also called Bayes’ rule, relates the two directions of conditioning: $P(A\mid B)=P(B\mid A)P(A)/P(B)$. It lets you compute the probability of $A$ given $B$ from the probability of $B$ given $A$ and the probabilities of the two events. The ELBO example uses it to find the probability of a hidden code given an observed image.

A language model scores a sequence one token at a time. By the chain rule of probability,

Each factor conditions on every earlier token, so the formula does not assume tokens are independent. If three tokens have conditional probabilities 0.5, 0.2 and 0.1:

The usual training loss is the negative of this sum, divided by the length:

During training, the earlier tokens $y_{<t}$ come from the dataset. During generation, they are the model’s own earlier choices.

Expectations and where the samples come from

An expectation is a probability-weighted average:

Read $\mathbb E_{x\sim p}[f(x)]$ as “the average of $f(x)$ when $x$ is drawn from $p$”. The notation $\sum_x$ sums over all possible discrete values of $x$. Without a subscript, $\mathbb E[X]$ averages a random variable $X$ over its own distribution: if $X$ is 1 with probability 0.75 and 3 with probability 0.25, then $\mathbb E[X]=1\times0.75+3\times0.25=1.5$.

When a variable has a density $p(x)$, meaning probability per unit of $x$, the weighted average is an integral, $\int f(x)p(x)\,dx$. The $dx$ identifies $x$ as the variable being integrated; with no limits written here, the integral covers its whole range. The density example explains probability per unit and shows an integral as an area under a curve.

Training code cannot sum over every possible input, so it samples and averages: $\mathbb E_{x\sim p}[f(x)]\approx\frac1N\sum_{i=1}^{N}f(x_i)$, with each $x_i$ drawn independently from $p$, which papers write as $x_i\overset{\text{iid}}{\sim}p$. Papers call this a Monte Carlo estimate, and in code it is loss.mean() over a batch. The batch mean estimates the expectation only if the batch is drawn from the distribution in the $\mathbb E$‘s subscript, so the subscript tells you how the training data is produced:

• DPO averages over saved examples $(x,y_w,y_l)\sim\mathcal D$: a prompt, a preferred answer and a rejected answer. The answers are collected and labeled before this training stage; they may come from earlier model generation. At each training step, the model computes probabilities for those supplied answers and updates its weights. It does not sample fresh answers for that step.
• GRPO in DeepSeek-R1: generate answers, then compare their rewards. Training draws a question from the dataset, generates several answers, and scores them with a reward function, such as checking a mathematical result or running unit tests. The update favors answers that scored above the group’s average. The notation $q\sim P(Q)$ means “sample a question $q$ from the question distribution”. The notation $\{o_i\}\sim\pi_{\theta_{old}}(O\mid q)$ means “sample answers $o_i$ to that question from the model copy named $\pi_{\theta_{old}}$”. The subscript old marks the copy that generated these answers; that copy stays fixed while the batch is used for updates. The GRPO walkthrough explains the reward comparison and updates.
• On-policy distillation in MiniLLM and DeepSeek-V4: generate text, then learn from teacher probabilities. The model being trained is the student. It generates text; a frozen teacher supplies next-token probabilities for the same prefixes, and training adjusts the student’s probabilities toward the teacher’s. These probabilities provide the training signal, rather than GRPO’s reward score for an answer. MiniLLM writes student-sampled text as $y\sim q_\theta$: $y$ is the text and $q_\theta$ names the student model. DeepSeek-V4 calls its student $\pi_\theta$ and uses several specialist teachers. MiniLLM’s actual sampling procedure also mixes in tokens drawn from the teacher. The distillation walkthrough explains the objectives and that sampling distinction.

The order tells you what training optimizes. Suppose a model assigns probabilities 0.9 and 0.1 to the correct outputs of two equally weighted training examples. Averaging their log-probabilities means taking each log first, then averaging: np.log(probs).mean(). Taking the log of their average means np.log(probs.mean()). If $X$ is the probability assigned to a randomly chosen example’s correct output:

Negating these scores gives losses of 1.20 and 0.69. A second model that assigns 0.5 to each correct output gets 0.69 under either calculation. Thus averaging probabilities before taking the log treats the two models as equally good, while the usual average negative log-probability penalizes the first model more for its low probability on the second correct output. When translating a loss into code, preserve the order of the log and the average.

Jensen’s inequality states the general relationship for positive $X$: $\mathbb E[\log X]\leq\log\mathbb E[X]$. It also helps construct alternative training objectives. When a model’s data probability requires an expensive average over hidden variables, an average of log terms can provide a lower bound that is easier to estimate. The ELBO walkthrough explains that training problem; ELBO stands for evidence lower bound. Section 2.4 of Tianhua Chen’s The Little Book of Generative AI Foundations derives the bound using Jensen’s inequality.

Densities, variance and error bars

$p(x)$ has two common meanings. For discrete outcomes such as tokens or classes, it gives the probability of outcome $x$: a number from 0 to 1, with probabilities summing to 1 over all outcomes.

To understand the other meaning, suppose $X$ is chosen uniformly between 0 and 0.5. Equal-length intervals have equal probabilities. The interval from 0.1 to 0.2 occupies one fifth of the whole range, so

Within this range, probability is the interval’s width multiplied by 2. That multiplier is the probability density: $p(x)=2$ for $0\leq x\leq0.5$, and zero outside. Density describes probability per unit of the variable, not the probability of one exact value.

Zero probability for a point does not mean nothing is sampled. Narrow an interval around 0.15: its probability is twice its width, so it approaches zero as the width approaches zero. The single exact point therefore has probability zero. A draw still produces some value in $[0,0.5]$, and that whole interval has probability 1. If you record the result rounded to two decimal places, a recorded value of 0.15 represents the interval from 0.145 to 0.155, whose probability is $2\times0.01=0.02$.

For a nonuniform density, the multiplier varies with $x$. The integral adds the contributions across the interval:

Geometrically, this is the area under the density curve between $a$ and $b$; the area over the whole range is 1. For the uniform example, the area is simply height 2 times width 0.5. A density can therefore exceed 1, and its log can be positive, as $\log2\approx0.69$ is here. A probability never exceeds 1.

The Gaussian distribution used for noise in DDPM has a density of this kind. Not every continuous distribution has an ordinary density: the Cantor distribution is a mathematical counterexample, outside the models discussed here. Real-valued storage does not imply a continuous distribution either: a variable restricted to 0.0 and 1.0 still has two discrete outcomes.

Papers often use the same letter for probabilities and densities. Check how the paper defines the variable’s distribution and which variable is summed or integrated. An integral over a hidden variable $z$, such as the latent code in Part 4’s ELBO, does not tell you whether an observed $x$ is discrete or continuous. Some papers write capital $P$ for probabilities and lowercase $p$ for densities, but many use $p$ for both.

To describe how much a distribution spreads around its mean, papers use variance: the average squared difference from the mean, $\operatorname{Var}(X)=\mathbb E[(X-\mu)^2]$, where $\mu=\mathbb E[X]$. The standard deviation is its square root, so it has the same units as $X$.

These quantities also appear in distribution names. The Gaussian distribution, also called the normal distribution, is written $\mathcal N(\mu,\sigma^2)$: $\mu$ is its mean and $\sigma^2$ its variance. For a vector, $\mathcal N(\mu,\Sigma)$ uses a covariance matrix $\Sigma$: entry $(i,j)$ measures how coordinates $i$ and $j$ vary together, and the diagonal holds their variances. $\mathcal N(0,I)$ describes a vector of independent standard normal numbers, each with mean 0 and variance 1; $I$ is the identity matrix.

Benchmark tables often report an average score together with a measure of variation. The spread of individual runs differs from the uncertainty of their average: for $N$ independent runs with standard deviation $\sigma$, the standard error of the mean is $\sigma/\sqrt N$.

Before comparing two entries such as $82.3\pm0.4$ and $81.5\pm0.4$, find in the caption whether 0.4 is a standard deviation across runs, a standard error or the half-width of a confidence interval. To compare the means, compute an interval for their difference. A 95% confidence-interval procedure produces ranges that contain the true difference in 95% of repeated experiments under its assumptions.

For this example, assume four independent runs per model, approximately normal run scores, and equal population variances. If the runs are paired, such as two models evaluated on the same independently sampled tasks, analyze the differences within pairs instead. NIST’s two-sample guide explains these cases.

For independent means, the variance of their difference is the sum of their variances. A standard normal variable falls between $-1.96$ and $1.96$ with probability about 0.95, which supplies the multiplier for a normal approximation to a 95% interval. If 0.4 is the standard deviation across four runs, that first calculation gives:

That 1.96 multiplier treats the estimated spread as known. With only four runs per model, use Student’s t-distribution to account for uncertainty in the spread. Under the assumptions above, its degrees of freedom are $4+4-2=6$ (eight runs minus two estimated means). scipy.stats.t.ppf(0.975, 6) gives about 2.447 instead of 1.96; 0.975 leaves 2.5% outside each end of a 95% interval. If the equal-variance assumption is unsuitable, use a Welch interval, whose degrees of freedom are computed differently.

Now consider the other reading of the caption: 0.4 is already each mean’s standard error, estimated from those four runs. Nothing is divided by $\sqrt4$, and the gap’s standard error is about 0.566. The t multiplier is still needed. The two readings give:

The figure compares the initial normal approximation with the two t intervals:

When 0.4 is the standard deviation, the t interval excludes zero: under these assumptions, the results support a positive difference. When 0.4 is the estimated standard error, the interval contains zero, so the results do not establish which model has the higher mean score.

Cross-entropy and KL divergence

During training, a model needs a numerical loss that measures how well its predicted probabilities agree with the training target. The training algorithm changes the model’s parameters to reduce that loss. Cross-entropy supplies such a loss for labeled examples or a teacher model’s probabilities. Kullback–Leibler (KL) divergence measures the mismatch between two distributions. Papers use it to train a student to imitate a teacher or to penalize changes from a fixed reference model.

Here the outcomes are discrete, such as tokens or classes. Write the target distribution as $p$ and the model’s predicted distribution over the same outcomes as $q$. Start with one observed outcome $i$: when the model gave it probability $q_i$, its loss is $-\log q_i$. It is 0 when $q_i=1$, about 0.69 when $q_i=0.5$, about 2.3 when $q_i=0.1$, and infinite when $q_i=0$.

Cross-entropy averages that loss over outcomes drawn from the target:

With a one-hot label $p$, which is 1 on the correct class $y$ and 0 elsewhere, only the correct class counts, and $H(p,q)=-\log q_y$: the ordinary classification loss. PyTorch’s nn.CrossEntropyLoss computes this from raw model scores, called logits, and applies log-softmax internally; its input is not the probability vector $q$. With a teacher model’s soft distribution as $p$, every class counts in proportion to the teacher’s probability. Classical distillation trains a small model on a large model’s probabilities this way. For the target $p=(0.9,0.1)$, two models have these cross-entropies $H(p,q)$:

The second model matches the target, and no model can do better. That minimum, 0.325 here, is $H(p)=-\sum_ip_i\log p_i$, the entropy of $p$. It is 0 only for a one-hot target: when the target itself is uncertain, some loss remains that no model can remove.

KL divergence subtracts that minimum and keeps only the loss caused by $q$ differing from $p$. It is written with a double bar $\Vert$, which only separates the two arguments; it is neither the conditional-probability bar $\mid$ from Conditional probability and the probability of text nor the length bars $\Vert x\Vert$ of the vectors reference. Papers use it instead of a comma to mark that the order matters: the average runs over outcomes drawn from the first argument.

For the first model above, $0.693-0.325\approx0.37$. KL is 0 when the two distributions match and positive otherwise.

The order changes which mismatches are penalized. A term with $p_i>0$ and $q_i=0$ is infinite, so $D_{\mathrm{KL}}(p\Vert q)$ penalizes $q$ for assigning zero probability to anything $p$ can produce. Outcomes with $p_i=0$ add nothing (by convention $0\log0=0$), so they contribute no direct penalty even if $q_i>0$. Reversing the arguments gives $D_{\mathrm{KL}}(q\Vert p)$, which is infinite wherever $q_i>0$ and $p_i=0$. Outcomes with $q_i=0$ now add nothing, even if $p_i>0$. With $p$ as the target, $D_{\mathrm{KL}}(p\Vert q)$ is called the forward KL and $D_{\mathrm{KL}}(q\Vert p)$ the reverse KL.

For $p=(0.5,0.5)$ and $q=(1,0)$, the forward KL is infinite, because $q$ misses an outcome $p$ produces, while the reverse KL is only $1\times\log(1/0.5)+0=\log2$: the outcome $q$ skips adds nothing. For $p=(0.9,0.1)$ and $q=(0.5,0.5)$, the forward KL is the 0.37 found above, computed here term by term, and the reverse KL differs:

This asymmetry is why KL is called a divergence rather than a distance.

Because $H(p,q)=H(p)+D_{\mathrm{KL}}(p\Vert q)$, and a fixed teacher’s entropy $H(p)$ does not depend on the student, cross-entropy against a fixed teacher gives the student the same gradients as $D_{\mathrm{KL}}(p\Vert q)$. Changing a student parameter changes both losses by the same amount, so their derivatives used for training are equal; Gradients and optimization explains that calculation. This lets classification and classical distillation code use cross-entropy. The swap works only with the teacher first. With the student first, $D_{\mathrm{KL}}(q\Vert p)=H(q,p)-H(q)$, and the student’s own entropy $H(q)$ changes as it trains, so the two no longer give the same gradients.

Vectors and matrices

Architecture equations describe calculations on arrays of numbers. An array’s shape tells you how its numbers are arranged, which operations are valid, and what those operations produce. A vector is an ordered list of numbers. A matrix arranges numbers in rows and columns. In ML code, tensor is the general term for these numerical arrays, including arrays with more than two axes. An axis is a dimension along which you index the array: a matrix has a row axis and a column axis.

A token’s hidden state is a vector of numbers computed inside the model to represent that token. Suppose we process a batch of $B$ sequences together, each with $T$ token positions, and each hidden state contains $d$ numbers. The hidden-state array has shape [B, T, d], which papers also write as $B\times T\times d$. Its three axes select the sequence, the token position, and the number within that token’s vector. For example, [2, 3, 4] holds two sequences, three token vectors per sequence, and four numbers per vector: 24 numbers in total. Here $d$ is the vector’s width. The shape lists the three sizes; multiplying them gives the number of entries.

Papers often write shapes as set membership: $a\in\mathbb R^d$ means that $a$ is a vector of $d$ real numbers, and $A\in\mathbb R^{m\times n}$ means that $A$ is a matrix with $m$ rows and $n$ columns. Bold letters sometimes mark vectors or tensors, but the paper’s definition determines the shape.

In the table below, $a$ and $b$ are column vectors: their numbers are arranged vertically. The superscript $\top$ means transpose: exchange rows and columns. Thus $a^\top$ is a row containing the same numbers as column $a$. The capital $A$ and $B$ here name matrices; $B$ no longer means batch size.

For a concrete comparison, take $a=\begin{pmatrix}1\\2\end{pmatrix}$ and $b=\begin{pmatrix}3\\4\end{pmatrix}$. Moving the transpose changes both the calculation and the result:

The dot product gives one score, as in the query–key comparison in attention. The outer product keeps a separate value for every pair of entries; the Gated DeltaNet update uses it to construct a matrix update from two vectors.

A matrix product can be regrouped, $(AB)C=A(BC)$, but not reordered; the MLA derivation uses the regrouping to skip rebuilding keys. NumPy writes matrix multiplication as @ and the elementwise product as *. Papers do not inherit NumPy’s broadcasting rules: adding a vector to a matrix needs a stated axis. Papers also differ on whether a vector is a row or a column: attention stacks tokens as the rows of $Q$, while MLA, Gated DeltaNet and LoRA write $Wh$ with column vectors, and PyTorch’s nn.Linear computes x @ W.T on rows.

An architecture equation may also describe a vector’s size, restrict which matrices are allowed, or rearrange a product:

Gradients and optimization

Training uses derivatives to decide how to change the model’s parameters. Let $\mathcal L$ be one scalar loss and $\theta$ the vector of all parameters; $\theta_i$ is one entry in that vector.

The partial derivative $\partial\mathcal L/\partial\theta_i$ is one number, evaluated at the current parameter values. It measures the loss’s rate of change when only $\theta_i$ changes and the other parameters stay fixed. Read it as “the partial derivative of the loss with respect to parameter $\theta_i$”.

The gradient $\nabla_\theta\mathcal L$ collects those numbers for every parameter into one vector. Its $i$th entry is $\partial\mathcal L/\partial\theta_i$. The symbol $\nabla$ is called nabla; here it means “take the gradient”. With only one parameter, the gradient has one entry equal to the ordinary derivative.

For example, take $\mathcal L(\theta_1,\theta_2)=\theta_1^2+3\theta_2^2$. At $\theta_1=2$ and $\theta_2=1$:

The first partial derivative, 4, means that increasing $\theta_1$ by 0.01 raises the loss by approximately 0.04 while $\theta_2$ stays fixed. The gradient contains this information for both parameters. Gradient descent subtracts each parameter’s derivative, multiplied by a positive learning rate $\eta$ that controls the step size:

With $\eta=0.1$, the example changes from $(2,1)$ to $(1.6,0.4)$, and the loss falls from 7 to 3.04. For a differentiable loss with a nonzero gradient, a sufficiently small step lowers the loss. A zero gradient gives no update.

The subscript on $\nabla$ says what varies. $\nabla_\theta\log p_\theta(x)$ varies the parameters; $\nabla_x\log p_\theta(x)$ varies the input, and diffusion papers call it the score, which has nothing to do with a benchmark score.

A Jacobian records derivatives for several outputs. A neural-network layer can produce many numbers, such as the components of a token’s hidden state. To describe how every output responds to every input, collect the partial derivatives into a matrix called the Jacobian. For $y=f(x)$ with $n$ input numbers and $m$ output numbers, its shape is $m\times n$: rows correspond to outputs, columns to inputs, and entry $J_{ij}=\partial y_i/\partial x_j$ measures how output $i$ changes when input $j$ changes.

For example, let a function take two numbers and return $y_1=x_1+2x_2$ and $y_2=3x_1-x_2$. Its Jacobian is

The second column says that increasing $x_2$ by 0.1 while holding $x_1$ fixed increases $y_1$ by 0.2 and decreases $y_2$ by 0.1. This example is linear, so its Jacobian is constant. For a nonlinear function, the entries can depend on the current input and describe small changes locally. With just one scalar output, the Jacobian has one row: the same derivatives as the gradient, transposed under our column-vector convention. Deep Learning, §4.3, gives the general definitions.

Backpropagation uses these derivatives to compute the loss gradient through successive layers. The change-of-variables example in this article’s Appendix B uses a different property: the absolute value of a square Jacobian’s determinant gives the local volume scaling factor, which is needed to adjust a probability density.

For training, derivatives describe local changes; an optimization objective states what the updates should minimize or maximize. Papers distinguish the best value of an objective from the parameters that achieve it: $\min_\theta f(\theta)$ is the smallest value of $f$, while $\operatorname*{arg\,min}_\theta f(\theta)$ gives the parameter values that achieve it. For $f(\theta)=(\theta-3)^2$, the minimum is 0 and the argmin is 3. An argmin specifies the desired result, but leaves the update algorithm open.

For example, MiniLLM writes its goal as $\theta=\operatorname*{arg\,min}_\theta D_{\mathrm{KL}}(q_\theta\Vert p)$ (equation 1), where $q_\theta$ is the student model and $p$ the teacher it learns from. Section 2.2 then derives a gradient and modifies it to make training more stable; equation (7) is the gradient used for updates. The on-policy distillation example explains those changes. Reading only the argmin would miss them.

A stop-gradient, $\operatorname{sg}(x)$, passes $x$ forward unchanged but blocks gradients through it, like .detach() in PyTorch. Two expressions with the same forward value can train differently if one of them is detached. In the GRPO example, Hugging Face TRL treats the old model’s log-probability as fixed. In the distillation shortcut discussed by DeepSeek-V4, a detached teacher–student log-ratio supplies a fixed weight for the update.

Sets and constraints

A constraint states which values are allowed. Papers often express constraints using sets: collections of distinct members, written in braces, such as $\{1,2,3\}$. Order does not matter, so $\{3,1,2\}$ names the same set. Repeated entries count only once. By comparison, the parentheses in a vector such as $(1,2,3)$ describe an ordered list: $(3,1,2)$ is a different vector.

Once a paper defines a set, it can test membership, combine sets, count their members or state a condition on them. Take $A=\{1,2,3\}$ and $B=\{3,4\}$:

The shape notation from Vectors and matrices also names sets: $\mathbb R^d$ is the set of all vectors of $d$ real numbers, and $\mathbb R^{n\times d}$ the set of all real matrices with $n$ rows and $d$ columns. Thus $Q\in\mathbb R^{n\times d_k}$ specifies which shape $Q$ must have.

Set-builder notation describes a set by a condition: $\{x\in S\mid\text{condition}\}$ reads “the members of $S$ that satisfy the condition”, like the Python set comprehension {x for x in S if condition(x)}. The bar means “such that”, not conditional probability; some papers use a colon instead. Examples in this series include mHC’s allowed matrices, MoE’s selected experts and sparse attention’s selected tokens.

DAPO, a training method that filters groups of generated answers, keeps a question only when some answers are correct and some are wrong. For $G$ sampled answers $o_1,\ldots,o_G$ and reference answer $a$, write the set of correct answer indices as