Relearning Mathematics: Three Languages for Reading AI

0. Key points

• The inside of an LLM can be described almost completely in three languages: linear algebra (embeddings, attention, low-rank approximation), calculus (gradients, the chain rule, optimisation) and probability and statistics (likelihood, cross-entropy, Bayesian updating). These three are the common language.
• The point of relearning mathematics is not to build models of your own. It is to predict behaviour, explain failures and design experiments. The learning rate was raised and training diverged; every pair of embeddings has similarity around $0.7$; the detector reports 99% accuracy and the operators say almost every alert is false. All of these can be explained by formulas, and estimated in advance.
• In high dimensions, two unrelated directions are almost orthogonal (Proposition 3.2, Corollary 3.5). “Why can billions of concepts be packed into a vector of a few thousand dimensions?” is a consequence of this fact.
• Whether gradient descent converges is decided almost entirely by the relation $0 < \eta < 2/L$ between the learning rate $\eta$ and the smoothness constant $L$ of the function (Lemma 4.2, Example 4.4). Tuning hyperparameters is not guesswork; it is a search around this inequality.
• The cross-entropy that training minimises is the same thing as the KL divergence being minimised (Proposition 5.2). The reading of perplexity as “the effective number of choices” also falls out of that formula.
• As a by-product of studying mathematics one acquires fluency with quantifiers (telling $\forall\exists$ from $\exists\forall$) and the habit of constructing counterexamples. Both transfer directly to writing specifications, analysing incidents and designing evaluations.

1. Motivation: why “being able to use it” is not enough

As we saw in the previous chapter, LLMs and Programming, a substantial part of the work of writing code can already be handed over to a model. The condition under which handing it over pays off was formulated in when delegation pays（Proposition 3.1）［LLMs and Programming］. Call the API, shape the prompt, inspect the output. Within that range almost no mathematics is needed, and there are many situations in which that is quite enough to deliver results.

The trouble starts when things do not work.

• You build search over internal documents with embedding vectors, and the same document comes out on top for every query.
• You try fine-tuning, and the loss becomes NaN within a few steps.
• The anomaly detection model is reported at 99% accuracy, yet the operators say “nine out of ten alerts are false”.
• The evaluation set improves while production gets worse.

None of these is a problem of how the tool is used. Each has its own mathematical cause: the geometry of the space, the dynamics of optimisation, the rule for updating probabilities. If you do not know the name of the cause, the only remedy left is to vary parameters and pray. If you do know the name, isolating the cause can take a single formula. Should the learning rate be halved, should more data be collected, or is the metric itself wrong? You gain grounds on which to decide.

Historically, this pattern is nothing new. Nineteenth-century engineers built steam engines without thermodynamics. Improving efficiency meant relying on experience and intuition, and progress came by trial and error. Only once Carnot and Clausius had formulated the second law could one say “this engine has a theoretical upper bound on its efficiency” and “that bound depends only on a ratio of temperatures”. Being able to build something and being able to state its limits are different abilities.

Those of us now working with LLMs resemble engineers who can build a steam engine but have no thermodynamics. Fortunately, the “thermodynamics” of AI already exists, and most of it is first- and second-year university mathematics. There is no new mathematics to invent; it suffices to recover what many people once learned and forgot. In this chapter we set out the contents of those three languages in a form you can work through by hand.

Note

This article is not the claim that people who know mathematics are superior. Mathematics is a tool for making testable predictions about the behaviour of AI. When a prediction fails, you go back and question the mathematics. Being able to make that round trip is where the value lies.

2. A map of the three languages

Let us first fix the overall picture. The path from a piece of text to an update of the parameters decomposes as in the following diagram, with the corresponding branch of mathematics attached to each stage.

flowchart TB
A["Text"] --> B["Token sequence"]
B --> C["Embedding vectors: linear algebra"]
C --> D["Attention = inner products and matrix products: linear algebra"]
D --> E["Next-token probability distribution: probability"]
E --> F["Cross-entropy loss: probability"]
F --> G["Gradients, chain rule, backpropagation: calculus"]
G --> H["Parameter update = optimisation"]
H --> C

One loop of an LLM, and the mathematics governing each stage

The roles of the three fields can be separated in one line each.

Field	What it handles	How it appears in an LLM
Linear algebra	Represents “meaning” as coordinates and transforms it	Embeddings, attention, low-rank approximation (LoRA), dimensionality reduction
Calculus	Measures “how things change under a small perturbation”	Gradients, chain rule, backpropagation, learning rate, convergence and divergence
Probability and statistics	Measures “plausibility” and “confidence”	Cross-entropy, perplexity, temperature, evaluation metrics, Bayesian updating

We now take them in turn. Theorems are stated in full, proofs fill in the gaps, and wherever the computation can be done by hand we carry the numbers to the end.

3. Linear algebra: putting meaning on coordinates

3.1. How an inner product becomes “closeness of meaning”

An embedding is a map sending words or documents to real vectors in $d$ dimensions. The reason this is useful is that the inner product between vectors serves as a proxy for closeness of meaning.

Definition 3.1（Inner product and cosine similarity）

For $\boldsymbol{x}, \boldsymbol{y} \in \mathbb{R}^d$, define the inner product by $\langle \boldsymbol{x}, \boldsymbol{y}\rangle = \sum_{i=1}^{d} x_i y_i$ and the norm by $\|\boldsymbol{x}\| = \sqrt{\langle \boldsymbol{x},\boldsymbol{x}\rangle}$. When $\boldsymbol{x} \ne \boldsymbol{0}$ and $\boldsymbol{y}\ne\boldsymbol{0}$, the quantity

$$\cos(\boldsymbol{x}, \boldsymbol{y}) = \frac{\langle \boldsymbol{x}, \boldsymbol{y}\rangle}{\|\boldsymbol{x}\|\,\|\boldsymbol{y}\|}$$

is called the cosine similarity. By the Cauchy–Schwarz inequality, $-1 \le \cos(\boldsymbol{x},\boldsymbol{y}) \le 1$.

What attention computes is, in essence, this same inner product. One lines up the inner products $\langle \boldsymbol{q}, \boldsymbol{k}_j\rangle$ of a query vector $\boldsymbol{q}$ with each key vector $\boldsymbol{k}_j$, divides by $\sqrt{d}$ and passes the result through a softmax. In other words, “which token to attend to” has been reduced to a computation of angles in $d$-dimensional space.

3.2. In high dimensions, unrelated things are almost orthogonal

A naive question arises here. Dimensions such as $d = 768$ or $d = 4096$ are far smaller than the number of human words and concepts, which runs from hundreds of thousands to tens of millions. Why can so many concepts be packed into so few dimensions without blurring into one another?

The answer lies in the geometry of high-dimensional space. In $\mathbb{R}^2$ one can find only six directions pairwise at least 60 degrees apart, but in $\mathbb{R}^{1024}$ the situation is transformed.

Proposition 3.2（Inner product of two random directions）

Let $d \ge 1$. Let $\boldsymbol{u}, \boldsymbol{v}$ be independent random vectors distributed uniformly on the unit sphere $S^{d-1} = \{\boldsymbol{x}\in\mathbb{R}^d : \|\boldsymbol{x}\|=1\}$. Then

$$\mathbb{E}[\langle \boldsymbol{u},\boldsymbol{v}\rangle] = 0, \qquad \operatorname{Var}[\langle \boldsymbol{u},\boldsymbol{v}\rangle] = \frac{1}{d}$$

holds.

Proof（Proposition 3.2）

The uniform distribution on the sphere is rotation invariant: for any orthogonal matrix $Q$, the vector $Q\boldsymbol{u}$ has the same distribution as $\boldsymbol{u}$. Since $\boldsymbol{v}$ and $\boldsymbol{u}$ are independent, condition on $\boldsymbol{v}$ and hold it fixed; taking an orthogonal matrix $Q$ carrying $\boldsymbol{v}$ to the first coordinate axis $\boldsymbol{e}_1$, the quantities $\langle \boldsymbol{u},\boldsymbol{v}\rangle$ and $\langle Q\boldsymbol{u}, \boldsymbol{e}_1\rangle = (Q\boldsymbol{u})_1$ have the same distribution. Hence $\langle\boldsymbol{u},\boldsymbol{v}\rangle$ is distributed like the first component $u_1$ of a uniformly random unit vector on the sphere.

For the mean: $\boldsymbol{u}$ and $-\boldsymbol{u}$ have the same distribution (because $Q = -I$ is orthogonal), so $u_1$ and $-u_1$ have the same distribution; the expectation exists since $|u_1|\le 1$ is bounded, and therefore $\mathbb{E}[u_1] = 0$.

For the variance: from $\|\boldsymbol{u}\|=1$ we get $\sum_{i=1}^{d} u_i^2 = 1$ with probability 1. Taking expectations on both sides gives $\sum_{i=1}^{d}\mathbb{E}[u_i^2] = 1$. Permuting coordinates is also an orthogonal transformation, so $\mathbb{E}[u_1^2] = \cdots = \mathbb{E}[u_d^2]$, whence $\mathbb{E}[u_1^2] = 1/d$. As the mean is $0$, the variance is $1/d$ as well.

∎

The standard deviation is $1/\sqrt{d}$. For $d = 1024$ this is about $0.031$: the cosine similarity of two unrelated vectors is typically no larger than about $0.03$. Moreover this concentration has Gaussian tails.

Theorem 3.3（Concentration of measure on the sphere）

Let $d \ge 2$ and let $\boldsymbol{u}$ be a random vector distributed uniformly on $S^{d-1}$. For every $t > 0$,

$$\Pr\bigl[\,|u_1| \ge t\,\bigr] \le 2 e^{-d t^2 / 2}$$

holds.

Remark 3.4

Theorem 3.3 is exactly an area estimate for a spherical cap. An elementary proof can be found in K. Ball’s lecture notes An Elementary Introduction to Modern Convex Geometry (Lemma 2.2). For a treatment inside the more general framework of concentration of measure, see Chapter 3 of Vershynin, High-Dimensional Probability. Here we simply use the result.

Corollary 3.5（How many almost orthogonal directions fit）

Let $d \ge 2$ and $t \in (0,1)$. If $n$ satisfies

$$n < e^{d t^2 / 4},$$

then there exist unit vectors $\boldsymbol{u}_1,\ldots,\boldsymbol{u}_n$ in $\mathbb{R}^d$ with $|\cos(\boldsymbol{u}_i, \boldsymbol{u}_j)| < t$ for all $i \ne j$.

Proof（Corollary 3.5）

Choose $\boldsymbol{u}_1,\ldots,\boldsymbol{u}_n$ independently from the uniform distribution on $S^{d-1}$. For a fixed pair $i \ne j$, the inner product $\langle \boldsymbol{u}_i,\boldsymbol{u}_j\rangle$ is distributed like $u_1$, as we saw in the proof of Proposition 3.2, so Theorem 3.3 gives

$$\Pr\bigl[\,|\langle \boldsymbol{u}_i,\boldsymbol{u}_j\rangle| \ge t\,\bigr] \le 2e^{-dt^2/2}.$$

The probability of the event “some pair $(i,j)$ reaches $t$ or more” is, by the union bound, at most

$$\binom{n}{2}\cdot 2e^{-dt^2/2} = n(n-1)e^{-dt^2/2} < n^2 e^{-dt^2/2}.$$

The hypothesis $n < e^{dt^2/4}$ gives $n^2 < e^{dt^2/2}$, so this upper bound is less than $1$. Therefore the event ”$|\langle\boldsymbol{u}_i,\boldsymbol{u}_j\rangle| < t$ for every pair” has positive probability, and at least one such configuration exists. Since the vectors are unit vectors, the cosine similarity is the inner product itself.

∎

Example 3.6（How many concept axes fit into 1024 dimensions）

Substituting $d = 1024$ into Corollary 3.5 and varying the threshold $t$:

• $t = 0.1$: $e^{1024 \times 0.01/4} = e^{2.56} \approx 12.9$, so about 12 directions.
• $t = 0.2$: $e^{1024 \times 0.04/4} = e^{10.24} \approx 2.8\times 10^{4}$, about 28 thousand.
• $t = 0.3$: $e^{1024 \times 0.09/4} = e^{23.04} \approx 1.0\times 10^{10}$, about 10 billion.

Merely allowing the loose criterion “cosine similarity below $0.3$ counts as effectively unrelated” already lets 10 billion directions coexist in 1024 dimensions. This is why embeddings of a comparatively modest dimension can hold an enormous number of concepts.

At the same time this computation is a practical warning. Because $t$ enters the exponent squared, the count drops precipitously as $t$ is tightened. A fixed threshold such as “similarity above 0.8 means the document is relevant” therefore depends strongly on the dimension and on the data distribution, and does not transfer as it stands. Set thresholds only after looking at the actual distribution, that is, a histogram of cosine similarities over unrelated pairs.

3.3. Low-rank approximation: shrinking without discarding information

Here is a second theorem with immediate practical consequences: the best-approximation theorem based on the singular value decomposition (SVD).

Theorem 3.7（Eckart–Young–Mirsky theorem (Frobenius norm version)）

Let $A \in \mathbb{R}^{m\times n}$ have rank $r$, with singular value decomposition

$$A = \sum_{i=1}^{r}\sigma_i \boldsymbol{u}_i \boldsymbol{v}_i^{\mathsf{T}}, \qquad \sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_r > 0$$

where $\{\boldsymbol{u}_i\}$ and $\{\boldsymbol{v}_i\}$ are orthonormal systems. For an integer $k$ with $1 \le k < r$, put $A_k = \sum_{i=1}^{k}\sigma_i\boldsymbol{u}_i\boldsymbol{v}_i^{\mathsf{T}}$. Then for every matrix $B \in \mathbb{R}^{m\times n}$ of rank at most $k$,

$$\|A - B\|_F \ \ge\ \|A - A_k\|_F = \sqrt{\sum_{i=k+1}^{r}\sigma_i^2}$$

holds, where $\|X\|_F = \sqrt{\sum_{i,j} X_{ij}^2}$ is the Frobenius norm.

Remark 3.8

A proof can be found in Chapter 2 of Golub & Van Loan, Matrix Computations, or Chapter 7 of Strang, Introduction to Linear Algebra. The original paper is C. Eckart and G. Young, “The approximation of one matrix by another of lower rank”, Psychometrika 1 (1936), 211–218. In this article we concentrate on how the result is used.

What the theorem says is that the naive operation of “keep the $k$ largest singular values and truncate the rest” is in fact optimal among all matrices of rank at most $k$. The approximation error is measured exactly by the square root of the sum of squares of the discarded singular values. Dimensionality reduction, recommender systems, denoising and parameter-efficient fine-tuning via LoRA are all applications of this single fact.

Example 3.9（Computing the best rank-1 approximation of a 3×2 matrix by hand）

$$A = \begin{pmatrix} 1 & 1 \\ 1 & 1 \\ 1 & -1 \end{pmatrix}$$

We find its best rank-1 approximation. First,

$$A^{\mathsf{T}}A = \begin{pmatrix} 3 & 1 \\ 1 & 3\end{pmatrix}$$

(the top-left entry is $1^2+1^2+1^2 = 3$, and the off-diagonal entry is $1\cdot 1 + 1\cdot 1 + 1\cdot(-1) = 1$). The characteristic equation $(3-\lambda)^2 - 1 = 0$ gives eigenvalues $\lambda = 4, 2$, with corresponding eigenvectors $\boldsymbol{v}_1 = (1,1)^{\mathsf{T}}/\sqrt{2}$ and $\boldsymbol{v}_2 = (1,-1)^{\mathsf{T}}/\sqrt{2}$. Hence the singular values are $\sigma_1 = \sqrt{4} = 2$ and $\sigma_2 = \sqrt{2}$.

The left singular vector is obtained from $\boldsymbol{u}_1 = A\boldsymbol{v}_1/\sigma_1$:

$$A\boldsymbol{v}_1 = \frac{1}{\sqrt{2}}\begin{pmatrix} 2 \\ 2 \\ 0\end{pmatrix}, \qquad \boldsymbol{u}_1 = \frac{1}{2}\cdot\frac{1}{\sqrt{2}}\begin{pmatrix}2\\2\\0\end{pmatrix} = \frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\\0\end{pmatrix}.$$

Therefore

$$A_1 = \sigma_1 \boldsymbol{u}_1\boldsymbol{v}_1^{\mathsf{T}} = 2\cdot\frac{1}{\sqrt2}\begin{pmatrix}1\\1\\0\end{pmatrix}\cdot\frac{1}{\sqrt2}\begin{pmatrix}1 & 1\end{pmatrix} = \begin{pmatrix}1 & 1\\ 1& 1\\ 0 & 0\end{pmatrix}.$$

Let us check the error:

$$A - A_1 = \begin{pmatrix}0&0\\0&0\\1&-1\end{pmatrix}, \qquad \|A-A_1\|_F = \sqrt{1^2+(-1)^2} = \sqrt2 = \sigma_2.$$

Just as Theorem 3.7 asserts, the error equals the discarded singular value $\sigma_2$. Moreover $\|A\|_F^2 = 6 = \sigma_1^2+\sigma_2^2 = 4 + 2$, so rank 1 retains $4/6 \approx 66.7\%$ of the total “energy”.

Remark 3.10

LoRA freezes a weight matrix $W \in \mathbb{R}^{m\times n}$ and restricts the update alone to the rank-$r$ form $\Delta W = BA$ with $B\in\mathbb{R}^{m\times r}$ and $A\in\mathbb{R}^{r\times n}$. Taking $m = n = 4096$ and $r = 8$, ordinary full-parameter updating handles $4096^2 = 16{,}777{,}216$ values, whereas LoRA needs only $8\times(4096+4096) = 65{,}536$. The ratio is $65{,}536 / 16{,}777{,}216 \approx 0.39\%$. The premise that makes the method work is the hypothesis that the update required for fine-tuning is essentially low rank, and the yardstick for judging that hypothesis is the error estimate of Theorem 3.7. Details are in the original paper of Hu et al. (2021).

4. Calculus: learning is walking downhill

4.1. Smoothness and the descent lemma

Learning means searching for parameters that make a loss function $f$ small. The most basic method is gradient descent, $\boldsymbol{x}_{k+1} = \boldsymbol{x}_k - \eta\nabla f(\boldsymbol{x}_k)$, of which the SGD and Adam used in practice are variants. The question “what should the learning rate $\eta$ be?” is quantified by the following definition.

Definition 4.1（L-smoothness）

Let $f:\mathbb{R}^n \to \mathbb{R}$ be differentiable. If there is a constant $L > 0$ such that for all $\boldsymbol{x},\boldsymbol{y}\in\mathbb{R}^n$

$$\|\nabla f(\boldsymbol{x}) - \nabla f(\boldsymbol{y})\| \le L\|\boldsymbol{x}-\boldsymbol{y}\|$$

holds, then $f$ is said to be $L$-smooth (its gradient is $L$-Lipschitz).

Here $L$ is an upper bound on “how abruptly the gradient can change”. If $f$ is twice differentiable, $L$ corresponds to an upper bound on the absolute values of the eigenvalues of the Hessian.

Lemma 4.2（Descent lemma and the decrease in one step）

Let $f:\mathbb{R}^n\to\mathbb{R}$ be differentiable and $L$-smooth. Then for all $\boldsymbol{x},\boldsymbol{y}$,

$$f(\boldsymbol{y}) \le f(\boldsymbol{x}) + \langle \nabla f(\boldsymbol{x}), \boldsymbol{y}-\boldsymbol{x}\rangle + \frac{L}{2}\|\boldsymbol{y}-\boldsymbol{x}\|^2$$

holds. In particular, setting $\boldsymbol{y} = \boldsymbol{x} - \eta\nabla f(\boldsymbol{x})$ gives

$$f(\boldsymbol{y}) \le f(\boldsymbol{x}) - \eta\Bigl(1 - \frac{L\eta}{2}\Bigr)\|\nabla f(\boldsymbol{x})\|^2 .$$

Consequently, if $\nabla f(\boldsymbol{x}) \ne \boldsymbol{0}$ and $0 < \eta < 2/L$ then $f(\boldsymbol{y}) < f(\boldsymbol{x})$, and at $\eta = 1/L$ the coefficient $\eta(1-L\eta/2)$ of the decrease on the right-hand side attains its maximum value $1/(2L)$.

Proof（Lemma 4.2）

Put $g(t) = f(\boldsymbol{x} + t(\boldsymbol{y}-\boldsymbol{x}))$. Then $g$ is differentiable on $[0,1]$ and, by the chain rule, $g'(t) = \langle \nabla f(\boldsymbol{x}+t(\boldsymbol{y}-\boldsymbol{x})), \boldsymbol{y}-\boldsymbol{x}\rangle$. By the fundamental theorem of calculus,

$$f(\boldsymbol{y}) - f(\boldsymbol{x}) = g(1)-g(0) = \int_0^1 \langle \nabla f(\boldsymbol{x}+t(\boldsymbol{y}-\boldsymbol{x})), \boldsymbol{y}-\boldsymbol{x}\rangle\,dt.$$

Subtracting $\langle\nabla f(\boldsymbol{x}), \boldsymbol{y}-\boldsymbol{x}\rangle = \int_0^1 \langle \nabla f(\boldsymbol{x}), \boldsymbol{y}-\boldsymbol{x}\rangle dt$ from both sides,

$$f(\boldsymbol{y}) - f(\boldsymbol{x}) - \langle\nabla f(\boldsymbol{x}),\boldsymbol{y}-\boldsymbol{x}\rangle = \int_0^1 \langle \nabla f(\boldsymbol{x}+t(\boldsymbol{y}-\boldsymbol{x})) - \nabla f(\boldsymbol{x}),\ \boldsymbol{y}-\boldsymbol{x}\rangle\,dt.$$

Apply the Cauchy–Schwarz inequality to the integrand, then the $L$-smoothness of Definition 4.1 to the pair $\boldsymbol{x}+t(\boldsymbol{y}-\boldsymbol{x})$ and $\boldsymbol{x}$, whose distance is $t\|\boldsymbol{y}-\boldsymbol{x}\|$:

$$\le \int_0^1 \|\nabla f(\boldsymbol{x}+t(\boldsymbol{y}-\boldsymbol{x})) - \nabla f(\boldsymbol{x})\|\,\|\boldsymbol{y}-\boldsymbol{x}\|\,dt \le \int_0^1 L t\|\boldsymbol{y}-\boldsymbol{x}\|^2 dt = \frac{L}{2}\|\boldsymbol{y}-\boldsymbol{x}\|^2 .$$

This proves the first inequality.

Next substitute $\boldsymbol{y}-\boldsymbol{x} = -\eta\nabla f(\boldsymbol{x})$. Since $\langle \nabla f(\boldsymbol{x}), -\eta\nabla f(\boldsymbol{x})\rangle = -\eta\|\nabla f(\boldsymbol{x})\|^2$ and $\|{-\eta}\nabla f(\boldsymbol{x})\|^2 = \eta^2\|\nabla f(\boldsymbol{x})\|^2$, we get

$$f(\boldsymbol{y}) \le f(\boldsymbol{x}) - \eta\|\nabla f(\boldsymbol{x})\|^2 + \frac{L\eta^2}{2}\|\nabla f(\boldsymbol{x})\|^2 = f(\boldsymbol{x}) - \eta\Bigl(1-\frac{L\eta}{2}\Bigr)\|\nabla f(\boldsymbol{x})\|^2 .$$

The coefficient $\varphi(\eta) = \eta - L\eta^2/2$ is a quadratic in $\eta$ with $\varphi(\eta) > 0 \iff 0 < \eta < 2/L$. Moreover $\varphi'(\eta) = 1 - L\eta = 0$ gives the maximum at $\eta = 1/L$, where $\varphi(1/L) = 1/L - 1/(2L) = 1/(2L)$.

∎

The condition $0 < \eta < 2/L$ is the true identity of the phenomenon known in practice as “raise the learning rate too far and everything breaks”. The threshold $2/L$ is determined by the curvature of the function, so it shifts whenever the model or the data changes.

Behaviour differs according to whether the learning rate lies inside or outside the threshold 2/L

4.2. Rate of convergence in the convex case

Merely decreasing is not enough. The next theorem tells us how fast we approach the minimum.

Theorem 4.3（Convergence of gradient descent (convex, L-smooth case)）

Let $f:\mathbb{R}^n\to\mathbb{R}$ be a differentiable convex function that is $L$-smooth, and suppose there is a point $\boldsymbol{x}^{*}$ attaining the minimum value $f^{*} = f(\boldsymbol{x}^{*})$. Define $\boldsymbol{x}_{k+1} = \boldsymbol{x}_k - \eta\nabla f(\boldsymbol{x}_k)$ with learning rate $\eta = 1/L$. Then for every $K \ge 1$,

$$f(\boldsymbol{x}_K) - f^{*} \le \frac{L\|\boldsymbol{x}_0 - \boldsymbol{x}^{*}\|^2}{2K}$$

holds.

Proof（Theorem 4.3）

Write $\boldsymbol{g}_k = \nabla f(\boldsymbol{x}_k)$.

Step 1 (decrease in one step). Putting $\eta = 1/L$ in Lemma 4.2 gives

$$f(\boldsymbol{x}_{k+1}) \le f(\boldsymbol{x}_k) - \frac{1}{2L}\|\boldsymbol{g}_k\|^2 .$$

In particular the sequence is monotonically non-increasing: $f(\boldsymbol{x}_0)\ge f(\boldsymbol{x}_1)\ge\cdots$.

Step 2 (convexity). Since $f$ is convex and differentiable, its tangent planes support it from below. Substituting $\boldsymbol{x}^{*}$,

$$f(\boldsymbol{x}^{*}) \ge f(\boldsymbol{x}_k) + \langle \boldsymbol{g}_k, \boldsymbol{x}^{*}-\boldsymbol{x}_k\rangle \quad\Longleftrightarrow\quad f(\boldsymbol{x}_k) - f^{*} \le \langle \boldsymbol{g}_k, \boldsymbol{x}_k - \boldsymbol{x}^{*}\rangle .$$

Step 3 (combining the two). Applying Step 2 to the term $f(\boldsymbol{x}_k)$ on the right-hand side of Step 1,

$$f(\boldsymbol{x}_{k+1}) - f^{*} \le \langle \boldsymbol{g}_k, \boldsymbol{x}_k-\boldsymbol{x}^{*}\rangle - \frac{1}{2L}\|\boldsymbol{g}_k\|^2 .$$

Step 4 (turning it into a telescoping sum). Using $\boldsymbol{x}_{k+1} = \boldsymbol{x}_k - \boldsymbol{g}_k/L$, expand the difference of distances:

$$\|\boldsymbol{x}_k-\boldsymbol{x}^{*}\|^2 - \|\boldsymbol{x}_{k+1}-\boldsymbol{x}^{*}\|^2 = \|\boldsymbol{x}_k-\boldsymbol{x}^{*}\|^2 - \Bigl\|\boldsymbol{x}_k - \boldsymbol{x}^{*} - \frac{\boldsymbol{g}_k}{L}\Bigr\|^2 = \frac{2}{L}\langle \boldsymbol{g}_k, \boldsymbol{x}_k-\boldsymbol{x}^{*}\rangle - \frac{1}{L^2}\|\boldsymbol{g}_k\|^2 .$$

Multiplying both sides by $L/2$, the right-hand side is exactly the right-hand side of Step 3. That is,

$$f(\boldsymbol{x}_{k+1}) - f^{*} \le \frac{L}{2}\bigl(\|\boldsymbol{x}_k-\boldsymbol{x}^{*}\|^2 - \|\boldsymbol{x}_{k+1}-\boldsymbol{x}^{*}\|^2\bigr).$$

Step 5 (summing, and monotonicity). Summing over $k = 0,1,\ldots,K-1$, the right-hand side telescopes and

$$\sum_{k=1}^{K}\bigl(f(\boldsymbol{x}_k)-f^{*}\bigr) \le \frac{L}{2}\bigl(\|\boldsymbol{x}_0-\boldsymbol{x}^{*}\|^2 - \|\boldsymbol{x}_K-\boldsymbol{x}^{*}\|^2\bigr) \le \frac{L}{2}\|\boldsymbol{x}_0-\boldsymbol{x}^{*}\|^2 .$$

By the monotonicity of Step 1, each term on the left is at least the final term $f(\boldsymbol{x}_K)-f^{*}$. Hence the left-hand side is $\ge K\bigl(f(\boldsymbol{x}_K)-f^{*}\bigr)$, and dividing by $K$ gives the claim.

∎

The conclusion that the error decreases as $O(1/K)$ is worth remembering, because it says that reducing the error by a factor of ten requires ten times as many iterations. In this regime, “just train a bit longer and it will suddenly get better” does not happen. To improve matters you must change the conditioning of the problem (preconditioning, normalisation, momentum) rather than the iteration count.

Example 4.4（Checking the learning-rate threshold on a one-dimensional quadratic）

Consider $f(x) = \frac{a}{2}x^2$ with $a > 0$. Since $f'(x) = ax$ we have $|f'(x)-f'(y)| = a|x-y|$, so this $f$ is $L$-smooth with $L = a$ (Definition 4.1). Gradient descent reads

$$x_{k+1} = x_k - \eta a x_k = (1-\eta a)x_k,$$

so $x_k = (1-\eta a)^k x_0$. Now $|1-\eta a| < 1 \iff 0 < \eta < 2/a = 2/L$, which agrees with the condition in Lemma 4.2.

Concretely, take $a = 1$ (so $L = 1$ and the threshold is $\eta = 2$) and $x_0 = 1$:

• $\eta = 1.0$: $x_k = 0$ for $k \ge 1$ (the minimum is reached in one step).
• $\eta = 0.5$: $x_k = 0.5^k$; after 10 steps $x_{10} \approx 9.8\times10^{-4}$.
• $\eta = 1.9$: $x_k = (-0.9)^k$; the sign flips each step and $|x_{10}| = 0.9^{10}\approx 0.349$. Slow, but convergent.
• $\eta = 2.5$: $x_k = (-1.5)^k$; $|x_{10}| = 1.5^{10} \approx 57.7$ and $|x_{30}| \approx 1.9\times10^{5}$. Divergent.

For $\eta = 2.5$ the loss $f(x_k) = x_k^2/2$ grows exponentially, so in finite-precision arithmetic it eventually becomes inf, and NaN propagates from there. Most incidents of the form “the loss became NaN right after training started” are explained by this simple piece of dynamics. The first remedy is to lower the learning rate; the second is gradient clipping, which caps the effective value of $\|\boldsymbol{g}_k\|$.

4.3. The chain rule becomes backpropagation

The other pillar is the chain rule. The derivative of a composite $f = f_T\circ\cdots\circ f_1$ is a product of Jacobian matrices,

$$\frac{\partial f}{\partial \boldsymbol{x}} = J_T J_{T-1}\cdots J_1 .$$

Whether this product is evaluated from the right or from the left makes a large difference to the amount of computation required. Since the loss is a scalar, multiplying from the output side (the left) keeps every operation a “row vector times matrix” product. That is backpropagation, and it is why the cost of computing gradients stays within a constant factor of one forward pass even when there are hundreds of millions of parameters.

Viewing the derivative as a product of Jacobians also leads directly to an understanding of vanishing and exploding gradients. If the singular values of each layer’s $J_t$ are on average below 1, the product decays exponentially to 0; if above 1, it blows up exponentially. Residual connections ($\boldsymbol{y} = \boldsymbol{x} + F(\boldsymbol{x})$, with Jacobian $I + J_F$) and normalisation layers can be read as devices for keeping this product near 1.

5. Probability and statistics: measuring plausibility and confidence

5.1. What cross-entropy measures

The output of an LLM is a probability distribution over the next token. What training minimises is the cross-entropy against the target distribution.

Definition 5.1（Cross-entropy and KL divergence）

For probability distributions $p = (p_x)_{x\in\mathcal{X}}$ and $q = (q_x)_{x\in\mathcal{X}}$ on a finite set $\mathcal{X}$, the quantities

$$H(p) = -\sum_{x} p_x\log p_x, \qquad H(p,q) = -\sum_{x} p_x \log q_x, \qquad D(p\,\|\,q) = \sum_{x} p_x\log\frac{p_x}{q_x}$$

are called the entropy, the cross-entropy and the KL divergence respectively. Terms with $p_x = 0$ are taken to be $0$, and if $p_x > 0$ while $q_x = 0$ we set $D(p\|q) = +\infty$. Logarithms are to base $e$ (units: nats). Directly from the definitions, $H(p,q) = H(p) + D(p\|q)$.

Proposition 5.2（Gibbs' inequality）

With the notation above, $D(p\,\|\,q) \ge 0$. Equality holds if and only if $p_x = q_x$ for every $x$ with $p_x > 0$. In particular $H(p,q)\ge H(p)$.

Proof（Proposition 5.2）

If there is an $x$ with $p_x > 0$ and $q_x = 0$ then $D(p\|q)=+\infty > 0$, so assume from now on that $p_x>0 \Rightarrow q_x>0$. Put $S = \{x : p_x > 0\}$.

We use the inequality $\log t \le t - 1$ for $t > 0$. (This follows because $\psi(t) = t-1-\log t$ has $\psi'(t) = 1 - 1/t$, hence attains its minimum $\psi(1)=0$ at $t=1$; equality holds only at $t=1$.) Setting $t = q_x/p_x$,

$$-D(p\|q) = \sum_{x\in S} p_x\log\frac{q_x}{p_x} \le \sum_{x\in S} p_x\Bigl(\frac{q_x}{p_x}-1\Bigr) = \sum_{x\in S} q_x - \sum_{x\in S} p_x \le 1 - 1 = 0 .$$

The last inequality uses $\sum_{x\in S} q_x \le \sum_{x\in\mathcal{X}} q_x = 1$ together with $\sum_{x\in S}p_x = 1$. Hence $D(p\|q)\ge 0$.

Now examine equality. It requires both inequalities to be equalities simultaneously. The first is an equality precisely when $q_x/p_x = 1$ for each $x\in S$, the second precisely when $\sum_{x\notin S} q_x = 0$. The former yields $p_x=q_x$ on $S$; conversely, if that holds then $\sum_{x\in S}q_x = 1$, so the latter is automatic and $D(p\|q)=0$. Finally, the identity $H(p,q) = H(p)+D(p\|q)$ from Definition 5.1 gives $H(p,q)\ge H(p)$.

∎

In practice this proposition means the following: minimising the cross-entropy is the same as minimising the KL divergence. Since $H(p)$ is a constant determined by the data, the only part training can move is $D(p\|q)$. The attainable loss therefore has a lower bound $H(p)$, which is the intrinsic ambiguity of the data itself, namely the fact that the same context admits several continuations. When the loss refuses to fall further, this gives a way to tell whether the model lacks capacity or the data is intrinsically ambiguous.

Example 5.3（Computing perplexity by hand）

Perplexity is $\mathrm{PPL} = \exp\bigl(\frac{1}{N}\sum_{i=1}^{N} -\log q(x_i)\bigr)$, the exponential of the average cross-entropy. Suppose the model assigns probabilities $0.5,\ 0.25,\ 0.1,\ 0.05$ to four tokens. Then

$$-\log 0.5 = 0.6931,\quad -\log 0.25 = 1.3863,\quad -\log 0.1 = 2.3026,\quad -\log 0.05 = 2.9957 .$$

The sum is $7.3777$ and the average is $1.8444$ nats, so $\mathrm{PPL} = e^{1.8444} \approx 6.32$.

This $6.32$ reads as “hesitating among roughly 6.3 options each time”. Indeed, for a uniform distribution over a vocabulary of $V$ items we always have $-\log(1/V) = \log V$, so $\mathrm{PPL} = V$: perplexity is a measure of the effective number of choices. The meaning of a drop of $0.1$ in the loss can be translated the same way: since $e^{0.1}\approx 1.105$, the effective number of choices has fallen by about 10%.

5.2. Bayes’ theorem: when 99% accuracy is useless

One further tool is indispensable when evaluating AI systems: the Bayesian update rule.

Example 5.4（How trustworthy is an alert from a model with 99% detection and 1% false-alarm rate）

Consider a model for detecting fraudulent transactions. Let the prior probability (the fraction that really are fraudulent) be $\Pr[D] = 0.001$, the detection rate (the probability of correctly alerting on fraud) $\Pr[+\mid D] = 0.99$, and the false-alarm rate (the probability of wrongly alerting on a legitimate transaction) $\Pr[+\mid \lnot D] = 0.01$. When an alert fires, what is the probability $\Pr[D\mid +]$ that it really is fraud?

By Bayes’ theorem,

$$\Pr[D\mid +] = \frac{\Pr[+\mid D]\Pr[D]}{\Pr[+\mid D]\Pr[D] + \Pr[+\mid\lnot D]\Pr[\lnot D]} .$$

The numerator is $0.99\times 0.001 = 0.00099$ and the second term of the denominator is $0.01\times0.999 = 0.00999$, so

$$\Pr[D\mid+] = \frac{0.00099}{0.00099+0.00999} = \frac{0.00099}{0.01098} \approx 0.0902 .$$

That is about 9%: one alert in eleven is genuine. The operators’ impression that “nine out of ten are false” is correct, and the report of “99% accuracy” is also correct (in a different sense). The two disagree because the prior of $0.001$ dominates.

The formula also tells us where to improve. The denominator is dominated by the term $\Pr[+\mid\lnot D]\Pr[\lnot D]$, so raising the detection rate from $0.99$ to $0.999$ changes almost nothing (it reaches only $0.0910$). What matters is the false-alarm rate: lowering $0.01$ to $0.001$ gives $\Pr[D\mid+] = 0.00099/(0.00099+0.000999) \approx 0.498$, about 50%. Which number to improve is settled by this computation alone. A comparison of the same shape, carried out in the context of delegating work to an LLM, is which to improve: false positive rate or success rate（Example 7.3）［LLMs and Programming］.

Caution

Discussing evaluation in terms of accuracy alone is close to meaningless for skewed distributions like the one above: a model that labels everything “legitimate” achieves 99.9% accuracy. Speak instead in terms of precision and recall, or of posterior probabilities with the prior stated explicitly. A framework that treats human inspection and testing as “verifiers with a false positive rate” is given in verifiers and false positive rates（Definition 2.2）［LLMs and Programming］.

6. Abstraction and logic: the order of quantifiers decides the specification

So far we have discussed “mathematics for understanding AI”. But the benefit of relearning mathematics is not only its content. Training in stating claims precisely is itself the ability to write specifications and analyse incidents. (The view that stating a specification necessarily carries an irreducible amount of information is summarised in a lower bound on specification length（Proposition 6.2）［A Survival Strategy for Software Engineers in the AI Era］.)

The training with the greatest practical payoff is attending to the order of quantifiers, $\forall$ (“for all”) and $\exists$ (“there exists”). Compare the following two definitions.

Definition 6.1（Continuity and uniform continuity）

Let $I\subset\mathbb{R}$ be an interval and $f: I\to\mathbb{R}$.

We say $f$ is continuous on $I$ if

$$\forall a\in I,\ \forall \varepsilon > 0,\ \exists \delta > 0,\ \forall x\in I:\quad |x-a| < \delta \implies |f(x)-f(a)| < \varepsilon$$

holds.

We say $f$ is uniformly continuous on $I$ if

$$\forall \varepsilon > 0,\ \exists \delta > 0,\ \forall a\in I,\ \forall x \in I:\quad |x-a| < \delta \implies |f(x)-f(a)| < \varepsilon$$

holds.

The two formulas differ only in the order of $\exists\delta$ and $\forall a$. For continuity, $\delta$ may be chosen afresh for each $a$; for uniform continuity, a single $\delta$ must work independently of $a$. The difference is essential.

Example 6.2（A continuous function that is not uniformly continuous）

Take $I = (0,1]$ and $f(x) = 1/x$. Then $f$ is continuous on $I$, since the denominator never vanishes at any point $a>0$. It is not uniformly continuous, however.

Here is the proof. Put $\varepsilon_0 = 1$. For an arbitrary $\delta > 0$, choose a natural number $n$ large enough that $1/(n(n+1)) < \delta$ (possible since $n(n+1)\to\infty$), and set $x_n = 1/n$ and $a_n = 1/(n+1)$. Then

$$|x_n - a_n| = \frac{1}{n}-\frac{1}{n+1} = \frac{1}{n(n+1)} < \delta,$$

whereas

$$|f(x_n)-f(a_n)| = |n - (n+1)| = 1 \ge \varepsilon_0 .$$

Thus “for $\varepsilon_0=1$, whatever $\delta$ is chosen there is a counterexample pair”, and the definition of uniform continuity fails. Because the slope grows steeper as one approaches the origin, $\varepsilon$ cannot be respected unless $\delta$ is allowed to depend on $a$.

This structure appears verbatim in system specifications.

• “For every request there is some server that responds within 200ms” — the responding server may differ from request to request.
• “There is some server that responds to every request within 200ms” — a single server can carry the entire load.

The second is a far stronger claim than the first. SLAs, permission design (“every user is assigned some role” versus “some role is assigned to every user”), retry design: there is no end to the places where mistaking the order of quantifiers breaks the design. Natural language has loose word order and can express both with the same sentence. That is exactly why the habit of rewriting into logical formulas and checking pays off.

The other thing mathematics trains is the habit of constructing counterexamples. What we did in Example 6.2 was to exhibit concretely what breaks when one hypothesis is dropped. That exercise has the same shape as designing evaluations for AI systems. Against the claim “this prompt works”, search systematically for counterexamples, that is, inputs on which it fails: boundary values, empty input, extremely long input, unexpected languages. For LLM outputs, this search for counterexamples is exactly the detection of plausible errors（Definition 7.1）［LLMs and Programming］. The habit of doubting a claim and building the smallest counterexample, acquired through mathematical exercises, is test case design.

Tip

When you read a theorem, try removing the hypotheses one at a time and asking what breaks. Remove convexity from Theorem 4.3 and the limit becomes a local minimum, losing any estimate against the global minimum. Remove $L$-smoothness and Lemma 4.2 is unavailable, so there is no guarantee that a single step decreases the loss at all. This way of reading turns a theorem from something to memorise into a tool you can use.

7. What to relearn, and in what order

Finally, a realistic path for practitioners. The goal is not to read a textbook cover to cover, but to be able to carry out computations like the ones above on your own.

Order	Field	Minimum target	Question that confirms you are there
1	Linear algebra	Compute matrix products, norms, inner products, eigenvalues and the SVD	Can you solve Example 3.9 without looking?
2	Probability and statistics	Conditional probability, Bayes’ theorem, expectation and variance, likelihood	Can you derive the improvement strategy of Example 5.4 yourself?
3	Calculus	Partial derivatives, gradients, the chain rule, Taylor expansion to second order	Can you explain where the $2/L$ of Lemma 4.2 comes from?
4	Logic and set theory	Order of quantifiers, forming negations, constructing counterexamples	Can you explain to someone the difference between the two formulas in Definition 6.1?

The order has reasons. Linear algebra comes first because it carries the largest number of applications you can use today: embeddings, attention, low-rank approximation. Probability comes second because it bears directly on evaluation and decision-making, and because misunderstandings there are expensive, as Example 5.4 shows. Calculus becomes necessary once you step inside training. Logic is placed last, though in truth it is at work from the beginning.

One judgement about how to study. It is better to set aside time for computing by hand, not only for reading proofs. Fifteen minutes decomposing the $3\times2$ matrix of Example 3.9 on paper will stay with you longer than reading ten expositions of the SVD. NumPy is convenient for checking the numbers.

import numpy as np

A = np.array([[1.0, 1.0],
              [1.0, 1.0],
              [1.0, -1.0]])

U, s, Vt = np.linalg.svd(A, full_matrices=False)
print(s)                      # [2.         1.41421356]

A1 = s[0] * np.outer(U[:, 0], Vt[0, :])
print(np.round(A1, 6))        # [[1. 1.] [1. 1.] [0. 0.]]
print(np.linalg.norm(A - A1)) # 1.4142135623730951  = sigma_2

This agrees with the hand computation. The round trip of “solve by hand, then verify with code” is what makes the material stick.

The overall strategy for continuing to add value in the age of AI is set out in A Survival Strategy for Software Engineers in the AI Era. Where automation hits a ceiling, and what that ceiling is, is given by the limit of automation（Corollary 3.2）［A Survival Strategy for Software Engineers in the AI Era］. Mathematics is one pillar within it. Think of it as an investment made not in order to become a person who builds models, but in order to stand on the side that doubts, explains and controls what the models produce.

8. Exercises

Exercise 8.1Easy

For two vectors $\boldsymbol{u},\boldsymbol{v}$ chosen independently and uniformly from the unit sphere in $d = 768$ dimensions, find the standard deviation of the cosine similarity. Then use Chebyshev’s inequality to bound $\Pr[|\cos(\boldsymbol{u},\boldsymbol{v})| \ge 0.1]$ from above, and compare with the bound obtained from Theorem 3.3.

Solution

Since the vectors are unit vectors, the cosine similarity is the inner product itself. By Proposition 3.2 the variance is $1/768$, so the standard deviation is $1/\sqrt{768} \approx 0.0361$.

Chebyshev’s inequality states that for a random variable $X$ with mean $0$ and variance $\sigma^2$ one has $\Pr[|X|\ge t]\le \sigma^2/t^2$. Substituting $t = 0.1$,

$$\Pr[|\langle\boldsymbol{u},\boldsymbol{v}\rangle|\ge 0.1] \le \frac{1/768}{0.01} = \frac{1}{7.68} \approx 0.130 .$$

On the other hand Theorem 3.3 gives

$$2e^{-768\times 0.01/2} = 2e^{-3.84} \approx 2\times 0.02147 \approx 0.0429 .$$

The exponential estimate is roughly three times stronger. The gap widens as $t$ grows: at $t=0.3$ the former gives $0.0145$ while the latter gives $2e^{-34.56}\approx 2\times10^{-15}$. Chebyshev uses only the variance and so cannot capture high-dimensional concentration.

Exercise 8.2Standard

Apply gradient descent with learning rate $\eta$ to $f(x) = \frac{a}{2}x^2$ with $a>0$. (1) Express the condition for convergence in terms of $a$ and $\eta$. (2) For $a = 4$ and $\eta = 0.1$, find $|x_{10}/x_0|$. (3) Describe what happens when $a = 4$ and $\eta = 0.5$.

Solution

(1) The computation is the same as in Example 4.4. Since $f'(x) = ax$ we get $x_{k+1} = (1-\eta a)x_k$ and hence $x_k = (1-\eta a)^k x_0$. For $x_0\ne0$, the condition $x_k\to0$ is $|1-\eta a| < 1$, that is, $0 < \eta < 2/a$. As $f$ is $L$-smooth with $L = a$, this agrees with the condition $0<\eta<2/L$ of Lemma 4.2.

(2) Here $1-\eta a = 1 - 0.1\times 4 = 0.6$, so

$$\left|\frac{x_{10}}{x_0}\right| = 0.6^{10} = 0.0060466\ldots \approx 6.0\times10^{-3} .$$

Since $f\propto x^2$, the loss shrinks by a factor of $0.6^{20}\approx 3.66\times10^{-5}$.

(3) Here $1-\eta a = 1 - 0.5\times4 = -1$, so $x_k = (-1)^k x_0$. The step sits exactly on the boundary $\eta = 2/a$, so the point oscillates forever between $x_0$ and $-x_0$ and the loss never decreases (nor does it diverge). Computing the coefficient $\eta(1-L\eta/2)$ of the decrease in Lemma 4.2 gives $0.5\times(1-4\times0.5/2) = 0.5\times 0 = 0$, consistent with the lemma guaranteeing a decrease of $0$. In practice this is worth remembering as a candidate cause when the loss stops falling and oscillates around a constant value.

Exercise 8.3Standard

A model has validation loss (average cross-entropy, in nats) equal to $2.0$. (1) Find the perplexity. (2) Find the perplexity and loss of a uniform model over a vocabulary of $50{,}000$ tokens, and compare with (1). (3) When the loss improves from $2.0$ to $1.9$, by what percentage does the perplexity fall?

Solution

(1) $\mathrm{PPL} = e^{2.0} = 7.389\ldots \approx 7.39$: effectively, hesitating among about 7.4 options.

(2) For the uniform distribution each token has probability $1/50000$, so the loss is $-\log(1/50000) = \log 50000$. Now $\log 50000 = \log 5 + 4\log 10 = 1.6094 + 9.2103 = 10.8197$ nats, and the perplexity is $e^{10.8197} = 50000$. In the notation of Definition 5.1, this is $H(p,q)$ with $q$ taken to be uniform. The trained model has narrowed $50000$ options down to an effective $7.39$.

(3) The ratio of perplexities is $e^{1.9}/e^{2.0} = e^{-0.1} = 0.9048$, a decrease of about $9.5\%$. Looking only at the difference in losses it appears to be “a 5% improvement”, but in perplexity it is about 9.5%. Remember it as: for an absolute loss difference $\Delta$, the perplexity is multiplied by $e^{-\Delta}$.

Exercise 8.4Hard

On $\mathcal{X}=\{0,1\}$ let $p = (0.5,\,0.5)$ and $q = (0.9,\,0.1)$. (1) Compute $D(p\|q)$ and $D(q\|p)$ and confirm that the KL divergence is not symmetric. (2) Discuss, for the case of minimising $D(p\|q)$, whether this asymmetry corresponds to “averaging the modes” or to “picking a single mode” in the training of a generative model.

Solution

(1) Following Definition 5.1,

$$D(p\|q) = 0.5\log\frac{0.5}{0.9} + 0.5\log\frac{0.5}{0.1} = 0.5\times(-0.5878) + 0.5\times 1.6094 = -0.2939+0.8047 = 0.5108 .$$$$D(q\|p) = 0.9\log\frac{0.9}{0.5} + 0.1\log\frac{0.1}{0.5} = 0.9\times 0.5878 + 0.1\times(-1.6094) = 0.5290 - 0.1609 = 0.3681 .$$

Since $0.5108 \ne 0.3681$, it is not symmetric. That both are positive is consistent with Proposition 5.2.

(2) Consider minimising $D(p\|q) = \sum_x p_x\log(p_x/q_x)$ over $q$. If $p_x$ is large while $q_x$ is extremely small, then $\log(p_x/q_x)$ is large and the penalty is heavy. Conversely, at points where $p_x = 0$ the term is $0$ no matter how large $q_x$ is, so there is no penalty. Minimising $D(p\|q)$ therefore pushes $q$ to cover every place where $p$ puts positive probability. This is mode-averaging: if $p$ has two well-separated peaks, $q$ ends up assigning probability to the region between them as well.

Ordinary language model training minimises the cross-entropy $H(p,q)$, and as we saw in Proposition 5.2 we have $H(p,q) = H(p) + D(p\|q)$, so this is the same as minimising $D(p\|q)$. The tendency of trained models to return safe, average outputs owes something to this asymmetry of the objective. Minimising the reverse divergence $D(q\|p)$ instead makes $q$ concentrate on one of the peaks of $p$, that is, produces mode-seeking behaviour.

References

• G. Strang, Introduction to Linear Algebra, 5th ed., Wellesley-Cambridge Press, 2016 — Chapter 7 (singular value decomposition and low-rank approximation).
• G. H. Golub and C. F. Van Loan, Matrix Computations, 4th ed., Johns Hopkins University Press, 2013 — Chapter 2 (matrix norms and best low-rank approximation).
• S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004 — Chapter 9 (unconstrained minimisation and descent methods). The full text is available on the authors’ site: https://web.stanford.edu/~boyd/cvxbook/
• T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006 — Chapter 2 (entropy, relative entropy and mutual information).
• R. Vershynin, High-Dimensional Probability: An Introduction with Applications in Data Science, Cambridge University Press, 2018 — Chapter 3 (concentration of measure on the sphere and in high-dimensional distributions).
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016 — Chapters 2–4 (foundations of linear algebra, probability and numerical computation). Online edition: https://www.deeplearningbook.org/
• E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, arXiv:2106.09685 (2021). https://arxiv.org/abs/2106.09685