Random Variables and Expectation: Measurable Functions and the Lebesgue Integral

0. Key points

• A random variable is a measurable function from a probability space to the real line. Measurability is demanded so that quantities such as $P(X \le a)$ make sense, that is, so that $\{\omega : X(\omega) \le a\}$ is an event to which a probability has been assigned.
• A random variable $X$ induces a distribution $P_X$ (its image measure) on the real line. The distinction between “discrete” and “continuous” is nothing more than the question of whether $P_X$ is concentrated on a countable set or has a density with respect to Lebesgue measure.
• The expectation is the Lebesgue integral $E[X] = \int_\Omega X \, dP$. It is built in four stages: indicator functions, then nonnegative simple functions, then nonnegative measurable functions, then integrable functions.
• The transfer formula $E[g(X)] = \int_{\mathbb{R}} g \, dP_X$ translates an integral over an abstract $\Omega$ into a familiar sum or an integral over the real line. Every actual computation passes through this theorem.
• Variance and covariance are precisely the inner-product structure of $L^2$. The correlation coefficient $\rho$ is the cosine of the angle between two centred random variables, and $|\rho| \le 1$ is nothing but the Cauchy–Schwarz inequality.
• Chebyshev’s inequality bounds tail probabilities using the variance alone, with no assumption whatever on the shape of the distribution. The weak law of large numbers is a two-line consequence of it.

1. Motivation

1.1. “A variable determined at random” is not a definition

In high-school texts and in elementary statistics, a random variable is described as “a variable whose value is determined by the outcome of a trial”. The description is intuitively correct, but as a mathematical object it defines nothing: neither “variable” nor “determined at random” has been given a meaning.

The resolution Kolmogorov gave in his 1933 Grundbegriffe der Wahrscheinlichkeitsrechnung is almost disappointingly simple. Push all the randomness into the probability measure $P$, and make the random variable itself a deterministic function. A point $\omega$ of the sample space $\Omega$ records which outcome occurred, and $X(\omega)$ is the number attached to that outcome. Only the selection of $\omega$ is random; the assignment rule $X$ contains no randomness at all.

For instance, for the experiment of rolling two dice we may take $\Omega = \{1,\dots,6\}^2$, and the sum of the faces is the completely deterministic function $X(\omega_1,\omega_2) = \omega_1 + \omega_2$. The “probability that the sum is $7$” is the $P$-measure of the set $X^{-1}(\{7\})$.

1.2. Why measurability is required

Once this point of view is adopted, the condition to impose on a random variable becomes visible by itself. What we wish to compute are quantities such as $P(X \le a)$ and $P(X \in B)$. But the probability measure $P$ is not defined on all subsets of $\Omega$; it is defined only on the sets (events) belonging to the $\sigma$-algebra $\mathcal{F}$. Hence unless

$$\{\omega \in \Omega : X(\omega) \le a\} \in \mathcal{F}$$

holds, the symbol $P(X \le a)$ is meaningless. Requiring this for every Borel subset of $\mathbb{R}$ is exactly measurability, and it becomes the definition of a random variable.

This condition is by no means automatic. Let $\Omega = [0,1]$, let $\mathcal{F}$ be the Lebesgue measurable sets, let $P$ be Lebesgue measure, and let $V \subset [0,1]$ be a Vitali non-measurable set. Then $X = \mathbf{1}_V$ is a real-valued function on $[0,1]$, but $X^{-1}(\{1\}) = V \notin \mathcal{F}$, so $P(X = 1)$ cannot be defined. Measurability is “the minimal etiquette that excludes pathological functions”. For details see Probability spaces and Kolmogorov’s axioms, and in particular the definition of a σ-algebra and a measurable space（Definition 3.1）［Probability Spaces and Kolmogorov's Axioms］.

1.3. Why the expectation is defined as a Lebesgue integral

Elementary probability defines the expectation in two separate ways: $\sum_k x_k p_k$ in the discrete case and $\int x f(x)\,dx$ in the continuous case. This carries three disadvantages.

First, random variables of neither kind arise constantly in practice. The payout of an insurance policy with a deductible, for example, is a mixture: an atom of positive probability at $0$ with a continuous distribution sitting above it. Second, every theorem has to be proved twice, once for each case. Third, and most importantly, the law of large numbers and the central limit theorem deal with limits of sequences of random variables, and the theorems guaranteeing that limits may be interchanged with integrals (monotone convergence, dominated convergence) are available only within the Lebesgue framework.

Defining the expectation as the single Lebesgue integral $E[X] = \int_\Omega X \, dP$ resolves all three at once. Below we follow that construction, starting from approximation by simple functions.

2. Preliminaries

2.1. Notation and standing assumptions

Throughout, $(\Omega, \mathcal{F}, P)$ is a probability space (Definition 4.1［Probability Spaces and Kolmogorov's Axioms］): $\Omega$ is a nonempty set, $\mathcal{F}$ is a $\sigma$-algebra on $\Omega$, and $P : \mathcal{F} \to [0,1]$ is a countably additive measure with $P(\Omega) = 1$. The $\sigma$-algebra generated by the open subsets of $\mathbb{R}$ is written $\mathcal{B}(\mathbb{R})$ and called the Borel $\sigma$-algebra; its members are Borel sets. We set $\mathbb{N} = \{1, 2, \ldots\}$.

The indicator function of a set $A$ is written $\mathbf{1}_A$: it equals $1$ when $\omega \in A$ and $0$ otherwise. Following the usual convention, sets describing events are abbreviated as $\{X \le a\} := \{\omega \in \Omega : X(\omega) \le a\}$.

2.2. Facts taken as known

We use the following theorem, established in The Lebesgue integral and the convergence theorems, as known; the proof is left to that article.

Theorem 2.1（Monotone convergence theorem）

Let $(\Omega, \mathcal{F}, \mu)$ be a measure space and let $X_n : \Omega \to [0, \infty]$ be a sequence of measurable functions. If $0 \le X_1(\omega) \le X_2(\omega) \le \cdots$ and $X_n(\omega) \to X(\omega)$ for every $\omega \in \Omega$, then $X$ is measurable and

$$\lim_{n \to \infty} \int_\Omega X_n \, d\mu = \int_\Omega X \, d\mu$$

holds (as an equality, including the case where both sides are $+\infty$).

Remark 2.2

The proof is the monotone convergence theorem (Beppo Levi)（Theorem 5.2）［ルベーグ積分の定義と収束定理］ in The Lebesgue integral and the convergence theorems. Wherever below we write “interchange the limit and the integral as $n \to \infty$”, the justification is, unless stated otherwise, an application of Theorem 2.1.

2.3. Independence

To handle additivity of the variance in Section 5, we record independence in the language of random variables.

Definition 2.3（Independence of random variables）

Random variables $X_1, \ldots, X_n$ are independent if for all Borel sets $B_1, \ldots, B_n \in \mathcal{B}(\mathbb{R})$

$$P\bigl(X_1 \in B_1,\ \ldots,\ X_n \in B_n\bigr) = \prod_{i=1}^{n} P(X_i \in B_i)$$

holds. An infinite family of random variables is independent if every finite subfamily is independent.

3. Random variables and their distributions

3.1. Definition and criteria for measurability

Definition 3.1（Random variable）

Let $(\Omega, \mathcal{F}, P)$ be a probability space. A map $X : \Omega \to \mathbb{R}$ is a random variable (equivalently, an $\mathcal{F}$-measurable function) if

$$\forall B \in \mathcal{B}(\mathbb{R}), \quad X^{-1}(B) := \{\omega \in \Omega : X(\omega) \in B\} \in \mathcal{F}$$

holds. If the range is enlarged to $[-\infty, +\infty]$, one uses the Borel $\sigma$-algebra of $[-\infty,+\infty]$ in place of $\mathcal{B}(\mathbb{R})$ and speaks of an extended-real-valued random variable.

Verifying the definition literally would require checking $X^{-1}(B) \in \mathcal{F}$ for every member of $\mathcal{B}(\mathbb{R})$, which is not feasible: the Borel sets include a vast supply of sets with no explicit description. The next proposition guarantees that it suffices to check on a generating family, and thereby turns the verification of measurability into a practical task.

Proposition 3.2（Measurability criterion via a generating family）

Let $\mathcal{E} \subset 2^{\mathbb{R}}$ be a family of sets with $\sigma(\mathcal{E}) = \mathcal{B}(\mathbb{R})$. If a map $X : \Omega \to \mathbb{R}$ satisfies

$$\forall E \in \mathcal{E}, \quad X^{-1}(E) \in \mathcal{F}$$

then $X$ is a random variable.

Proof（Proposition 3.2）

Put $\mathcal{G} := \{B \subset \mathbb{R} : X^{-1}(B) \in \mathcal{F}\}$; we show that $\mathcal{G}$ is a $\sigma$-algebra on $\mathbb{R}$. The key is the elementary fact that preimages commute with all set operations.

(i) $X^{-1}(\mathbb{R}) = \Omega \in \mathcal{F}$, so $\mathbb{R} \in \mathcal{G}$.

(ii) Let $B \in \mathcal{G}$. The statement $\omega \in X^{-1}(B^c)$ is equivalent to $X(\omega) \notin B$, that is, to $\omega \notin X^{-1}(B)$; hence $X^{-1}(B^c) = (X^{-1}(B))^c$. Since $\mathcal{F}$ is closed under complements, $(X^{-1}(B))^c \in \mathcal{F}$, so $B^c \in \mathcal{G}$.

(iii) Let $B_1, B_2, \ldots \in \mathcal{G}$. The statement $X(\omega) \in \bigcup_n B_n$ is equivalent to ”$X(\omega) \in B_n$ for some $n$”, so $X^{-1}\bigl(\bigcup_n B_n\bigr) = \bigcup_n X^{-1}(B_n)$. Since $\mathcal{F}$ is closed under countable unions, this lies in $\mathcal{F}$, whence $\bigcup_n B_n \in \mathcal{G}$.

Therefore $\mathcal{G}$ is a $\sigma$-algebra. By hypothesis $\mathcal{E} \subset \mathcal{G}$, and $\sigma(\mathcal{E})$ is the smallest $\sigma$-algebra containing $\mathcal{E}$, so $\mathcal{B}(\mathbb{R}) = \sigma(\mathcal{E}) \subset \mathcal{G}$. This says that $X^{-1}(B) \in \mathcal{F}$ for every Borel set $B$, which is the condition in Definition 3.1.

∎

Corollary 3.3（Criterion via half-lines）

A map $X : \Omega \to \mathbb{R}$ is a random variable if and only if

$$\forall a \in \mathbb{R}, \quad \{X \le a\} \in \mathcal{F}.$$

Proof（Corollary 3.3）

Necessity follows from Definition 3.1, since $(-\infty, a]$ is a Borel set. For sufficiency, put $\mathcal{E} := \{(-\infty, a] : a \in \mathbb{R}\}$; we claim $\sigma(\mathcal{E}) = \mathcal{B}(\mathbb{R})$. Indeed, every open interval can be produced from members of $\mathcal{E}$ by countably many set operations:

$$(a, b) = \Bigl(\bigcup_{n \ge n_0} (-\infty, b - 1/n]\Bigr) \setminus (-\infty, a]$$

(where $n_0$ is a natural number large enough that $b - 1/n_0 > a$), and every open subset of $\mathbb{R}$ is a countable union of open intervals. Hence $\sigma(\mathcal{E})$ contains all open sets, so $\mathcal{B}(\mathbb{R}) \subset \sigma(\mathcal{E})$. The reverse inclusion is clear because $(-\infty,a]$ is closed. It now suffices to apply Proposition 3.2 to $\mathcal{E}$.

∎

We must also check that operations on random variables produce random variables again. Without this we could not even write “the expectation of $X + Y$”.

Proposition 3.4（Closure of random variables under operations）

Let $X, Y$ be random variables on a probability space $(\Omega, \mathcal{F}, P)$ and let $c \in \mathbb{R}$. Then $cX$, $X + Y$, $XY$ and $\max(X, Y)$ are all random variables. Moreover, for a sequence $(X_n)_{n \in \mathbb{N}}$ of random variables, $\sup_n X_n$, $\inf_n X_n$, $\limsup_n X_n$ and $\liminf_n X_n$ are extended-real-valued random variables.

Proof（Proposition 3.4）

Everything is reduced to Corollary 3.3.

Sum. For every $a \in \mathbb{R}$,

$$\{X + Y < a\} = \bigcup_{q \in \mathbb{Q}} \bigl(\{X < q\} \cap \{Y < a - q\}\bigr).$$

Indeed, if $X(\omega) + Y(\omega) < a$ then $X(\omega) < a - Y(\omega)$, so by density of the rationals there is $q \in \mathbb{Q}$ with $X(\omega) < q < a - Y(\omega)$, placing $\omega$ in the right-hand side; the reverse inclusion is clear. Since $\{X < q\} = \bigcup_{n} \{X \le q - 1/n\} \in \mathcal{F}$ and $\mathbb{Q}$ is countable, the right-hand side lies in $\mathcal{F}$. Consequently $\{X+Y \le a\} = \bigcap_n \{X + Y < a + 1/n\} \in \mathcal{F}$.

Scalar multiple. If $c > 0$ then $\{cX \le a\} = \{X \le a/c\}$; if $c < 0$ then $\{cX \le a\} = \{X \ge a/c\} = \{X < a/c\}^c$; if $c = 0$ then $cX \equiv 0$. In each case the set lies in $\mathcal{F}$.

Product. First consider $X^2$: if $a < 0$ then $\{X^2 \le a\} = \emptyset$, and if $a \ge 0$ then $\{X^2 \le a\} = \{-\sqrt{a} \le X \le \sqrt{a}\} \in \mathcal{F}$. So $X^2$ is a random variable, and by the identity $XY = \frac{1}{2}\bigl((X+Y)^2 - X^2 - Y^2\bigr)$ together with the closure under sums and scalar multiples already proved, $XY$ is a random variable too.

Maximum. $\{\max(X,Y) \le a\} = \{X \le a\} \cap \{Y \le a\} \in \mathcal{F}$.

Supremum and infimum. $\{\sup_n X_n \le a\} = \bigcap_n \{X_n \le a\}$ is a countable intersection, hence lies in $\mathcal{F}$; the infimum follows from $\inf_n X_n = -\sup_n(-X_n)$. Finally $\limsup_n X_n = \inf_{m} \sup_{n \ge m} X_n$ and $\liminf_n X_n = \sup_m \inf_{n \ge m} X_n$, so applying the supremum and infimum results twice gives the claim.

∎

Tip

The last statement matters more than it appears to. The law of large numbers has the form “the set of $\omega$ with $\bar{X}_n \to \mu$ has probability $1$”, and it is Proposition 3.4 that guarantees this set is an event, that is, $\{\limsup_n \bar{X}_n = \liminf_n \bar{X}_n = \mu\} \in \mathcal{F}$.

3.2. Distribution (image measure) and distribution function

The “probabilistic information” carried by a random variable does not depend on what kind of set $\Omega$ is. It is determined solely by how much probability mass is placed where on the real line. The object that extracts this is the distribution.

Definition 3.5（Distribution and distribution function）

Let $X$ be a random variable on a probability space $(\Omega, \mathcal{F}, P)$. The set function $P_X$ on $\mathcal{B}(\mathbb{R})$ defined by

$$P_X(B) := P\bigl(X^{-1}(B)\bigr) \qquad (B \in \mathcal{B}(\mathbb{R}))$$

is called the distribution (or image measure, or law) of $X$. Furthermore

$$F_X(a) := P(X \le a) = P_X\bigl((-\infty, a]\bigr) \qquad (a \in \mathbb{R})$$

is called the distribution function of $X$.

Proposition 3.6（The distribution is a probability measure, determined by its distribution function）

The set function $P_X$ of Definition 3.5 is a probability measure on $(\mathbb{R}, \mathcal{B}(\mathbb{R}))$. Moreover its distribution function $F_X$ has the following three properties.

1. Nondecreasing: $a \le b \implies F_X(a) \le F_X(b)$.
2. Right-continuous: $\lim_{h \downarrow 0} F_X(a + h) = F_X(a)$ for every $a$.
3. Limits at the ends: $\lim_{a \to -\infty} F_X(a) = 0$ and $\lim_{a \to +\infty} F_X(a) = 1$.

Proof（Proposition 3.6）

It is a measure. $P_X(B) = P(X^{-1}(B)) \ge 0$ by nonnegativity of $P$, and $P_X(\mathbb{R}) = P(\Omega) = 1$ because $X^{-1}(\mathbb{R}) = \Omega$. For countable additivity, let $B_1, B_2, \ldots \in \mathcal{B}(\mathbb{R})$ be pairwise disjoint; then their preimages are pairwise disjoint as well, since $\omega \in X^{-1}(B_i) \cap X^{-1}(B_j)$ would give $X(\omega) \in B_i \cap B_j = \emptyset$, a contradiction. Hence, using the identity $X^{-1}(\bigcup_n B_n) = \bigcup_n X^{-1}(B_n)$ from the proof of Proposition 3.2 together with countable additivity of $P$,

$$P_X\Bigl(\bigsqcup_n B_n\Bigr) = P\Bigl(\bigsqcup_n X^{-1}(B_n)\Bigr) = \sum_n P(X^{-1}(B_n)) = \sum_n P_X(B_n).$$

Property 1. If $a \le b$ then $(-\infty,a] \subset (-\infty,b]$, so monotonicity of measures applies.

Property 2. Let $h_n \downarrow 0$ be any decreasing sequence; then $(-\infty, a + h_n] \downarrow (-\infty, a]$. Since $P_X$ is a finite measure, continuity from above applies and $F_X(a + h_n) = P_X((-\infty,a+h_n]) \to P_X((-\infty,a]) = F_X(a)$. As $F_X$ is monotone, validity along every decreasing sequence gives the limit as $h \downarrow 0$.

Property 3. From $(-\infty, -n] \downarrow \emptyset$ and continuity from above, $F_X(-n) \to P_X(\emptyset) = 0$; from $(-\infty, n] \uparrow \mathbb{R}$ and continuity from below, $F_X(n) \to P_X(\mathbb{R}) = 1$. Monotonicity makes this sufficient.

∎

Remark 3.7

Conversely, given any $F : \mathbb{R} \to [0,1]$ with properties 1–3, there is exactly one probability measure having $F$ as its distribution function (the Lebesgue–Stieltjes construction). Thus “distributions of random variables” and “functions with properties 1–3” correspond one to one. The construction uses the same outer-measure machinery as in Measurable sets and Lebesgue measure.

3.3. Discrete and continuous types

Definition 3.8（Discrete and continuous types）

A random variable $X$ is discrete if there is a countable set $S = \{x_1, x_2, \ldots\} \subset \mathbb{R}$ with $P(X \in S) = 1$. In this case $p_k := P(X = x_k)$ is called the probability mass function.

$X$ is continuous (absolutely continuous) if there is a Borel measurable $f : \mathbb{R} \to [0,\infty)$ with

$$P_X(B) = \int_B f(x) \, dx \qquad (\forall B \in \mathcal{B}(\mathbb{R})),$$

where the right-hand side is a Lebesgue integral. Such an $f$ is called a probability density function.

These two do not exhaust the possibilities. There are random variables whose distribution function is continuous but which have no density (the Cantor distribution), and there are mixtures of a discrete and a continuous part. Defining the expectation by the Lebesgue integral is stronger than the elementary dichotomy precisely because it treats all of these without distinction.

Example 3.9（Standard discrete distributions）

Bernoulli distribution $\mathrm{Ber}(p)$ (with $0 \le p \le 1$): $P(X=1)=p$, $P(X=0)=1-p$. It records whether a single trial succeeded.

Binomial distribution $\mathrm{Bin}(n,p)$: $P(X=k) = \binom{n}{k} p^k (1-p)^{n-k}$ for $k = 0,1,\ldots,n$. That this is a probability mass function follows from the binomial theorem:

$$\sum_{k=0}^{n} \binom{n}{k} p^k (1-p)^{n-k} = \bigl(p + (1-p)\bigr)^n = 1 .$$

It is the distribution of the number of successes in $n$ independent trials with success probability $p$.

Poisson distribution $\mathrm{Po}(\lambda)$ (with $\lambda > 0$): $P(X=k) = e^{-\lambda} \dfrac{\lambda^k}{k!}$ for $k = 0,1,2,\ldots$. The total mass is $1$ by the Taylor expansion of the exponential:

$$\sum_{k=0}^{\infty} e^{-\lambda} \frac{\lambda^k}{k!} = e^{-\lambda} e^{\lambda} = 1 .$$

It arises as the limit of the binomial distribution when $n \to \infty$, $p \to 0$ and $np \to \lambda$ (the law of small numbers), and it is used to model the number of rare events per unit time.

Example 3.10（Standard continuous distributions）

Exponential distribution $\mathrm{Exp}(\lambda)$ (with $\lambda > 0$): density $f(x) = \lambda e^{-\lambda x} \mathbf{1}_{\{x > 0\}}$. The total integral is

$$\int_0^\infty \lambda e^{-\lambda x} \, dx = \Bigl[-e^{-\lambda x}\Bigr]_0^\infty = 0 - (-1) = 1 .$$

Its distribution function is $F(a) = 1 - e^{-\lambda a}$ for $a \ge 0$, and it is memoryless: $P(X > s + t \mid X > s) = e^{-\lambda t} = P(X > t)$.

Normal distribution $N(\mu, \sigma^2)$ (with $\mu \in \mathbb{R}$ and $\sigma > 0$): density

$$f(x) = \frac{1}{\sqrt{2\pi}\,\sigma} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right).$$

That the total integral is $1$ follows by substituting $z = (x-\mu)/\sigma$ and then using the Gaussian integral $\int_{-\infty}^{\infty} e^{-z^2/2} dz = \sqrt{2\pi}$. By the central limit theorem it appears universally as the limit of sums of many small independent effects. For details see the central limit theorem (Lindeberg–Lévy)（Theorem 5.6）［The Law of Large Numbers and the Central Limit Theorem］ in The law of large numbers and the central limit theorem.

4. Defining the expectation as a Lebesgue integral

4.1. The four stages of the construction

The expectation $E[X] = \int_\Omega X \, dP$ is defined in the following four stages. At each stage one must check that the definition extends the previous one consistently.

The four stages in the construction of the expectation (Lebesgue integral)

Definition 4.1（Expectation）

Let $(\Omega, \mathcal{F}, P)$ be a probability space.

(Stages 1 and 2.) If $A_1, \ldots, A_n \in \mathcal{F}$ form a partition of $\Omega$ and $a_1, \ldots, a_n \in [0,\infty)$, a random variable of the form $S = \sum_{i=1}^n a_i \mathbf{1}_{A_i}$ is called a nonnegative simple function, and its expectation is defined by

$$E[S] := \sum_{i=1}^{n} a_i P(A_i).$$

(Stage 3.) For measurable $X : \Omega \to [0, \infty]$ set

$$E[X] := \sup\bigl\{\, E[S] \ :\ S \text{ is a nonnegative simple function},\ 0 \le S \le X \,\bigr\} \in [0, \infty].$$

(Stage 4.) For a general random variable $X$ put $X^{+} := \max(X, 0)$ and $X^{-} := \max(-X, 0)$, so that $X = X^+ - X^-$ and $|X| = X^+ + X^-$. If $E[|X|] < \infty$ we call $X$ integrable and define

$$E[X] := E[X^{+}] - E[X^{-}].$$

The set of all integrable random variables is written $L^1(\Omega, \mathcal{F}, P)$.

The definition in stages 1 and 2 requires a check: since the same simple function admits several representations, we must show that the expectation does not depend on the representation chosen.

Lemma 4.2（The expectation of a simple function is well defined）

If a nonnegative simple function $S$ has two representations $S = \sum_{i=1}^{m} a_i \mathbf{1}_{A_i} = \sum_{j=1}^{n} b_j \mathbf{1}_{B_j}$, where $\{A_i\}$ and $\{B_j\}$ are both partitions of $\Omega$ into members of $\mathcal{F}$ and $a_i, b_j \ge 0$, then

$$\sum_{i=1}^{m} a_i P(A_i) = \sum_{j=1}^{n} b_j P(B_j).$$

Proof（Lemma 4.2）

Consider the common refinement $C_{ij} := A_i \cap B_j$. The family $\{C_{ij}\}_{i,j}$ is a partition of $\Omega$ with $A_i = \bigsqcup_j C_{ij}$ and $B_j = \bigsqcup_i C_{ij}$, so finite additivity of $P$ gives

$$P(A_i) = \sum_{j=1}^{n} P(C_{ij}), \qquad P(B_j) = \sum_{i=1}^{m} P(C_{ij}).$$

If $C_{ij} \ne \emptyset$, then for $\omega \in C_{ij}$ the value $S(\omega)$ equals $a_i$ in the first representation and $b_j$ in the second, so $a_i = b_j$. Therefore

$$\sum_{i} a_i P(A_i) = \sum_{i}\sum_{j} a_i P(C_{ij}) = \sum_{i}\sum_{j} b_j P(C_{ij}) = \sum_{j} b_j P(B_j)$$

(the terms with $C_{ij} = \emptyset$ have $P(C_{ij}) = 0$ and hence do no harm even when $a_i \ne b_j$).

∎

Remark 4.3

One must also check that the definition at stage 3 is consistent with stage 2, that is, that the two agree when $X$ is itself a nonnegative simple function. Since $X$ itself is among the simple functions $S \le X$, we have $\sup \ge E[X]$; conversely, for any simple $S$ with $0 \le S \le X$, passing to a common refinement shows $E[S] \le E[X]$ by finite additivity. Hence $\sup = E[X]$.

4.2. The standard approximating sequence

The definition by a supremum at stage 3 is not usable for computation as it stands. What one actually uses is the following construction, which approximates an arbitrary nonnegative measurable function from below by an increasing sequence of simple functions. (The same statement for a general measure space is the simple approximation theorem（Theorem 3.7）［ルベーグ積分の定義と収束定理］.)

Proposition 4.4（Standard approximating sequence）

Let $X : \Omega \to [0,\infty]$ be measurable and for $n \in \mathbb{N}$ set

$$X_n := \sum_{k=0}^{n2^n - 1} \frac{k}{2^n}\, \mathbf{1}_{\left\{\frac{k}{2^n} \le X < \frac{k+1}{2^n}\right\}} \;+\; n\, \mathbf{1}_{\{X \ge n\}} .$$

Then each $X_n$ is a nonnegative simple function, and for every $\omega$ we have $X_n(\omega) \le X_{n+1}(\omega)$ and $X_n(\omega) \to X(\omega)$. Moreover $E[X_n] \to E[X]$.

Proof（Proposition 4.4）

They are simple functions. Since $X$ is measurable, each of the sets $\{k2^{-n} \le X < (k+1)2^{-n}\} = X^{-1}([k2^{-n}, (k+1)2^{-n}))$ and $\{X \ge n\}$ lies in $\mathcal{F}$ by Definition 3.1. They form a finite partition of $\Omega$, and the coefficients are nonnegative.

Monotonicity. Passing from stage $n$ to stage $n+1$ halves the width of the intervals from $2^{-n}$ to $2^{-(n+1)}$. If $X(\omega) \in [k2^{-n}, (k+1)2^{-n})$, then the value at stage $n+1$ is either $2k \cdot 2^{-(n+1)} = k2^{-n}$ or $(2k+1)2^{-(n+1)} > k2^{-n}$, so $X_{n+1}(\omega) \ge X_n(\omega)$. On the part where $X(\omega) \ge n$, the value at stage $n+1$ is at least $n$.

Pointwise convergence. If $X(\omega) < \infty$, then for all $n$ beyond some $n > X(\omega)$ we have $0 \le X(\omega) - X_n(\omega) \le 2^{-n} \to 0$. If $X(\omega) = \infty$, then $X_n(\omega) = n \to \infty = X(\omega)$.

Convergence of the integrals. The sequence $(X_n)$ consists of nonnegative measurable functions increasing pointwise to $X$, so Theorem 2.1 gives $E[X_n] \to E[X]$.

∎

The standard approximating sequence slices the range into steps of size 2⁻ⁿ and makes the function constant on the preimages. Where Riemann integration partitions the domain, Lebesgue integration partitions the range.

4.3. The transfer formula

The expectation was defined as an integral over $\Omega$, but $\Omega$ is an abstract set and nothing can be computed there directly. The theorem that makes every actual computation possible is the following. In statistics texts it is sometimes called the law of the unconscious statistician.

Theorem 4.5（Transfer formula）

Let $X$ be a random variable on a probability space $(\Omega,\mathcal{F},P)$ and let $g : \mathbb{R} \to \mathbb{R}$ be Borel measurable.

1. If $g \ge 0$, then as an identity in $[0,\infty]$, $$E[g(X)] = \int_{\mathbb{R}} g(x) \, P_X(dx).$$
2. For general $g$, integrability of $g(X)$ with respect to $P$ is equivalent to integrability of $g$ with respect to $P_X$, and in that case the identity above holds.

Proof（Theorem 4.5）

We argue by the standard machine (indicator functions, then simple functions, then nonnegative functions, then the general case). Note first that $g(X) = g \circ X$ is a random variable: for any Borel set $B$ we have $(g\circ X)^{-1}(B) = X^{-1}(g^{-1}(B))$, and measurability of $g$ gives $g^{-1}(B) \in \mathcal{B}(\mathbb{R})$ while measurability of $X$ gives $X^{-1}(g^{-1}(B)) \in \mathcal{F}$.

Stage 1 ($g = \mathbf{1}_B$). Let $B \in \mathcal{B}(\mathbb{R})$. We have $\mathbf{1}_B(X(\omega)) = 1$ exactly when $X(\omega) \in B$, that is, when $\omega \in X^{-1}(B)$; hence $\mathbf{1}_B \circ X = \mathbf{1}_{X^{-1}(B)}$. So stage 1 of Definition 4.1 gives

$$E[\mathbf{1}_B(X)] = E[\mathbf{1}_{X^{-1}(B)}] = P(X^{-1}(B)) = P_X(B) = \int_{\mathbb{R}} \mathbf{1}_B \, dP_X .$$

The third equality is Definition 3.5, and the fourth is stage 1 of the definition of the integral with respect to $P_X$.

Stage 2 (nonnegative simple functions). Let $g = \sum_{i=1}^{n} a_i \mathbf{1}_{B_i}$ with $a_i \ge 0$ and the $B_i$ Borel sets partitioning $\mathbb{R}$. Then $g(X) = \sum_i a_i \mathbf{1}_{X^{-1}(B_i)}$, and $\{X^{-1}(B_i)\}$ is a partition of $\Omega$. Hence

$$E[g(X)] = \sum_{i=1}^n a_i P(X^{-1}(B_i)) = \sum_{i=1}^n a_i P_X(B_i) = \int_{\mathbb{R}} g \, dP_X .$$

Here the outer equalities are independent of the chosen representation by Lemma 4.2.

Stage 3 ($g \ge 0$). Apply the standard approximating sequence of Proposition 4.4 to $g$, obtaining an increasing sequence of nonnegative simple functions $g_n \uparrow g$. Composing, $g_n(X(\omega)) \uparrow g(X(\omega))$ for every $\omega$, so $g_n \circ X$ is an increasing sequence of nonnegative simple functions as well. Applying Theorem 2.1 on both sides (with measure $P$ on the left and $P_X$ on the right),

$$E[g(X)] = \lim_{n\to\infty} E[g_n(X)] = \lim_{n\to\infty} \int_{\mathbb{R}} g_n \, dP_X = \int_{\mathbb{R}} g \, dP_X ,$$

where the middle equality is the result of stage 2.

Stage 4 (general case). Decompose $g = g^+ - g^-$. Since $(g(X))^{+} = g^{+}(X)$ and $(g(X))^{-} = g^{-}(X)$, applying stage 3 to $|g| = g^+ + g^-$ gives

$$E[|g(X)|] = \int_{\mathbb{R}} |g| \, dP_X ,$$

so one side is finite exactly when the other is; this is the asserted equivalence of integrability. When $g$ is integrable, apply stage 3 separately to $g^{+}$ and $g^{-}$ and subtract; stage 4 of Definition 4.1 then yields the stated identity.

∎

Corollary 4.6（Computational formulas in the discrete and continuous cases）

Let $g : \mathbb{R} \to \mathbb{R}$ be Borel measurable.

1. If $X$ is discrete with $P(X = x_k) = p_k$ for $k \in \mathbb{N}$ and $\sum_k p_k = 1$, then whenever $\sum_k |g(x_k)| p_k < \infty$ the variable $g(X)$ is integrable and $$E[g(X)] = \sum_{k} g(x_k)\, p_k .$$
2. If $X$ is continuous with density $f$, then whenever $\int_{\mathbb{R}} |g(x)| f(x) \, dx < \infty$ the variable $g(X)$ is integrable and $$E[g(X)] = \int_{\mathbb{R}} g(x) f(x) \, dx .$$

Proof（Corollary 4.6）

By Theorem 4.5, both reduce to computing $\int_{\mathbb{R}} g \, dP_X$.

1. Put $S := \{x_1, x_2, \ldots\}$, so that $P_X(S) = 1$ and $P_X(\{x_k\}) = p_k$. Suppose first that $g \ge 0$ and set $g_n := g \mathbf{1}_{\{x_1,\ldots,x_n\}}$; then $g_n$ is simple and $g_n \uparrow g\mathbf{1}_S$. By the definition at stage 2, $\int g_n \, dP_X = \sum_{k=1}^n g(x_k) p_k$, and letting $n \to \infty$ with Theorem 2.1 gives $\int g \mathbf{1}_S dP_X = \sum_k g(x_k)p_k$. Since $P_X(S^c) = 0$, we have $\int g\, dP_X = \int g\mathbf{1}_S \, dP_X$. For general $g$ split into $g^{+} - g^{-}$, run the same argument, and subtract under the hypothesis of absolute convergence.

2. The claim has the form “if $dP_X = f\,dx$ then $\int g \, dP_X = \int g f \, dx$”, and this too is proved by the standard machine: for $g = \mathbf{1}_B$ it is exactly the definition of a density in Definition 3.8; simple functions follow by linearity; nonnegative measurable functions follow from Theorem 2.1 (if $g_n \uparrow g$ then $g_n f \uparrow g f$); and the general case follows by splitting into positive and negative parts.

∎

4.4. Worked computations

Example 4.7（Means and second moments of standard distributions）

Poisson distribution $X \sim \mathrm{Po}(\lambda)$: by part 1 of Corollary 4.6,

$$E[X] = \sum_{k=0}^{\infty} k \, e^{-\lambda}\frac{\lambda^k}{k!} = \lambda e^{-\lambda} \sum_{k=1}^{\infty} \frac{\lambda^{k-1}}{(k-1)!} = \lambda e^{-\lambda} e^{\lambda} = \lambda .$$

Here we used that the $k=0$ term vanishes and that $k/k! = 1/(k-1)!$. Similarly, for $g(x) = x(x-1)$,

$$E[X(X-1)] = \sum_{k=2}^{\infty} k(k-1) e^{-\lambda}\frac{\lambda^k}{k!} = \lambda^2 e^{-\lambda}\sum_{k=2}^{\infty}\frac{\lambda^{k-2}}{(k-2)!} = \lambda^2 .$$

Hence $E[X^2] = E[X(X-1)] + E[X] = \lambda^2 + \lambda$.

Exponential distribution $X \sim \mathrm{Exp}(\lambda)$: by part 2 of Corollary 4.6, integrating by parts,

$$E[X] = \int_0^\infty x \lambda e^{-\lambda x} dx = \Bigl[-x e^{-\lambda x}\Bigr]_0^\infty + \int_0^\infty e^{-\lambda x} dx = 0 + \frac{1}{\lambda} = \frac{1}{\lambda},$$$$E[X^2] = \int_0^\infty x^2 \lambda e^{-\lambda x} dx = \Bigl[-x^2 e^{-\lambda x}\Bigr]_0^\infty + \int_0^\infty 2x e^{-\lambda x} dx = \frac{2}{\lambda}\cdot\frac{1}{\lambda} = \frac{2}{\lambda^2}.$$

In the last equality we used the value $\int_0^\infty x\lambda e^{-\lambda x}dx = 1/\lambda$ just computed.

Normal distribution $X \sim N(\mu, \sigma^2)$: substituting $z = (x-\mu)/\sigma$,

$$E[X] = \int_{\mathbb{R}} \frac{x}{\sqrt{2\pi}\sigma} e^{-(x-\mu)^2/(2\sigma^2)} dx = \int_{\mathbb{R}} (\mu + \sigma z) \frac{1}{\sqrt{2\pi}} e^{-z^2/2} dz = \mu$$

(the function $z e^{-z^2/2}$ is odd and absolutely integrable, so its integral is $0$). Next, $E[(X-\mu)^2] = \sigma^2 \int_{\mathbb{R}} z^2 \frac{1}{\sqrt{2\pi}}e^{-z^2/2}dz$, and integrating by parts with $u = z$ and $dv = z e^{-z^2/2}dz$ (so $v = -e^{-z^2/2}$),

$$\int_{\mathbb{R}} z^2 \frac{e^{-z^2/2}}{\sqrt{2\pi}} dz = \frac{1}{\sqrt{2\pi}}\Bigl(\Bigl[-z e^{-z^2/2}\Bigr]_{-\infty}^{\infty} + \int_{\mathbb{R}} e^{-z^2/2} dz\Bigr) = 0 + 1 = 1 ,$$

so $E[(X-\mu)^2] = \sigma^2$.

Example 4.8（A random variable with no expectation (the Cauchy distribution)）

Consider a random variable $X$ with density $f(x) = \dfrac{1}{\pi(1+x^2)}$. This really is a density, since $\int_{\mathbb{R}} f = \frac{1}{\pi}[\arctan x]_{-\infty}^{\infty} = \frac{1}{\pi}\bigl(\frac{\pi}{2} + \frac{\pi}{2}\bigr) = 1$.

The density is symmetric about the origin, so one is tempted to say that the mean “ought to be $0$”; but the integrability condition at stage 4 of Definition 4.1 fails. Indeed

$$E[|X|] = \frac{2}{\pi}\int_0^{\infty} \frac{x}{1+x^2} dx = \frac{1}{\pi}\Bigl[\log(1+x^2)\Bigr]_0^{\infty} = \infty .$$

Thus $E[X^{+}] = E[X^{-}] = \infty$, and $E[X] = \infty - \infty$ cannot be defined. The correct statement is that $E[X]$ does not exist.

This is not a pathological example. The Cauchy distribution arises naturally as the distribution of a ratio $Z_1/Z_2$ of independent standard normal variables, and moreover the sample mean $\bar{X}_n$ of independent Cauchy samples follows the same Cauchy distribution for every $n$. In other words, increasing the sample size does not improve the precision. Here one sees what it means that the law of large numbers assumes the existence of the expectation.

5. Variance, covariance and correlation

5.1. Definitions and basic properties

The expectation is a single number describing the “location” of a distribution. The standard quantity measuring its “spread” is the variance.

Definition 5.1（Variance, covariance and correlation coefficient）

Write $L^2 = L^2(\Omega,\mathcal{F},P)$ for the set of random variables with $E[X^2] < \infty$. For $X, Y \in L^2$, put $\mu_X := E[X]$ and $\mu_Y := E[Y]$ and define

$$\mathrm{Var}(X) := E\bigl[(X - \mu_X)^2\bigr], \qquad \mathrm{Cov}(X,Y) := E\bigl[(X - \mu_X)(Y - \mu_Y)\bigr] ,$$

called the variance of $X$ and the covariance of $X$ and $Y$. The quantity $\sigma_X := \sqrt{\mathrm{Var}(X)}$ is the standard deviation. If moreover $\sigma_X > 0$ and $\sigma_Y > 0$, then

$$\rho(X,Y) := \frac{\mathrm{Cov}(X,Y)}{\sigma_X \sigma_Y}$$

is the correlation coefficient. When $\mathrm{Cov}(X,Y) = 0$ we say $X$ and $Y$ are uncorrelated.

For this definition to make sense, $X$ and $(X-\mu_X)(Y-\mu_Y)$ must be integrable whenever $X, Y \in L^2$.

Proposition 5.2（On a probability space, L² ⊂ L¹）

On a probability space we have $L^2 \subset L^1$, and if $X, Y \in L^2$ then $XY \in L^1$.

Proof（Proposition 5.2）

Expanding $(|a| - |b|)^2 \ge 0$ for real $a, b$ gives $|ab| \le \frac{1}{2}(a^2 + b^2)$. Taking $b = 1$ yields $|a| \le \frac{1}{2}(1 + a^2)$, so

$$E[|X|] \le \tfrac{1}{2}\bigl(1 + E[X^2]\bigr) < \infty$$

(we used $E[1] = P(\Omega) = 1$; that this $1$ is finite is exactly what a probability measure buys us). Taking $a = X(\omega)$ and $b = Y(\omega)$ in the same inequality gives $|XY| \le \frac{1}{2}(X^2 + Y^2)$, whose right-hand side is integrable, so $XY \in L^1$. Consequently $(X-\mu_X)(Y-\mu_Y) = XY - \mu_X Y - \mu_Y X + \mu_X\mu_Y$ is integrable as well.

∎

Proposition 5.3（Basic properties of variance and covariance）

Let $X, Y, X_1, \ldots, X_n \in L^2$ and $a, b \in \mathbb{R}$.

1. $\mathrm{Var}(X) = E[X^2] - (E[X])^2$ and $\mathrm{Cov}(X,Y) = E[XY] - E[X]E[Y]$.
2. $\mathrm{Var}(aX + b) = a^2 \mathrm{Var}(X)$. In particular the variance is invariant under translation by a constant.
3. $\mathrm{Cov}$ is a symmetric bilinear form, and $\mathrm{Cov}(X,X) = \mathrm{Var}(X)$.
4. $\mathrm{Var}\Bigl(\sum_{i=1}^{n} X_i\Bigr) = \sum_{i=1}^{n}\mathrm{Var}(X_i) + 2\sum_{1 \le i < j \le n} \mathrm{Cov}(X_i, X_j)$.
5. $\mathrm{Var}(X) = 0$ holds if and only if $P(X = E[X]) = 1$.

Proof（Proposition 5.3）

1. By linearity of the expectation (a basic property following from Definition 4.1),

$$E[(X-\mu_X)(Y-\mu_Y)] = E[XY] - \mu_X E[Y] - \mu_Y E[X] + \mu_X\mu_Y = E[XY] - \mu_X\mu_Y .$$

Setting $Y = X$ gives the formula for the variance. By Proposition 5.2 every term is finite, so the expansion is legitimate.

2. Since $E[aX+b] = a\mu_X + b$, we get $(aX+b) - E[aX+b] = a(X - \mu_X)$, whence $\mathrm{Var}(aX+b) = E[a^2(X-\mu_X)^2] = a^2\mathrm{Var}(X)$.

3. Symmetry holds because the defining expression is symmetric in $X$ and $Y$. Bilinearity follows by expanding $\mathrm{Cov}(aX_1 + X_2, Y) = E[(a(X_1-\mu_1) + (X_2-\mu_2))(Y-\mu_Y)]$ using linearity of the expectation.

4. Put $S := \sum_i X_i$. Then $S - E[S] = \sum_i (X_i - \mu_i)$, so by the bilinearity of part 3,

$$\mathrm{Var}(S) = \mathrm{Cov}\Bigl(\sum_i (X_i-\mu_i), \sum_j (X_j-\mu_j)\Bigr) = \sum_{i}\sum_{j} \mathrm{Cov}(X_i, X_j) .$$

The diagonal terms $i = j$ give $\mathrm{Var}(X_i)$, and by symmetry the off-diagonal terms amount to twice the terms with $i < j$.

5. Put $Z := (X - \mu_X)^2 \ge 0$. If $E[Z] = 0$, then for every $\varepsilon > 0$ the simple function $\varepsilon \mathbf{1}_{\{Z \ge \varepsilon\}} \le Z$ gives $\varepsilon P(Z \ge \varepsilon) \le E[Z] = 0$, that is, $P(Z \ge \varepsilon) = 0$. Since $\{Z > 0\} = \bigcup_{n} \{Z \ge 1/n\}$ is a countable union, $P(Z > 0) = 0$, i.e. $P(X = \mu_X) = 1$. Conversely, if $P(X = \mu_X) = 1$ then $Z = 0$ with probability $1$ and hence $E[Z] = 0$.

∎

Example 5.4（Computing variances, and additivity）

From the results of Example 4.7 and part 1 of Proposition 5.3 we immediately obtain the following.

• $X \sim \mathrm{Po}(\lambda)$: $\mathrm{Var}(X) = (\lambda^2 + \lambda) - \lambda^2 = \lambda$. That the mean and the variance coincide is characteristic of the Poisson distribution.
• $X \sim \mathrm{Exp}(\lambda)$: $\mathrm{Var}(X) = 2/\lambda^2 - 1/\lambda^2 = 1/\lambda^2$. The standard deviation equals the mean $1/\lambda$.
• $X \sim N(\mu,\sigma^2)$: $\mathrm{Var}(X) = \sigma^2$. The parameter is the variance itself.

For the binomial distribution it is quicker to decompose into an independent sum than to compute directly. Let $Y_1, \ldots, Y_n$ be independent with distribution $\mathrm{Ber}(p)$, so that $X := \sum_i Y_i \sim \mathrm{Bin}(n,p)$. Since $Y_i^2 = Y_i$,

$$\mathrm{Var}(Y_i) = E[Y_i^2] - (E[Y_i])^2 = p - p^2 = p(1-p) .$$

By independence and Proposition 7.5 (Appendix) we have $\mathrm{Cov}(Y_i,Y_j) = 0$ for $i \ne j$, so part 4 of Proposition 5.3 gives

$$E[X] = np, \qquad \mathrm{Var}(X) = n p(1-p) .$$

No sum involving binomial coefficients has to be evaluated directly.

5.2. Uncorrelated is not the same as independent

Example 5.5（Uncorrelated but not independent）

Let $X$ be uniformly distributed on the interval $(-1,1)$, with density $f(x) = \frac{1}{2}\mathbf{1}_{(-1,1)}(x)$, and put $Y := X^2$. Since $Y$ is completely determined by $X$, intuitively the dependence is as strong as possible. Nevertheless

$$E[X] = \int_{-1}^{1} \frac{x}{2} dx = 0, \qquad E[XY] = E[X^3] = \int_{-1}^{1}\frac{x^3}{2}dx = 0$$

(the integrands are odd), so part 1 of Proposition 5.3 gives $\mathrm{Cov}(X,Y) = 0 - 0\cdot E[Y] = 0$. They are uncorrelated.

Let us verify from the definition that they are not independent. Take $B := (-\tfrac{1}{2}, \tfrac{1}{2})$ and $C := [0, \tfrac{1}{9}]$. Since $Y \in C$ is equivalent to $|X| \le 1/3$,

$$P(X \in B,\ Y \in C) = P(|X| \le \tfrac{1}{3}) = \tfrac{1}{3}, \qquad P(X \in B)\,P(Y \in C) = \tfrac{1}{2}\cdot\tfrac{1}{3} = \tfrac{1}{6} ,$$

and the two do not agree. Hence they are not independent in the sense of Definition 2.3.

The lesson is clear. The covariance detects only linear relationships; it is blind to even-function dependences such as $Y = X^2$. Reading “uncorrelated” as “unrelated” is a mistake.

5.3. The Cauchy–Schwarz inequality and the meaning of the correlation coefficient

Theorem 5.6（Cauchy–Schwarz inequality for the covariance）

Let $X, Y \in L^2$. Then

$$\bigl|\mathrm{Cov}(X,Y)\bigr| \le \sigma_X \, \sigma_Y .$$

Moreover, when $\sigma_X > 0$ and $\sigma_Y > 0$, equality holds if and only if there exist real numbers $b \ne 0$ and $a$ with $P(Y = a + bX) = 1$.

Proof（Theorem 5.6）

Put $U := X - \mu_X$ and $V := Y - \mu_Y$, so that $E[U] = E[V] = 0$, $E[U^2] = \sigma_X^2$, $E[V^2] = \sigma_Y^2$ and $E[UV] = \mathrm{Cov}(X,Y)$. All of these are finite by Proposition 5.2.

Case $\sigma_X = 0$: by part 5 of Proposition 5.3 we have $P(U = 0) = 1$, so $UV = 0$ with probability $1$ and $\mathrm{Cov}(X,Y) = 0$; the inequality holds with equality.

Now suppose $\sigma_X > 0$. For every $t \in \mathbb{R}$ we have $(V + tU)^2 \ge 0$, so by monotonicity of the expectation

$$\varphi(t) := E\bigl[(V + tU)^2\bigr] = \sigma_Y^2 + 2t\,\mathrm{Cov}(X,Y) + t^2 \sigma_X^2 \ge 0$$

(the expansion uses linearity of the expectation). This is a quadratic in $t$ with leading coefficient $\sigma_X^2 > 0$. Nonnegativity for all $t$ is equivalent to the discriminant being at most $0$, so

$$4\,\mathrm{Cov}(X,Y)^2 - 4\sigma_X^2\sigma_Y^2 \le 0,$$

that is, $|\mathrm{Cov}(X,Y)| \le \sigma_X\sigma_Y$.

Equality. Equality holds exactly when the discriminant vanishes, i.e. when there is a (double) root $t_0$ with $\varphi(t_0) = 0$. From $\varphi(t_0) = E[(V + t_0 U)^2] = 0$ and the argument inside the proof of part 5 of Proposition 5.3 (a nonnegative random variable with expectation $0$ vanishes with probability $1$), we get $P(V + t_0 U = 0) = 1$. Rewriting,

$$P\bigl(Y = (\mu_Y + t_0\mu_X) - t_0 X\bigr) = 1 ,$$

so it suffices to set $b := -t_0$ and $a := \mu_Y + t_0\mu_X$. We do need $b \ne 0$, i.e. $t_0 \ne 0$: if $t_0 = 0$ then $\varphi(0) = \sigma_Y^2 = 0$, contradicting the assumption $\sigma_Y > 0$, so indeed $t_0 \ne 0$. (In the degenerate case $\sigma_Y = 0$ the variable $Y$ is constant, and although equality does hold, $\mathrm{Cov}(X,Y) = 0 = \sigma_X\sigma_Y$, one has $b = 0$.) Conversely, if $P(Y = a + bX) = 1$ with $b \ne 0$, then parts 2 and 3 of Proposition 5.3 give $\mathrm{Cov}(X,Y) = b\,\sigma_X^2$ and $\sigma_Y = |b|\sigma_X$, so $|\mathrm{Cov}(X,Y)| = |b|\sigma_X^2 = \sigma_X\sigma_Y$ and equality holds.

∎

Corollary 5.7（Range of the correlation coefficient）

If $\sigma_X > 0$ and $\sigma_Y > 0$ then $-1 \le \rho(X,Y) \le 1$. Moreover $\rho(X,Y) = \pm 1$ holds if and only if there exist $a, b$ with $P(Y = a + bX) = 1$, the sign of $b$ agreeing with the sign of $\rho$.

Proof（Corollary 5.7）

Dividing both sides in Theorem 5.6 by $\sigma_X\sigma_Y > 0$ gives $|\rho| \le 1$, and the equality condition follows from the same theorem. As for the sign, when $P(Y = a+bX) = 1$ we have $\rho = b\sigma_X^2/(\sigma_X \cdot |b|\sigma_X) = b/|b| = \mathrm{sgn}(b)$.

∎

The statement that the correlation coefficient measures “the strength of a linear relationship” acquires a precise meaning through the following computation.

Example 5.8（The correlation coefficient determines the error of the best linear prediction）

Suppose we predict $Y$ from $X$ by an expression of the form $a + bX$; we find the $a, b$ minimising the mean squared error, and the minimum value. Assume $\sigma_X > 0$ and $\sigma_Y > 0$.

First fix $b$ and minimise over $a$. Put $h(a,b) := E[(Y - a - bX)^2]$. With $W := Y - bX$ we have $h = E[(W - a)^2] = \mathrm{Var}(W) + (E[W] - a)^2$ (apply part 1 of Proposition 5.3 to $W - a$). So $a^{*} = E[W] = \mu_Y - b\mu_X$ is optimal, and the resulting value is

$$\mathrm{Var}(Y - bX) = \sigma_Y^2 - 2b\,\mathrm{Cov}(X,Y) + b^2\sigma_X^2$$

(by parts 4, 2 and 3 of Proposition 5.3). This is a quadratic in $b$; completing the square gives

$$\sigma_X^2\left(b - \frac{\mathrm{Cov}(X,Y)}{\sigma_X^2}\right)^2 + \sigma_Y^2 - \frac{\mathrm{Cov}(X,Y)^2}{\sigma_X^2} ,$$

so the minimum is attained at

$$b^{*} = \frac{\mathrm{Cov}(X,Y)}{\sigma_X^2} = \rho\,\frac{\sigma_Y}{\sigma_X}, \qquad \min_{a,b} E[(Y - a - bX)^2] = \sigma_Y^2\bigl(1 - \rho^2\bigr) .$$

Thus $\rho^2$ is “the fraction of the variance of $Y$ explained by linear prediction from $X$”. If $\rho = \pm 1$ the error is $0$, that is, $Y$ is an affine function of $X$, exactly as in Corollary 5.7. If $\rho = 0$, linear prediction cannot beat the constant $\mu_Y$. The coefficient of determination $R^2$ in regression analysis is a generalisation of this $\rho^2$. Handling more complicated dependences requires dropping the linearity constraint, which leads to the conditional expectation (see Conditional expectation, and in particular the conditional expectation as an orthogonal projection（Theorem 5.1）［条件付き期待値］).

6. The Markov and Chebyshev inequalities

Even with no knowledge whatever of the shape of a distribution, tail probabilities can be estimated from the mean and the variance alone. This is the content of the next two inequalities. Both proofs take two lines, yet they are among the most heavily used tools in probability.

Theorem 6.1（Markov's inequality）

Let $X$ be a nonnegative random variable, i.e. $P(X \ge 0) = 1$. Then for every $a > 0$,

$$P(X \ge a) \le \frac{E[X]}{a} .$$

Proof（Theorem 6.1）

Put $A := \{X \ge a\} \in \mathcal{F}$ (measurability of $X$). For every $\omega$,

$$a\,\mathbf{1}_A(\omega) \le X(\omega) .$$

Indeed, if $\omega \in A$ the left-hand side is $a$ and by definition of $A$ we have $X(\omega) \ge a$; if $\omega \notin A$ the left-hand side is $0$ and the right-hand side is nonnegative since $X \ge 0$.

The expectation is monotone ($U \le V$ implies $E[U] \le E[V]$, which follows at once from the supremum in stage 3 of Definition 4.1), so taking expectations of both sides,

$$a\,P(A) = E[a\mathbf{1}_A] \le E[X] .$$

Dividing by $a > 0$ gives the claim.

∎

Corollary 6.2（Chebyshev's inequality）

Let $X \in L^2$, $\mu := E[X]$ and $\sigma^2 := \mathrm{Var}(X)$. Then for every $a > 0$,

$$P\bigl(|X - \mu| \ge a\bigr) \le \frac{\sigma^2}{a^2} .$$

In particular, when $\sigma > 0$, taking $a = k\sigma$ with $k > 0$ gives

$$P\bigl(|X - \mu| \ge k\sigma\bigr) \le \frac{1}{k^2}.$$

Proof（Corollary 6.2）

Put $Y := (X-\mu)^2 \ge 0$; by Proposition 5.2 we have $E[Y] = \sigma^2 < \infty$. The events satisfy

$$\{|X - \mu| \ge a\} = \{(X-\mu)^2 \ge a^2\} = \{Y \ge a^2\}$$

(since $t \mapsto t^2$ is strictly increasing on $[0,\infty)$, the conditions $|X-\mu| \ge a > 0$ and $(X-\mu)^2 \ge a^2$ are equivalent). Applying Theorem 6.1 to $Y$ and $a^2 > 0$,

$$P(|X-\mu| \ge a) = P(Y \ge a^2) \le \frac{E[Y]}{a^2} = \frac{\sigma^2}{a^2} .$$

Taking $a = k\sigma$ gives $\sigma^2/(k\sigma)^2 = 1/k^2$.

∎

Example 6.3（How loose is the Chebyshev bound?）

For $k = 2$, Corollary 6.2 gives $P(|X-\mu| \ge 2\sigma) \le 0.25$. Let us compare with the true values for concrete distributions.

• $X \sim N(\mu,\sigma^2)$: $P(|X-\mu| \ge 2\sigma) = 2\bigl(1 - \Phi(2)\bigr) \approx 0.0455$.
• $X \sim \mathrm{Exp}(1)$ (so $\mu = \sigma = 1$): $P(|X - 1| \ge 2) = P(X \ge 3) = e^{-3} \approx 0.0498$.

Both are about one fifth of the bound $0.25$. If the distribution is known, far sharper estimates are available.

Chebyshev’s inequality is nonetheless important because it assumes nothing about the distribution. It applies where no distributional form can be posited: unknown populations, the outputs of complicated machine-learning models. Moreover the bound $1/k^2$ cannot be improved if one demands validity across all distributions, because there are distributions attaining equality (Exercise 7.3).

Example 6.4（Application to the weak law of large numbers）

Let $X_1, X_2, \ldots$ be independent and identically distributed with $E[X_i] = \mu$ and $\mathrm{Var}(X_i) = \sigma^2 < \infty$, and put $\bar{X}_n := \frac{1}{n}\sum_{i=1}^n X_i$. Linearity of the expectation gives $E[\bar{X}_n] = \mu$. Using parts 2 and 4 of Proposition 5.3 together with the vanishing of the covariance terms under independence (Proposition 7.5),

$$\mathrm{Var}(\bar{X}_n) = \frac{1}{n^2}\sum_{i=1}^{n}\mathrm{Var}(X_i) = \frac{\sigma^2}{n} .$$

Applying Corollary 6.2 to $\bar{X}_n$, for every $\varepsilon > 0$,

$$P\bigl(|\bar{X}_n - \mu| \ge \varepsilon\bigr) \le \frac{\sigma^2}{n\varepsilon^2} \xrightarrow[n\to\infty]{} 0 .$$

This is the weak law of large numbers (the law of large numbers in the sense of convergence in probability). Versions dispensing with the variance assumption, and the strong law asserting almost sure convergence (the strong law of large numbers (Kolmogorov)（Theorem 4.5）［The Law of Large Numbers and the Central Limit Theorem］), are treated in The law of large numbers and the central limit theorem.

This estimate can be used directly to design a sample size. With $\sigma = 1$ and $\varepsilon = 0.1$, to keep the error probability at most $5\%$ it suffices to take $n \ge 1/(0.05 \times 0.01) = 2000$. The central limit theorem shows that $n \approx 385$ is already enough, but that is an approximation valid for large $n$, whereas the Chebyshev estimate is an inequality that is exactly correct for every $n$.

7. Exercises

Exercise 7.1Standard

Suppose $X : \Omega \to \mathbb{R}$ satisfies $\{X < q\} \in \mathcal{F}$ for every rational $q$. Show that $X$ is a random variable.

Solution

Apply Proposition 3.2 with $\mathcal{E} := \{(-\infty, q) : q \in \mathbb{Q}\}$. We must show $\sigma(\mathcal{E}) = \mathcal{B}(\mathbb{R})$.

First, $\sigma(\mathcal{E}) \subset \mathcal{B}(\mathbb{R})$ because each $(-\infty,q)$ is open.

For the converse, let $a$ be any real number. By density of the rationals there is a rational sequence $q_n \uparrow a$, and

$$(-\infty, a) = \bigcup_{n=1}^{\infty} (-\infty, q_n)$$

(if $x < a$ then $x < q_n$ for all large $n$). Hence $(-\infty,a) \in \sigma(\mathcal{E})$.

Next, $(-\infty, a]$ also lies in $\sigma(\mathcal{E})$, since

$$(-\infty, a] = \bigcap_{n=1}^{\infty}\Bigl(-\infty,\ a + \tfrac{1}{n}\Bigr)$$

(if $x \le a$ then $x < a + 1/n$ for every $n$; conversely if $x < a+1/n$ for every $n$ then letting $n\to\infty$ gives $x \le a$), and each set on the right lies in $\sigma(\mathcal{E})$ by the previous step. Therefore, for any open interval,

$$(a,b) = (-\infty,b) \setminus (-\infty,a] \in \sigma(\mathcal{E}) .$$

Every open subset of $\mathbb{R}$ is a countable union of open intervals (take intervals with rational endpoints around each point), so all open sets lie in $\sigma(\mathcal{E})$, and hence $\mathcal{B}(\mathbb{R}) \subset \sigma(\mathcal{E})$.

This proves $\sigma(\mathcal{E}) = \mathcal{B}(\mathbb{R})$, and Proposition 3.2 shows that $X$ is a random variable. The point is that countably many conditions already imply measurability.

Exercise 7.2Standard

Let $0 < p < 1$ and $q := 1-p$, and let $X$ have the geometric distribution, that is,

$$P(X = k) = q^{k-1} p \qquad (k = 1, 2, 3, \ldots).$$

Compute $E[X]$ and $\mathrm{Var}(X)$.

Solution

First we check that this is a probability mass function: $\sum_{k\ge1} q^{k-1}p = p \cdot \frac{1}{1-q} = \frac{p}{p} = 1$, the geometric series converging because $0 < q < 1$.

For $|x| < 1$, differentiating $\sum_{k \ge 0} x^k = \frac{1}{1-x}$ termwise gives $\sum_{k\ge1} k x^{k-1} = \frac{1}{(1-x)^2}$, and differentiating once more gives $\sum_{k \ge 2} k(k-1)x^{k-2} = \frac{2}{(1-x)^3}$ (a power series may be differentiated termwise inside its radius of convergence).

By part 1 of Corollary 4.6,

$$E[X] = \sum_{k=1}^{\infty} k\, q^{k-1} p = p \cdot \frac{1}{(1-q)^2} = \frac{p}{p^2} = \frac{1}{p}.$$

Next, for $g(x) = x(x-1)$,

$$E[X(X-1)] = \sum_{k=2}^{\infty} k(k-1) q^{k-1} p = pq\sum_{k=2}^{\infty}k(k-1)q^{k-2} = pq \cdot \frac{2}{p^3} = \frac{2q}{p^2}.$$

Hence $E[X^2] = \frac{2q}{p^2} + \frac{1}{p}$, and part 1 of Proposition 5.3 gives

$$\mathrm{Var}(X) = \frac{2q}{p^2} + \frac{1}{p} - \frac{1}{p^2} = \frac{2q + p - 1}{p^2} = \frac{2q - q}{p^2} = \frac{q}{p^2} = \frac{1-p}{p^2},$$

where we used $p - 1 = -q$. For example, with $p = 1/6$ (the number of rolls until a die first shows a $1$) we get $E[X] = 6$ and $\mathrm{Var}(X) = 30$.

Exercise 7.3Standard

Fix $k > 1$ and $\sigma > 0$. Construct a random variable $X$ with $E[X] = 0$, $\mathrm{Var}(X) = \sigma^2$ and

$$P\bigl(|X| \ge k\sigma\bigr) = \frac{1}{k^2} ,$$

thereby showing that the bound $1/k^2$ in Corollary 6.2 cannot be improved.

Solution

Take the following three-point distribution.

$$P(X = k\sigma) = P(X = -k\sigma) = \frac{1}{2k^2}, \qquad P(X = 0) = 1 - \frac{1}{k^2}.$$

Since $k > 1$ we have $1 - 1/k^2 > 0$, and the probabilities sum to $\frac{1}{2k^2} + \frac{1}{2k^2} + 1 - \frac{1}{k^2} = 1$.

By part 1 of Corollary 4.6,

$$E[X] = k\sigma \cdot \frac{1}{2k^2} + (-k\sigma)\cdot\frac{1}{2k^2} + 0 = 0,$$$$E[X^2] = (k\sigma)^2 \cdot \frac{1}{2k^2} + (k\sigma)^2\cdot\frac{1}{2k^2} + 0 = k^2\sigma^2 \cdot \frac{1}{k^2} = \sigma^2 .$$

Hence part 1 of Proposition 5.3 gives $\mathrm{Var}(X) = \sigma^2 - 0^2 = \sigma^2$.

On the other hand $|X| \ge k\sigma$ holds exactly when $X = \pm k\sigma$, so

$$P(|X| \ge k\sigma) = \frac{1}{2k^2} + \frac{1}{2k^2} = \frac{1}{k^2} ,$$

and Chebyshev’s inequality holds with equality. Therefore, among bounds of the form $P(|X-\mu| \ge k\sigma) \le c/k^2$ valid for all $L^2$ random variables, one cannot take $c$ smaller than $1$. Note that the example above is symmetric about the origin, so imposing symmetry alone does not improve the bound; extra assumptions such as unimodality or boundedness are needed.

Exercise 7.4Hard

Let $X$ be a random variable with $P(X \ge 0) = 1$. Show that

$$E[X] = \int_0^{\infty} P(X > t)\, dt$$

holds as an identity in $[0,\infty]$, the right-hand side being a Lebesgue integral.

Solution

On the product space $\bigl(\Omega \times (0,\infty),\ \mathcal{F}\otimes\mathcal{B}((0,\infty)),\ P \otimes \mathrm{Leb}\bigr)$ consider the function

$$H(\omega, t) := \mathbf{1}_{\{X(\omega) > t\}} .$$

Measurability. The map $(\omega,t) \mapsto X(\omega)$ is the composition of the projection onto the first coordinate with $X$, hence measurable with respect to the product $\sigma$-algebra; the same holds for $(\omega,t)\mapsto t$. By the result on sums (differences) in Proposition 3.4, the map $(\omega,t)\mapsto X(\omega) - t$ is measurable, and

$$\{(\omega,t) : H = 1\} = \{(\omega,t) : X(\omega) - t > 0\}$$

is the preimage of $(0,\infty)$ under that measurable function, hence a measurable set. Thus $H$ is nonnegative and measurable.

Applying Tonelli’s theorem. Since $P$ is a probability measure and Lebesgue measure on $(0,\infty)$ is $\sigma$-finite, Tonelli’s theorem for nonnegative measurable functions (The Lebesgue integral and the convergence theorems) says that the two iterated integrals agree.

Integrating in $t$ first, with $\omega$ fixed,

$$\int_0^{\infty} \mathbf{1}_{\{X(\omega) > t\}}\, dt = \int_0^{X(\omega)} 1 \, dt = X(\omega)$$

(since $X(\omega) \ge 0$, the integrand in $t$ is the indicator of the interval $(0, X(\omega))$). So this iterated integral equals $E[X]$.

Integrating in $\omega$ first, with $t > 0$ fixed,

$$\int_{\Omega} \mathbf{1}_{\{X > t\}} \, dP = P(X > t),$$

so this iterated integral is $\int_0^\infty P(X>t)\,dt$.

By Tonelli’s theorem the two are equal in $[0,\infty]$, which proves the claim.

Check. For $X \sim \mathrm{Exp}(\lambda)$ we have $P(X > t) = e^{-\lambda t}$, so $\int_0^\infty e^{-\lambda t}dt = 1/\lambda$, agreeing with the result of Example 4.7. The formula is especially useful when the density is unknown and only the survival probability $P(X>t)$ is available, as in reliability engineering and survival analysis.

References

• A. N. Kolmogorov, Grundbegriffe der Wahrscheinlichkeitsrechnung, Springer, 1933 (English translation: Foundations of the Theory of Probability, Chelsea, 1956) — the original source defining random variables as measurable functions. Chapter III.
• Kiyosi Ito, Kakuritsuron (Probability Theory), Iwanami Shoten (Iwanami Foundations of Mathematics), 1991 (in Japanese) — Chapters 1 and 2. A standard Japanese reference on measure-theoretic probability.
• Naohisa Funaki, Kakuritsuron (Probability Theory), Asakura Shoten (Ways of Mathematical Thinking 20), 2004 (in Japanese) — Chapter 2 organises random variables, expectation and the various inequalities.
• P. Billingsley, Probability and Measure, 3rd ed., Wiley, 1995 — Sections 13, 15, 21. A careful treatment of the construction of the expectation and of the transfer formula.
• R. Durrett, Probability: Theory and Examples, 5th ed., Cambridge University Press, 2019 — Chapter 1. Concise, with a wealth of exercises.
• D. Williams, Probability with Martingales, Cambridge University Press, 1991 — Chapters 3–6. Makes the “standard machine” (from indicators to the general case) explicit as a method.

Appendix: independence and products of expectations

Here we prove the fact used in Example 5.4 and Example 6.4 in the main text, namely that independence implies uncorrelatedness.

Proposition 7.5（Expectation of a product of independent integrable random variables）

Let $X, Y$ be independent random variables, both integrable. Then $XY$ is integrable and

$$E[XY] = E[X]\,E[Y] .$$

In particular, if $X, Y \in L^2$ are independent then $\mathrm{Cov}(X,Y) = 0$.

Proof（Proposition 7.5）

Write $P_{(X,Y)}$ for the joint distribution of $(X,Y)$ on $\mathbb{R}^2$. Definition 2.3 says that

$$P_{(X,Y)}(B \times C) = P_X(B)\,P_Y(C)$$

for all Borel sets $B, C$. The rectangles $\{B\times C\}$ form a multiplicative family (closed under finite intersections) generating $\mathcal{B}(\mathbb{R}^2)$, and both sides are probability measures on $\mathbb{R}^2$, so Dynkin’s $\pi$-$\lambda$ theorem gives $P_{(X,Y)} = P_X \otimes P_Y$ (the product measure) everywhere.

Suppose first that $X, Y \ge 0$. Apply the two-dimensional version of Theorem 4.5 (with $g(x,y) = xy$, regarding $(X,Y)$ as an $\mathbb{R}^2$-valued measurable map; the proof is the same standard machine as in the main text), and then Tonelli’s theorem for the product measure $P_X\otimes P_Y$:

$$E[XY] = \int_{\mathbb{R}^2} xy \, P_{(X,Y)}(dx\,dy) = \int_{\mathbb{R}}\!\!\int_{\mathbb{R}} xy \, P_X(dx) P_Y(dy) = \Bigl(\int x\,P_X(dx)\Bigr)\Bigl(\int y \, P_Y(dy)\Bigr) ,$$

and by Theorem 4.5 again the right-hand side is $E[X]E[Y]$.

In the general case decompose $X = X^{+} - X^{-}$ and $Y = Y^{+} - Y^{-}$. Since $X^{\pm}$ is a Borel measurable function of $X$ and $Y^{\pm}$ is a Borel measurable function of $Y$, and Borel measurable functions of independent random variables are again independent (because the events have the form $\{X^{+} \in B\} = \{X \in (\cdot)^{-1}(B)\}$), the nonnegative case applies to all four products $X^{\pm}Y^{\pm}$. Integrability follows from $E[|XY|] = E[|X|]E[|Y|] < \infty$, and the identity from

$$E[XY] = E[X^{+}Y^{+}] - E[X^{+}Y^{-}] - E[X^{-}Y^{+}] + E[X^{-}Y^{-}] = \bigl(E[X^+]-E[X^-]\bigr)\bigl(E[Y^+]-E[Y^-]\bigr) .$$

Finally, part 1 of Proposition 5.3 gives $\mathrm{Cov}(X,Y) = E[XY] - E[X]E[Y] = 0$.

The converse fails: Example 5.5 is a counterexample.

∎

Remark 7.6

The space $L^2$ is a Hilbert space under the inner product $\langle X, Y\rangle := E[XY]$; the variance is the squared norm $\|X - \mu_X\|^2$ of the centred variable, the covariance is the corresponding inner product, and the correlation coefficient is the cosine of the angle between them. This is why Theorem 5.6 is nothing but the Cauchy–Schwarz inequality in a Hilbert space. The viewpoint becomes essential when the conditional expectation is understood as an “orthogonal projection onto a subspace”. For $L^2$ as a Hilbert space see L² is a Hilbert space（Theorem 6.2）［L^p 空間と関数解析への導入］ in L^p spaces and an introduction to functional analysis.