Dark Matter and Dark Energy: 95% of the Universe Has Only a Name So Far

0. Key points

• The mass of an astronomical object can be measured in two ways: with light, and with gravity. That the two answers differ by a factor of more than five is exactly what the dark matter problem is.
• Because the orbital speed of stars does not fall off in the outskirts of a galaxy (a flat rotation curve), the mass enclosed within radius $r$ must keep growing as $M(r) \propto r$ — long after the luminous stars have run out.
• Dark matter is not “ordinary matter that happens to be dim”. Big Bang nucleosynthesis and the cosmic microwave background independently measure the total amount of protons and neutrons, and the missing mass exceeds that total by more than a factor of five.
• In 1998, distant Type Ia supernovae turned out to be fainter than predicted. Far from decelerating, the universe was accelerating. Acceleration requires negative pressure; the condition is that the equation-of-state parameter satisfy $w < -1/3$.
• The present budget is 4.9% ordinary matter, 26.4% dark matter, 68.5% dark energy. The matter whose name and properties we know is the rounding error of the universe.
• Neither identity is settled. Forty years of direct dark matter searches have come up empty, and the naive theoretical value of the dark energy density misses the observed one by about 120 orders of magnitude.

1. Motivation: an object can be weighed in two ways

When an astronomer says “that galaxy weighs as much as a hundred billion suns”, the number comes from one of two broad procedures.

The first is to measure with light. The brightness of a star is essentially fixed by its mass: heavier stars have higher central pressures, burn their fuel faster, and shine brighter. So one adds up all the light arriving from a galaxy and multiplies by a conversion rate — so many kilograms per unit of luminosity — to obtain the total stellar mass. It is like totalling a shopping receipt.

The second is to measure with gravity. If anything orbits anything else, the way it orbits reports the central mass. From the single fact that the Earth goes around the Sun at $29.8$ km/s, one can work backwards to the Sun’s mass of $2 \times 10^{30}$ kg. We cannot put a star on a scale, but we can look at how hard it pulls on its neighbours.

Since both procedures weigh the same object, they ought to return the same answer. They do not.

In 1933 Fritz Zwicky, then working in Switzerland, measured how fast the galaxies belonging to the Coma cluster were flying about. A cluster is a group held together by gravity, so if the galaxies inside it move too fast, the cluster cannot hold them and they escape. What he found was that the galaxies were moving far too fast to be bound by the mass inferred from their light. In his paper Zwicky wrote down the phrase dunkle Materie — dark matter.

At the time this was largely ignored. Clusters are awkward objects to measure, and the discrepancy was dismissed with “there must be plenty of faint stars we have not seen yet”. The tide turned in the 1970s, when Vera Rubin and Kent Ford measured the rotation of spiral galaxies precisely. What they studied was not a cluster but the interior of a single galaxy: data with nowhere to hide, and plain to read.

Note

The “dark” in dark matter means “does not interact with light”, not “black”. Black soot absorbs light; dark matter neither absorbs nor reflects nor emits it. “Transparent matter” would be the more accurate term, and I suspect it was passed over for not sounding impressive enough.

2. Preliminaries: weighing things with circular motion

We first assemble the tool that reads a mass off a rotation. High-school circular motion is all we need.

Definition 2.1（Rotation curve）

For a disc-shaped object such as a spiral galaxy, the graph of the orbital speed $v(r)$ of the stars and gas at distance $r$ from the centre, plotted as a function of $r$, is called its rotation curve.

Definition 2.2（Dark matter）

A component of matter that neither emits nor absorbs electromagnetic radiation (light, radio waves, X-rays and so on), and whose existence is detectable only through gravity, is called dark matter. To match the observed formation of cosmic structure, its velocities are further assumed to be small compared with the speed of light — that is, it is taken to be “cold”.

Proposition 2.3（Keplerian rotation）

Suppose a body of mass $M$ sits at the centre and that the mass outside it is negligible. Then the speed $v(r)$ of a body of mass $m$ in a circular orbit at distance $r$ from the centre is

$$v(r) = \sqrt{\frac{GM}{r}} \propto r^{-1/2},$$

where $G = 6.674 \times 10^{-11}\ \mathrm{N\,m^2/kg^2}$ is the gravitational constant. In particular $v(r)$ decreases monotonically as $r$ grows.

Proof（Proposition 2.3）

A body of mass $m$ moving on a circle of radius $r$ requires a centripetal force of magnitude $mv^2/r$ directed at the centre. That force is supplied by gravity, of magnitude $GMm/r^2$. Equating the two,

$$\frac{mv^2}{r} = \frac{GMm}{r^2}.$$

Dividing both sides by $m$ (which is nonzero) and multiplying by $r$ gives $v^2 = GM/r$. Since $v > 0$, taking the square root yields $v = \sqrt{GM/r}$. If $M$ is constant, then $v \propto r^{-1/2}$.

∎

The content of this proposition is that things farther out move more slowly. The central gravity weakens with distance, so a distant body can stay in orbit while moving leisurely.

Example 2.4（The solar system rotates Keplerian, as advertised）

In the solar system almost all the mass (99.86%) is concentrated in the central Sun, so Proposition 2.3 should apply directly.

The Earth orbits at $1$ au with a speed of $29.8$ km/s. Neptune sits at $30.1$ au, so its predicted speed is

$$v_{\text{Neptune}} = 29.8 \times \frac{1}{\sqrt{30.1}} = \frac{29.8}{5.49} = 5.43\ \mathrm{km/s}.$$

Neptune’s observed orbital speed is about $5.43$ km/s: agreement to two decimal places. Keplerian rotation is not a fantasy but a law holding in plain sight.

3. Dark matter (1): the rotation curve as unmovable evidence

The same thing should happen in a galaxy. A spiral galaxy has stars packed into a central bulge and thinning out towards the edge. Far enough from the centre there is essentially no further mass, so by Proposition 2.3 the rotation speed ought to fall off as $r^{-1/2}$.

It does not fall off.

Galaxy rotation curves: the prediction from visible matter (dashed) against the actual observation (solid)

What Rubin and Ford saw was a flat rotation curve: the speed barely dropped even in the outer parts of the galaxy. And this was not a peculiarity of Andromeda — every spiral galaxy they examined behaved the same way. Translated into formulas, the fact says the following.

Proposition 3.1（What a flat rotation curve implies）

Suppose the rotation curve of a galaxy takes the constant value $v_0 > 0$ on an interval $r_1 \le r \le r_2$. Assuming a spherically symmetric mass distribution, the total mass $M(r)$ contained inside the sphere of radius $r$ and the mass density $\rho(r)$ there satisfy, on that interval,

$$M(r) = \frac{v_0^{2}\, r}{G}, \qquad \rho(r) = \frac{v_0^{2}}{4\pi G\, r^{2}}.$$

That is, $M(r)$ keeps growing in proportion to $r$, and the density thins out only slowly, as $r^{-2}$.

Proof（Proposition 3.1）

For a spherically symmetric mass distribution, the gravity acting at radius $r$ is the same as if all the mass $M(r)$ inside were concentrated at the centre (Newton’s shell theorem). Hence the same balance equation as in the proof of Proposition 2.3 holds with $M$ replaced by $M(r)$:

$$\frac{v(r)^2}{r} = \frac{G M(r)}{r^{2}} \quad \Longrightarrow \quad M(r) = \frac{v(r)^{2} r}{G}.$$

By hypothesis $v(r) = v_0$ is constant, so $M(r) = v_0^2 r / G$. That is the first claim.

Now for the density. Differentiating both sides of $M(r) = \displaystyle\int_0^r 4\pi s^2 \rho(s)\, ds$ with respect to $r$ gives

$$\frac{dM}{dr} = 4\pi r^{2} \rho(r).$$

On the other hand, differentiating the $M(r) = v_0^2 r/G$ just obtained gives $dM/dr = v_0^2/G$. Equating the two,

$$4\pi r^{2}\rho(r) = \frac{v_0^{2}}{G} \quad \Longrightarrow \quad \rho(r) = \frac{v_0^{2}}{4\pi G r^{2}}$$

as claimed.

∎

What Proposition 3.1 forces on us is this. Starlight is concentrated near the centre of a galaxy and fades sharply outwards. The mass, however, keeps dutifully increasing in proportion to $r$. There is no escaping the conclusion that something non-luminous extends around the galaxy as an enormous sphere — a dark matter halo.

Example 3.2（Weighing the Milky Way at two radii）

Our Sun orbits at about $8$ kpc (kiloparsecs) from the centre of the Milky Way at $220$ km/s. First convert units: since $1\ \mathrm{pc} = 3.086 \times 10^{16}$ m, we have $8\ \mathrm{kpc} = 8000 \times 3.086 \times 10^{16} = 2.47 \times 10^{20}$ m. Substituting into the formula of Proposition 3.1,

$$M(8\ \mathrm{kpc}) = \frac{(2.2\times 10^{5})^{2} \times 2.47\times 10^{20}}{6.674\times 10^{-11}} = \frac{1.20\times 10^{31}}{6.674\times 10^{-11}} = 1.79\times 10^{41}\ \mathrm{kg}.$$

Dividing by the solar mass $M_\odot = 1.989 \times 10^{30}$ kg gives $9.0 \times 10^{10}\,M_\odot$, that is, ninety billion suns. So far this roughly balances against the amount of visible stars and gas.

The trouble lies farther out. Radio observations of hydrogen gas in the galactic disc show that the rotation speed is still around $200$ km/s at $r = 50$ kpc. The same computation, with $50\ \mathrm{kpc} = 1.543\times10^{21}$ m, gives

$$M(50\ \mathrm{kpc}) = \frac{(2.0\times 10^{5})^{2} \times 1.543\times 10^{21}}{6.674\times 10^{-11}} = 9.25\times 10^{41}\ \mathrm{kg} = 4.7\times 10^{11}\,M_\odot.$$

The mass has grown by more than a factor of five. Yet between $8$ kpc and $50$ kpc there are hardly any stars to account for the extra four hundred billion solar masses, since starlight has essentially run out by about $15$ kpc.

Including still larger radii, the total mass of the Milky Way is estimated at about $1 \times 10^{12}\,M_\odot$, some 15 to 20 times the total mass in stars (about $6 \times 10^{10}\,M_\odot$).

Remark 3.3

The local dark matter density in the Sun’s neighbourhood is roughly $0.4\ \mathrm{GeV/cm^3}$, which in kilograms is about $7 \times 10^{-22}\ \mathrm{kg/m^3}$. All the dark matter contained in one Earth’s worth of volume ($1.08 \times 10^{21}\ \mathrm{m^3}$) does not add up to one kilogram. That is remarkably thin for “the matter that rules the universe”, but the volume of a galactic halo is more than $10^{40}$ times that of the Earth, so summed up it outweighs the stars. Thinness and total amount are separate questions.

4. Dark matter (2): it is not “ordinary matter that fails to shine”

Rotation curves alone leave open the escape route “there must be extra dim stars and cold gas”. In this section we see that every such route is now closed.

4.1. Clusters: redoing Zwicky’s calculation

We first prepare the tool Zwicky used.

Proposition 4.1（The virial theorem）

For a system of $N$ point masses bound by their mutual gravity alone and in a statistically steady state, the long-time averages of the total kinetic energy $T$ and the total gravitational potential energy $U$ satisfy

$$2\langle T \rangle + \langle U \rangle = 0.$$

In particular, writing $M$ for the total mass of the system, $R$ for its typical extent and $\sigma$ for the line-of-sight velocity dispersion, and assuming an isotropic velocity distribution ($\langle v^2 \rangle = 3\sigma^2$) together with the approximation $U \simeq -GM^2/R$, one estimates $M \simeq 3\sigma^2 R / G$.

Proof（Proposition 4.1）

Write $m_i$ for the mass of point $i$, $\boldsymbol{r}_i$ for its position, $\boldsymbol{v}_i$ for its velocity and $\boldsymbol{F}_i$ for the total force on it, and consider the moment-of-inertia-like quantity $I = \sum_i m_i |\boldsymbol{r}_i|^2$. Differentiate twice in time:

$$\frac{dI}{dt} = 2\sum_i m_i \boldsymbol{r}_i \cdot \boldsymbol{v}_i, \qquad \frac{d^2 I}{dt^2} = 2\sum_i m_i |\boldsymbol{v}_i|^2 + 2\sum_i \boldsymbol{r}_i \cdot \boldsymbol{F}_i = 4T + 2\sum_i \boldsymbol{r}_i \cdot \boldsymbol{F}_i.$$

The second equality uses Newton’s equation of motion $m_i \boldsymbol{a}_i = \boldsymbol{F}_i$. Now compute the second term on the right. Since gravity gives $\boldsymbol{F}_i = -\sum_{j \ne i} G m_i m_j (\boldsymbol{r}_i - \boldsymbol{r}_j)/|\boldsymbol{r}_i - \boldsymbol{r}_j|^3$, collecting terms pair by pair in $i$ and $j$ yields

$$\sum_i \boldsymbol{r}_i \cdot \boldsymbol{F}_i = -G\sum_{i < j} m_i m_j \frac{(\boldsymbol{r}_i - \boldsymbol{r}_j)\cdot(\boldsymbol{r}_i - \boldsymbol{r}_j)}{|\boldsymbol{r}_i - \boldsymbol{r}_j|^{3}} = -G\sum_{i<j} \frac{m_i m_j}{|\boldsymbol{r}_i - \boldsymbol{r}_j|} = U.$$

Hence $d^2 I/dt^2 = 4T + 2U$. If the system is bound, $I$ stays bounded, so the average of $d^2I/dt^2$ over a long time tends to zero. Therefore $4\langle T\rangle + 2\langle U\rangle = 0$, that is, $2\langle T\rangle + \langle U\rangle = 0$.

For the estimate in the second part, substitute $\langle T \rangle = \frac{1}{2}M\langle v^2\rangle = \frac{3}{2}M\sigma^2$ and $\langle U \rangle \simeq -GM^2/R$ to get $3M\sigma^2 = GM^2/R$, and divide both sides by $M$ to obtain $M \simeq 3\sigma^2 R/G$. The coefficient in $U$ varies around unity depending on the shape of the mass distribution, so this is an order-of-magnitude estimate.

∎

Example 4.2（The mass of the Coma cluster）

The line-of-sight velocity dispersion of the galaxies in the Coma cluster is $\sigma \approx 1000$ km/s and its extent is $R \approx 1.4$ Mpc. Since $1.4\ \mathrm{Mpc} = 1.4\times10^{6}\times3.086\times10^{16} = 4.32\times10^{22}$ m, Proposition 4.1 gives

$$M \simeq \frac{3 \times (1.0\times10^{6})^{2} \times 4.32\times10^{22}}{6.674\times10^{-11}} = 1.94\times10^{45}\ \mathrm{kg} \approx 1\times10^{15}\,M_\odot.$$

The total stellar mass of the cluster, on the other hand, is about $10^{13}\,M_\odot$. Adding the hot gas that shines in X-rays brings the total only to some $1.5 \times 10^{14}\,M_\odot$. Around 85% of the whole is therefore unaccounted for.

In his 1933 paper Zwicky reported an outlandish discrepancy of a factor of 400. That number came out too large because the Hubble constant of the day was overestimated by nearly a factor of eight, which made the distance to the cluster far too small (for how distances are measured see How we know the distance to a star, and for the history of the miscalibrated scale in particular see the ruler was once wrong（Remark 4.6）［How We Know the Distance to a Star］). Correcting the distance to modern values reduces the factor to about six. The order of magnitude came down; the conclusion survived.

4.2. The Bullet Cluster: the moment dark matter became visible

Example 4.3（The Bullet Cluster 1E 0657-56）

We have observations of the site of a head-on collision between two galaxy clusters. The contents of a cluster split into three parts: (1) stars, that is, the galaxies themselves, (2) hot gas, and (3) mass of unknown identity.

What happens in a collision? Galaxies are so sparse that they simply pass through each other with almost no deceleration. The hot gas, being a fluid, drives shock waves when it collides head-on, decelerates, and is left stranded in the middle. And indeed, in X-rays the hot gas appears at the centre, while in visible light the galaxies appear separated to either side.

Where, then, is the mass? This can be measured by gravitational lensing: the degree to which foreground mass distorts the images of background galaxies allows one to draw a map of the mass distribution (for how light bends near a heavy object see also What lies beyond a black hole, in particular the size of the shadow of M87*（Example 7.1）［What Lies Beyond a Black Hole］).

The answer was “with the galaxies”. The centres of mass had passed straight through to either side together with the galaxies, not with the gas. Most of the baryons (ordinary matter) are in the gas, yet the mass is not where the gas is. This is very hard to accommodate in any explanation of the type “the law of gravity changes according to the distribution of visible matter”, and it constitutes direct evidence that a mass component exists which passes through without colliding.

4.3. Why every “ordinary matter” candidate collapses

Candidate	Why it is ruled out
Dim stars, planets, rocks (MACHOs)	When one crosses in front of a background star, that star brightens temporarily (microlensing). MACHO, EROS, OGLE and others monitored tens of millions of stars and did not detect anywhere near enough objects to fill the halo
Cold hydrogen gas	Gas absorbs the light of background sources, so in large quantities it would inevitably show up as absorption lines
Neutrinos	The upper bound on their mass (below $0.12$ eV summed over the three species) gives $\Omega_\nu \lesssim 0.003$ for the universe as a whole, less than 1% of what is needed. They are also too fast, and would smooth out small-scale structure
Black holes (stellar mass)	Making them requires progenitor stars, so one runs into the ceiling on the total amount of baryons anyway

And there are two decisive arguments.

The total amount of baryons has been measured independently in two ways. In nucleosynthesis three minutes after the Big Bang, the higher the density of protons and neutrons, the more completely deuterium burns into helium. Measuring the amount of leftover deuterium therefore fixes the baryon density of the universe, giving about $\Omega_b h^2 = 0.0224$. The temperature fluctuations of the cosmic microwave background (CMB), a completely different piece of physics, also give $\Omega_b h^2 = 0.0224$. But the CMB simultaneously measures the density of the component that clusters gravitationally without interacting with light, and that comes out at $\Omega_c h^2 = 0.120$, about $5.4$ times larger. No number of dim stars closes this gap (for the details of Big Bang nucleosynthesis see Was there really a Big Bang?, and for the argument fixing the baryon density from deuterium see two completely different methods giving the same answer（Example 5.5）［Was There Really a Big Bang? Expansion, Background Radiation, and the Ratio of the Elements］).

The mere existence of galaxies. At the epoch when the CMB was released — 380,000 years after the beginning; see when the universe became transparent（Example 4.3）［Was There Really a Big Bang? Expansion, Background Radiation, and the Ratio of the Elements］ — the density contrast of matter was only one part in a hundred thousand. Before that, baryons were colliding constantly with light and could not grow their contrast, because radiation pressure pushed them back. If the universe contained only baryons, the contrast would have grown at most by a factor of about a thousand from the moment growth became free until today, that is, to $10^{-5} \times 10^{3} = 10^{-2}$. Nothing forms unless the contrast reaches unity. The fact that galaxies do exist leaves no alternative but a component which, precisely because it ignores light, began growing its contrast at an earlier epoch, with the baryons falling into its gravitational wells afterwards.

4.4. So what is it?

Everything so far proved that it exists. What it is remains unknown. Here are the leading candidates.

Candidate	What it is	Points in its favour	Points against
WIMP	An undiscovered particle with a mass roughly 10 to 1000 times the proton’s	Arises naturally from standard extensions of particle physics, and the relic abundance left over from the early universe comes out right at the observed value (the “WIMP miracle”)	Neither underground experiments nor accelerators have found it in more than thirty years
Axion	An ultralight particle proposed to solve the strong CP problem	Appears naturally as a by-product of an entirely different problem	The allowed mass range is vast, so the search never ends
Primordial black holes	Small black holes born directly from density fluctuations just after the Big Bang	Requires no new particle	Microlensing observations have excluded most of the mass range as a dominant component
Sterile neutrinos	Relatives of the neutrino that do not even feel the weak interaction	Would simultaneously explain why neutrinos have mass	The X-ray hints have never been confirmed

The underground experiments hunting WIMPs (XENONnT, LUX-ZEPLIN and others) try to catch the faint flash of a nucleus being knocked about inside liquid xenon. Their sensitivity has improved year after year and has now reached cross sections of order $10^{-48}\ \mathrm{cm^2}$. Still no signal. They are now entering the “neutrino fog”, where neutrinos from the Sun and the atmosphere produce indistinguishable signals, and the search has reached a critical point.

5. Dark energy: the universe is accelerating

Dark matter was a problem of gravity being too strong. Next comes the opposite problem: gravity being too weak.

5.1. Supernovae fainter than predicted

In the 1990s two international teams drew up the same plan: to observe distant Type Ia supernovae. A Type Ia supernova occurs when a white dwarf exceeds a limiting mass and explodes, and since the scale of the explosion is nearly fixed, its apparent brightness gives its distance (standard candle（Definition 2.3）［How We Know the Distance to a Star］). At the same time, the redshift of its light tells us how much smaller the universe was when it exploded. Put the two together and the expansion history of the universe becomes readable.

The aim was “to measure how much the universe is decelerating”. Matter attracts matter, so the expansion must surely be slowing down; determining by how much would settle whether the universe eventually collapses or expands forever. That was the plan.

In 1998 both teams reached the same answer. Distant supernovae were fainter than a decelerating universe predicts. Fainter means farther, and farther means that the universe expanded more than expected while the light was in flight. The universe was not decelerating. It was accelerating.

Even inside one of the two teams, the result was suspected of harbouring a mistake somewhere. It did not go away, and it earned the 2011 Nobel Prize in Physics.

5.2. Acceleration requires negative pressure

Let us see with formulas why acceleration is a surprise.

Definition 5.1（Equation-of-state parameter）

For a uniformly spread component with pressure $p$ and energy density $\rho c^2$, the ratio

$$w = \frac{p}{\rho c^{2}}$$

is called the equation-of-state parameter of that component. Ordinary matter (dust) has negligible pressure, so $w = 0$; radiation such as light has $w = 1/3$.

Proposition 5.2（The condition for the universe to accelerate）

Let $a(t)$ be the scale factor describing the size of a homogeneous and isotropic universe, $\rho c^2$ the total energy density and $p$ the total pressure. Einstein’s equations then give

$$\frac{\ddot a}{a} = -\frac{4\pi G}{3}\left(\rho + \frac{3p}{c^{2}}\right).$$

Consequently, when $\rho > 0$, a necessary and sufficient condition for the expansion to accelerate ($\ddot a > 0$) is $w < -1/3$.

Proof（Proposition 5.2）

The first term (the $\rho$ part) follows from Newtonian mechanics alone. Consider a sphere of radius $R$ filled with uniform matter of density $\rho$, and follow the motion of a particle of mass $m$ on its surface. By the shell theorem the gravity on this particle is the same as if the interior mass $M = \frac{4\pi}{3}\rho R^3$ were at the centre, so

$$m\ddot R = -\frac{GMm}{R^{2}} = -\frac{4\pi G}{3}\rho m R \quad \Longrightarrow \quad \frac{\ddot R}{R} = -\frac{4\pi G}{3}\rho .$$

Since $R \propto a$, this gives $\ddot a/a = -\frac{4\pi G}{3}\rho$. The pressure term $3p/c^2$ is specific to general relativity, a consequence of Einstein’s equations in which energy and momentum both act as sources of gravity. Here we take that result for granted.

Now for the necessary and sufficient condition. By Definition 5.1 we have $p = w\rho c^2$, so

$$\rho + \frac{3p}{c^{2}} = \rho + 3w\rho = \rho\,(1 + 3w).$$

We have $\ddot a > 0$ exactly when $-\frac{4\pi G}{3}\rho(1+3w) > 0$, that is, when $\rho(1+3w) < 0$. Since we assumed $\rho > 0$, we may divide both sides by $\rho$, obtaining $1 + 3w < 0$, that is, $w < -1/3$. Reading the argument backwards shows the condition is sufficient as well.

∎

Ordinary matter has $w = 0$ and radiation has $w = 1/3$, so neither satisfies $w < -1/3$. With only the familiar constituents, the expansion must decelerate. That it was accelerating means the universe is filled with a component of large negative pressure. The name Michael Turner gave to this component in 1998 is dark energy.

Caution

“Negative pressure” is often pictured as a pulling force, but that is not the cause of the acceleration. Even something with tension, like a rubber band, exerts no net push or pull once it is spread out uniformly, because the forces cancel in every direction. What matters is that pressure enters the formula in Proposition 5.2 as a source of gravity. Negative pressure acts as negative gravity, that is, as repulsion.

5.3. Vacuum energy as the leading candidate

The simplest candidate is the cosmological constant $\Lambda$, introduced by Einstein in 1917 for an entirely different purpose, or in modern language “a constant energy density carried by the vacuum itself”. That this candidate produces acceleration can be verified using nothing but the first law of thermodynamics.

Proposition 5.3（Constant energy density implies negative pressure）

Suppose that in an expanding universe the energy density $\rho c^2$ of some component is always the same regardless of volume. If that component exchanges no heat with its surroundings (that is, the change is adiabatic), then its pressure is $p = -\rho c^2$, so $w = -1$.

Proof（Proposition 5.3）

The energy of this component contained in a region of volume $V$ is $U = \rho c^2 V$. By hypothesis $\rho c^2$ is constant, so when the volume increases by $dV$ the energy changes by $dU = \rho c^2\, dV$.

On the other hand, the first law of thermodynamics for an adiabatic change reads $dU = -p\, dV$: the internal energy drops by the work done in pushing outwards. Equating the two expressions,

$$\rho c^{2}\, dV = -p\, dV .$$

Since $dV \ne 0$, dividing both sides by $dV$ gives $p = -\rho c^2$, hence $w = p/(\rho c^2) = -1$.

Intuitively, the situation is this. An ordinary gas dilutes as it expands, but vacuum energy does not: the extra volume comes with extra energy. To respect conservation of energy, somebody must have done the work that supplied the increase, and the only candidate is “the universe did work stretching the vacuum”. Stretching costs work — that is what the negative pressure is.

∎

Corollary 5.4（Vacuum energy accelerates the expansion）

A universe consisting solely of a component of constant energy density ($\rho_\Lambda > 0$) necessarily expands with acceleration.

Proof（Corollary 5.4）

By Proposition 5.3 this component has $w = -1$. Since $-1 < -1/3$, the condition $w < -1/3$ of Proposition 5.2 is satisfied. Substituting explicitly,

$$\frac{\ddot a}{a} = -\frac{4\pi G}{3}\left(\rho_\Lambda - 3\rho_\Lambda\right) = \frac{8\pi G}{3}\rho_\Lambda > 0,$$

which confirms the acceleration.

∎

How “thick” is this dark energy in practice? To write the budget in densities rather than percentages, we set up the reference density used in cosmology.

Definition 5.5（Critical density and density parameters）

For a Hubble constant $H_0$, the quantity $\rho_c = \dfrac{3H_0^{2}}{8\pi G}$ is called the critical density. In a universe whose space is flat (Euclidean), the sum of the densities of all components (energy densities divided by $c^2$) equals exactly this value. For a component of density $\rho_i$, the ratio $\Omega_i = \rho_i/\rho_c$ is called its density parameter. By definition, the $\Omega_i$ of all components sum to $1$ in a flat universe.

Example 5.6（How thick is dark energy?）

The observed universe is flat to within the accuracy of measurement, so we take the sum of all densities to be the critical density. Convert $H_0 = 67.4$ km/s/Mpc to SI units. Since $1\ \mathrm{Mpc} = 3.086\times10^{22}$ m,

$$H_0 = \frac{6.74\times10^{4}\ \mathrm{m/s}}{3.086\times10^{22}\ \mathrm{m}} = 2.184\times10^{-18}\ \mathrm{s^{-1}} .$$

Substituting,

$$\rho_c = \frac{3\times(2.184\times10^{-18})^{2}}{8\pi\times 6.674\times10^{-11}} = \frac{1.431\times10^{-35}}{1.677\times10^{-9}} = 8.5\times10^{-27}\ \mathrm{kg/m^{3}} .$$

A single hydrogen atom weighs $1.67\times10^{-27}$ kg, so the mean density of the whole universe is about five hydrogen atoms per cubic metre. It is the best vacuum on record.

Dark energy makes up $\Omega_\Lambda = 0.685$ of this, so $\rho_\Lambda = 5.8\times10^{-27}\ \mathrm{kg/m^3}$, or $3.5$ hydrogen atoms’ worth. Ordinary matter has $\Omega_b = 0.049$, one hydrogen atom per four cubic metres. That the dark energy driving the universe has any effect at all despite being this thin is only because the universe is vast and contains nothing else.

Example 5.7（When did the acceleration begin?）

Matter dilutes as the universe expands, falling off as $\rho_m \propto a^{-3}$. Dark energy does not dilute. So matter must have dominated in the past, giving deceleration, until at some point the balance tipped and acceleration set in. Let us find when.

Writing the bracket in Proposition 5.2 as the sum over matter ($w=0$) and the cosmological constant ($w=-1$),

$$\rho + \frac{3p}{c^{2}} = \rho_m + (\rho_\Lambda - 3\rho_\Lambda) = \rho_m - 2\rho_\Lambda .$$

The condition for acceleration is that this be negative, that is, $\rho_m < 2\rho_\Lambda$. Normalising the present scale factor to $a=1$, we have $\rho_m = \Omega_m \rho_c / a^{3}$ and $\rho_\Lambda = \Omega_\Lambda \rho_c$, so

$$\frac{\Omega_m}{a^{3}} < 2\Omega_\Lambda \quad \Longrightarrow \quad a^{3} > \frac{\Omega_m}{2\Omega_\Lambda} = \frac{0.315}{2\times 0.685} = 0.230 .$$

Taking the cube root gives $a > 0.613$, and the relation $1+z = 1/a$ turns this into $z < 0.63$. A redshift of $z = 0.63$ corresponds to a light travel time of roughly six billion years. Given that the universe is $13.8$ billion years old, the universe spent more than half its life decelerating and only switched to acceleration in the second half. We happen to live after the switch.

6. The cosmic budget, and the puzzles that remain

Combining three independent lines of observation — the temperature fluctuations of the CMB, the distances of supernovae, and the ripples in the galaxy distribution known as baryon acoustic oscillations — fixes the contents of the universe as follows.

The energy density budget of the universe (Planck 2018 results)

That thin band on the far left is everything we can write down in the periodic table. Stars, planets and the body of whoever is reading this all belong to that 4.9%, and less than a tenth of it has taken the form of stars, the rest being tenuous gas drifting between galaxies. Four hundred years after the idea that the Earth is the centre of the universe was abandoned, we have accepted that we occupy an unremarkable place in the cosmos — and we are now discovering that we are not even made of the ordinary ingredients.

The budget also comes with a list of unresolved entries.

The identity of dark matter. Rotation curves, clusters, gravitational lensing, the CMB and structure formation give five or more independent lines of evidence that agree, and this part is not going to move. But direct detection keeps coming up empty. Observations do not even guarantee that it is made of particles.

The theoretical value of dark energy is nowhere near the observed one. In quantum theory the vacuum carries zero-point energy. Adding it up naively to the Planck scale gives a vacuum energy density of order $10^{96}\ \mathrm{kg/m^3}$. The observed value is $5.8\times10^{-27}\ \mathrm{kg/m^3}$. The ratio is about $10^{122}$, a discrepancy of 120 orders of magnitude. No other example in the history of physics has theory and observation this far apart. Something must be cancelling, but the heart of the problem is that nobody knows why the cancellation stops 120 orders short of exact.

The disagreement over the Hubble constant (the Hubble tension). The expansion rate inferred indirectly from the CMB is $67.4 \pm 0.5$ km/s/Mpc; measured directly by stacking up nearby objects it is $73.0 \pm 1.0$ km/s/Mpc. If the error estimates are right, this is a discrepancy of more than five standard deviations, at a level hard to attribute to chance. Whether one side carries an unknown systematic error, or whether the early universe contains a component we do not know about, is unsettled (the current state of this disagreement is also discussed in the Hubble constant is still disputed（Remark 3.8）［The Edge and the Age of the Universe］).

Is $w$ really $-1$? Recent large galaxy surveys hint that $w$ may be changing with time. Should that be confirmed, dark energy would not be “vacuum energy” but some field evolving in time, and the very premise of Proposition 5.3 would fail. For the moment this is a hint, not a confirmation.

How the age and the edge of the universe are determined is treated in The edge and age of the universe (the boundary of what can be observed is the particle horizon（Definition 4.1）［The Edge and the Age of the Universe］), and how the plain fact that the night sky is dark connects to this expansion is treated in Why is the night sky dark? (Olbers' paradox（Theorem 3.4）［Why Is the Night Sky Dark? Olbers' Paradox and the Finite Age of the Universe］).

7. Exercises

Exercise 7.1Easy

Suppose the rotation speed of the Milky Way remains constant at $220$ km/s even at $20$ kpc from the centre. Find the total mass inside that radius in solar masses. Take $1\ \mathrm{pc} = 3.086\times10^{16}$ m, $M_\odot = 1.989\times10^{30}$ kg and $G = 6.674\times10^{-11}\ \mathrm{N\,m^2/kg^2}$.

Solution

By Proposition 3.1, $M(r) = v_0^2 r/G$. First convert the distance to SI units:

$$r = 20 \times 10^{3} \times 3.086\times10^{16} = 6.17\times10^{20}\ \mathrm{m}.$$

The speed is $v_0 = 2.2\times10^{5}$ m/s, so $v_0^2 = 4.84\times10^{10}\ \mathrm{m^2/s^2}$. Hence

$$M = \frac{4.84\times10^{10}\times6.17\times10^{20}}{6.674\times10^{-11}} = \frac{2.99\times10^{31}}{6.674\times10^{-11}} = 4.48\times10^{41}\ \mathrm{kg}.$$

Dividing by the solar mass,

$$\frac{4.48\times10^{41}}{1.989\times10^{30}} = 2.25\times10^{11}\,M_\odot .$$

About 225 billion solar masses, which is $2.5$ times the $9.0\times10^{10}\,M_\odot$ found within $8$ kpc in Example 3.2. The radius grew by a factor of $2.5$ and so did the mass — the proportionality shows up directly. Yet between $8$ kpc and $20$ kpc the starlight barely increases at all.

Exercise 7.2Standard

(a) If the equation-of-state parameter of dark energy were $w = -1/2$ rather than $w = -1$, would a universe made of that component alone accelerate? What about exactly $w = -1/3$?

(b) Show, from the first law of thermodynamics for adiabatic changes, that in general the energy density of a component with constant equation-of-state parameter $w$ obeys $\rho \propto a^{-3(1+w)}$ as a function of the scale factor $a$. Use the result to check the cases $w=0$ (matter), $w=1/3$ (radiation) and $w=-1$ (cosmological constant).

Solution

(a) The condition in Proposition 5.2 was $w < -1/3$. Since $-1/2 < -1/3$, the value $w = -1/2$ does accelerate. Explicitly, $\rho(1+3w) = \rho(1 - 3/2) = -\rho/2 < 0$, so $\ddot a/a = \frac{4\pi G}{3}\cdot\frac{\rho}{2} > 0$. The acceleration is weaker than for $w=-1$, however.

For exactly $w = -1/3$ we have $1 + 3w = 0$, so $\ddot a = 0$: neither acceleration nor deceleration, and with this component alone the expansion speed stays constant. So $w = -1/3$ is precisely the boundary between acceleration and deceleration.

(b) Consider a region of volume $V \propto a^3$. The energy of the component in it is $U = \rho c^2 V$. Since the change is adiabatic,

$$d(\rho c^{2} V) = -p\, dV = -w\rho c^{2}\, dV .$$

Expanding the left-hand side gives $c^2(V d\rho + \rho\, dV)$, so dividing both sides by $c^2$ and rearranging,

$$V\, d\rho = -\rho\,dV - w\rho\,dV = -(1+w)\rho\, dV .$$

Here $V \propto a^3$ gives $dV/V = 3\,da/a$. Dividing both sides of the displayed equation by $\rho V$,

$$\frac{d\rho}{\rho} = -(1+w)\frac{dV}{V} = -3(1+w)\frac{da}{a}.$$

Since $w$ is constant, integrating both sides gives $\ln \rho = -3(1+w)\ln a + \text{const}$, that is, $\rho \propto a^{-3(1+w)}$.

Now the checks. For $w = 0$ we get $\rho \propto a^{-3}$, matching the picture of a fixed number of particles in a volume growing as $a^3$. For $w = 1/3$ we get $\rho \propto a^{-4}$: on top of the $a^{-3}$ dilution by volume there is an extra factor from the wavelength of light stretching in proportion to $a$, which drops each photon’s energy as $1/a$. For $w = -1$ we get $\rho \propto a^{0}$, that is, a constant — reproducing exactly the hypothesis of Proposition 5.3.

Exercise 7.3Standard

Consider the position (modified gravity) that “there is no dark matter; rather, at large distances the law of gravity departs from Newton’s inverse-square law”. Explain why the observations of the Bullet Cluster in Example 4.3 are severe for this position.

Solution

On the modified-gravity view, the source of gravity is by definition visible matter, and the strength of gravity varies according to its distribution. The “centre of mass” and the “centre of visible matter” must therefore always coincide.

In the Bullet Cluster, however, most of the baryons (ordinary matter) are in the hot gas, and that gas was left stranded in the middle by the collision. If baryons were the only source of gravity, the peaks of the mass distribution measured by gravitational lensing should sit at the position of the central gas.

In the actual observation, the mass peaks were not with the gas but off to either side, together with the galaxies that had passed straight through. In other words, “whatever generates the gravity” moved separately from the gas and was not decelerated by the collision. This shows two things at once:

1. the dominant mass component is something other than baryons;
2. that component barely collides — neither with itself nor with the gas.

Point 1 denies the very starting point of modified gravity, and point 2 shows that the component cannot be ordinary matter such as dim stars or cold gas, since ordinary matter would collide and decelerate.

In fairness, this is not a complete refutation of modified gravity. The compromise “modified gravity plus a small amount of invisible matter” survives logically. But in that case the original motivation for introducing modified gravity — to avoid postulating invisible matter — is lost.

Exercise 7.4Hard

Take $\Omega_m = 0.315$ and $\Omega_\Lambda = 0.685$ (with $w=-1$).

(a) At what redshift $z$ were the energy density of matter and the density of dark energy exactly equal?

(b) In the far future, at what value of the scale factor will the energy density of matter have thinned to one thousandth of the present dark energy density?

Solution

(a) Normalise the present to $a=1$. As seen in Exercise 7.2, $\rho_m = \Omega_m \rho_c\, a^{-3}$ and $\rho_\Lambda = \Omega_\Lambda \rho_c$ (constant). Equality requires

$$\frac{\Omega_m}{a^{3}} = \Omega_\Lambda \quad \Longrightarrow \quad a^{3} = \frac{\Omega_m}{\Omega_\Lambda} = \frac{0.315}{0.685} = 0.460 .$$

Taking the cube root gives $a = 0.772$. From $1+z = 1/a$, the redshift is

$$z = \frac{1}{0.772} - 1 = 1.296 - 1 = 0.296 \approx 0.30 .$$

This corresponds to roughly $3.5$ billion years ago. Note that it is later than the onset of acceleration ($z \approx 0.63$) found in Example 5.7: acceleration began before the densities became equal. The reason is that pressure enters with three times the weight of $\rho$ (through $\rho + 3p/c^2$), so dark energy can force acceleration while still in the minority.

(b) The condition is $\Omega_m a^{-3} = 10^{-3}\,\Omega_\Lambda$, so

$$a^{3} = \frac{\Omega_m}{10^{-3}\,\Omega_\Lambda} = \frac{0.315}{6.85\times10^{-4}} = 460 .$$

Taking the cube root gives $a = 7.72$. It is exactly ten times the answer to (a) because the cube root of $1000$ is $10$. Once the universe has expanded a further factor of $7.7$, matter becomes completely negligible compared with dark energy.

By then only dark energy remains, so the expansion becomes exponential, $a \propto e^{H_\Lambda t}$ with $H_\Lambda = H_0\sqrt{\Omega_\Lambda} = 0.83\,H_0$. Using $1/H_0 \approx 14.5$ billion years, the time for $a$ to go from $1$ to $7.72$ is

$$t = \frac{\ln 7.72}{0.83\,H_0} = \frac{2.04}{0.83} \times 14.5\ \text{billion years} \approx 36\ \text{billion years} .$$

That is about $2.6$ times the present age of the universe of $13.8$ billion years into the future.

References

• F. Zwicky, “Die Rotverschiebung von extragalaktischen Nebeln”, Helvetica Physica Acta 6 (1933), 110–127. The paper that first pointed out dark matter, applying the virial theorem to the Coma cluster.
• V. C. Rubin and W. K. Ford Jr., “Rotation of the Andromeda Nebula from a Spectroscopic Survey of Emission Regions”, The Astrophysical Journal 159 (1970), 379. The observations that established flat rotation curves.
• A. G. Riess et al., “Observational Evidence from Supernovae for an Accelerating Universe and a Cosmological Constant”, The Astronomical Journal 116 (1998), 1009 (arXiv:astro-ph/9805201), and S. Perlmutter et al., “Measurements of Ω and Λ from 42 High-Redshift Supernovae”, The Astrophysical Journal 517 (1999), 565 (arXiv:astro-ph/9812133). The two independent reports of accelerating expansion.
• D. Clowe et al., “A Direct Empirical Proof of the Existence of Dark Matter”, The Astrophysical Journal Letters 648 (2006), L109 (arXiv:astro-ph/0608407). The analysis of the Bullet Cluster.
• Planck Collaboration, “Planck 2018 results. VI. Cosmological parameters”, Astronomy & Astrophysics 641 (2020), A6 (arXiv:1807.06209). The source of the cosmological parameters used in this article.
• Takahiko Matsubara, Gendai Uchūron: Jikū to Busshitsu no Kyōshinka (Modern Cosmology: The Coevolution of Spacetime and Matter), University of Tokyo Press, 2010 (in Japanese) — Chapters 1 to 3 for the Friedmann equations and density parameters.

Appendix: MOND, the other road

The idea. In 1983 Mordehai Milgrom proposed not to add dark matter but to change the law of gravity instead. The proposal is called Modified Newtonian Dynamics (MOND). The hypothesis is that in regions of very small acceleration — specifically, below $a_0 \approx 1.2\times10^{-10}\ \mathrm{m/s^2}$ — the gravitational acceleration switches from $GM/r^2$ to $\sqrt{GMa_0}/r$.

Why rotation curves come out flat. Writing the balance for circular motion under this hypothesis gives $v^2/r = \sqrt{GMa_0}/r$; multiplying both sides by $r$ gives $v^2 = \sqrt{GMa_0}$, that is, $v = (GMa_0)^{1/4}$. No $r$ appears on the right. Flat rotation curves therefore follow automatically from the assumption. Moreover the relation $v^4 = GMa_0$ has the same form as the empirical relation observed between the luminosity and the rotation speed of spiral galaxies (the Tully–Fisher relation). In fact, the accuracy with which MOND reproduces the rotation curves of individual galaxies is sometimes better than that of dark matter models.

Why it nevertheless is not mainstream. Away from the scale of a single galaxy, MOND runs into trouble. Applied to galaxy clusters, MOND alone still falls short of the required mass by about a factor of two. The Bullet Cluster of Example 4.3 is difficult to explain in principle, in the sense that the centres of visible matter and of mass are displaced from one another. And the greatest obstacle is the CMB. The power spectrum of the temperature fluctuations of the cosmic microwave background has a series of peaks, and their relative heights (especially that of the third peak) are set by how much mass there is that does not interact with light. No MOND-type theory currently reproduces this observation without an additional dark component.

Why it is still worth studying. MOND has survived because it works startlingly well on galactic scales. On the dark matter side, nobody has yet explained from first principles why the behaviour appears to switch around one particular value of acceleration, $a_0$. Nor is it known whether it is a coincidence or a clue that $a_0$ is roughly of the order of $cH_0/(2\pi)$. The history of science offers many episodes in which the question posed by a minority theory turned out to be the right one.