Docker and Kubernetes: Why Containers Are Light and Clusters Heal Themselves

0. Key points

• A container is not a small virtual machine. It is an ordinary process on the host, with the kernel’s namespaces restricting what it can see and cgroups restricting how much it can use. There is only one kernel, and it belongs to the host.
• The real content of the “it works on my machine” problem is that a program’s true dependencies are far larger than its requirements.txt — they include the operating system’s libraries, its file layout, and its environment variables — and that all of this is left implicit. A Dockerfile turns those implicit dependencies into code whose changes Git can track.
• An image is a sequence of immutable layers. From that structure alone we obtain two facts: the build cache can only hit on a prefix of the instruction sequence (Proposition 4.2), and deleting a file in a later layer does not shrink the image (Proposition 4.5). Nearly all the conventional wisdom about writing Dockerfiles is a consequence of these two.
• The core of Kubernetes is “a declarative API plus a reconciliation loop”. The user writes down the desired state, and controllers keep closing the gap between it and the current state. We show in Proposition 6.2 that this design converges in finitely many steps and that it survives dropped notifications.
• The benefit of redundancy can be computed. Under independent failures the availability is $1 - p^{N}$, but if all replicas sit on the same node, adding replicas hits a ceiling set by the node’s own availability (Theorem 7.1, Corollary 7.3).
• The speed of a zero-downtime update is likewise fixed by configuration. The duration of a rolling update has the lower bound $Nt/(u+s)$ (Proposition 7.7), and the accident of “we played it safe and it took 20 minutes” can be predicted from this formula in advance.

1. Motivation: what exactly is the problem with “it works on my machine”

Someone joins the team, clones the repository, follows the README, and nothing runs. Half a day of investigation later, the cause turns out to be that developer A’s macOS had libjpeg installed through Homebrew while the newcomer’s machine did not. In software development, time disappears into this sort of thing without limit. Failures that occur only on the production server, or only in CI, are other symptoms of the same disease.

Naively this is puzzling. The source code is shared in full through Git; not a single byte differs. The behaviour differs anyway because the source code is not the only thing the program depends on. Enumerating the dependencies shows that there are more of them than one expects.

Layer of dependency	Examples	How it used to be shared
Language packages	requirements.txt, package-lock.json	Shared through Git
OS shared libraries	versions of libssl, libjpeg, glibc	“Please install these” in a setup document
OS configuration	locale, time zone, file descriptor limits	Nobody wrote it down
File layout	absolute paths of config files, location of certificates	Different for each person
Runtime environment variables	DATABASE_URL, PATH	A .env file sent over Slack
Kernel and architecture	Linux 5.x or 6.x, amd64 or arm64	Not even consciously considered

Only the top row was shared through Git. The five rows below it were shared through a setup document — a program written in natural language that is never executed. A document that is never executed cannot be tested, and what cannot be tested rots.

Historically, the answers to this problem were refined roughly in the following order.

1. Setup documents (a wiki, or a Word file). They are never executed, so they inevitably drift away from reality.
2. Configuration management tools (CFEngine, Puppet, Chef, Ansible). These turned setup procedures into code. That was a large step forward, but their target is “an existing server on which something is already installed”, so the result depends on the previous state. Running the same playbook against a machine that has been in production for ten years and against a brand-new one can give different results.
3. Virtual machines (VMware, VirtualBox, Vagrant). Shipping the whole operating system raised reproducibility dramatically. The price was weight: images of several gigabytes, boot times of tens of seconds, and at best a handful running simultaneously on one laptop.
4. Containers. These achieve the same “ship the whole environment” as virtual machines without duplicating an entire operating system.

One historical remark. The component technologies of containers were not invented by Docker. chroot, which replaces the root of the file system a process sees, was in Version 7 Unix in 1979; FreeBSD jail appeared in 2000, Solaris Zones in 2005, and Linux cgroups — developed at Google as “process containers” — entered kernel 2.6.24 in 2008. The ingredients were all on the table by 2008.

What Docker changed when it appeared in 2013 was not the isolation mechanism but the format of the image as a distributable artefact and the procedure for building it reproducibly from a single-file recipe. Once the whole chain was in place — put the recipe (the Dockerfile) in Git, distribute the artefact (the image) through a registry, run it identically anywhere — the environment became, for the first time, an object under version control. It is an example of a technology whose value lay not in the parts but in the way the parts were combined.

In what follows we first define precisely what a container is (§3), then derive the Dockerfile conventions as theorems (§4, §5), then look at what Kubernetes does in a world where one machine is no longer enough (§6), and finally evaluate the effects of redundancy, autoscaling and zero-downtime updates numerically (§7).

2. Preliminaries: the runtime environment as a 4-tuple

To make the discussion precise, we regard “the environment in which a program runs” as the following 4-tuple.

$$E = (K,\ R,\ V,\ Q)$$

• $K$: the kernel. The implementation and ABI of the system calls, and the CPU architecture.
• $R$: the root file system. Which shared libraries live in /usr/lib, what is written in /etc.
• $V$: the visible scope. The list of processes, network interfaces, mounts, host name and user-ID mapping that the process can see.
• $Q$: the available resources. CPU time, memory, number of PIDs, I/O bandwidth.

“It works on my machine” arises because $R$, $V$ and $Q$ differ from person to person. A dynamically linked executable, for instance, searches $R$ for its shared libraries at startup.

$ ldd ./myapp
    linux-vdso.so.1 (0x00007ffd8b7f0000)
    libjpeg.so.62 => not found
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2e4c000000)

That single not found line is what costs half a day.

Note

A virtual machine duplicates everything, $K$ included. A container duplicates only $R$, restricts $V$ and $Q$, and shares $K$ with the host. This one sentence explains both the lightness of containers and the limits of their isolation, discussed below.

3. What a container really is: namespaces and cgroups

Definition 3.1（Container）

A container on Linux is a collection of one or more processes that have been given the following three things.

1. Namespaces: the view of resources such as process IDs, mounts, networking, host name, inter-process communication and user IDs is separated from that of the host and of other containers. In the notation of §2 this is a restriction of $V$.
2. cgroups (control groups): upper bounds are imposed on the use of CPU, memory, number of PIDs and so on. This is a restriction of $Q$.
3. A root file system: by means of pivot_root or similar, the processes see a file system prepared for the container as their root. This is a replacement of $R$.

The kernel $K$ is the host’s, used as it is. Consequently a process inside a container is, seen from the host, an ordinary process appearing in ps.

Namespaces are independent by kind, so one can separate only what needs separating. The principal ones are these.

Namespace	What is separated	What happens without it
mount	the set of mount points	files of another container are visible
PID	the process-ID space	one can kill processes of another container
network	interfaces, routing, ports	two containers cannot both use port 80
IPC	shared memory, semaphores	name collisions occur
UTS	host name, domain name	everything has the same host name
user	mapping of user and group IDs	root inside the container is root on the host
cgroup	the view of the cgroup hierarchy	one can see outside one’s own resource limits

The effect of the PID namespace is an easy illustration. Running ps inside a container shows one’s own process first, as PID 1. Looking at the same process from the host, the PID may be 24187. The same process carries different numbers depending on where you look at it — that is the content of “isolation”, and it is a rather modest contraption for something called virtualization.

cgroups, by contrast, are about upper bounds. If a memory limit of 512 MiB is set in cgroup v2, then when the processes in that group reach the limit the kernel attempts reclamation and, if that is not enough, stops a process in the group with the OOM killer. A Kubernetes Pod that suddenly becomes OOMKilled is the result of this mechanism at work.

Virtual machines versus containers: how many kernels are running

The difference between the two halves of the figure comes down to the number of kernels, drawn in the accent colour. With virtual machines, each additional machine brings an additional kernel, and that kernel needs memory and boot time of its own. With containers the kernel count does not grow. This difference shows up as a difference in practical properties.

	Virtual machine	Container
Kernel	one per guest	one on the host (shared)
Boot time	tens of seconds (including OS boot)	tens to hundreds of milliseconds
Image size	several GB	tens to hundreds of MB
Number per host	a few to a dozen or so	dozens to hundreds
Strength of isolation	strong isolation by the hypervisor	isolation by kernel features
Running a different OS	possible (Windows on Linux)	not possible (same kernel only)

“Starts fast” and “many at once” are the preconditions for the autoscaling and rolling updates we look at later. A unit that takes tens of seconds to start cannot follow a sudden surge in load, and cannot be deployed dozens of times a day.

Remark 3.2

Sharing the kernel means that a kernel vulnerability is directly a way through the isolation. Containers were not designed to be the primary barrier in a multi-tenant environment. In practice, observe at least the following.

• Do not run as root inside the container (specify USER in the Dockerfile; see the example in §4).
• Enable the user namespace, so that uid 0 inside the container maps to an unprivileged uid on the host.
• Avoid --privileged, unnecessary capabilities, and mounts of the host file system.
• If untrusted code must run, consider execution environments that interpose a virtualization boundary, such as gVisor, Kata Containers or Firecracker.

Note also that “Docker” is no longer the only implementation. The image and runtime specifications are standardized under the OCI (Open Container Initiative), and since v1.24 Kubernetes does not call Docker Engine directly but runs containers through containerd or CRI-O, which implement the CRI. It is thanks to this standardization that an image built from a Dockerfile simply runs.

4. Images and layers: two consequences of immutability

Definition 4.1（Image and layer）

An image is a pair consisting of an ordered sequence of layers $L_1, L_2, \ldots, L_n$ together with metadata describing the startup configuration (environment variables, working directory, default command, and so on). Each layer is immutable data, named uniquely by its content, recording a difference (addition, modification or deletion of files) against the file system produced by the layers before it.

When a container is started, the runtime stacks $L_1$ through $L_n$ (a union file system) and places one thin writable layer on top. Deleting a file in $L_i$ does not erase the actual data in $L_{i-1}$ or earlier; it only records a mark in $L_i$ (a whiteout) saying “this path has been deleted”.

That layers are immutable and content-addressed — the same idea by which Git names objects by the hash of their content (content-addressable store（Definition 2.1）［Git］) — yields the pleasant property that an identical layer need be transferred and stored only once. If ten images use the same base image, the base exists physically once on the host. On the other hand, immutability also creates two constraints. Both bear directly on how Dockerfiles should be written, so we state them as theorems.

4.1. The cache only helps on a prefix

Let the instructions of a Dockerfile be $I_1, \ldots, I_n$. At build time a cache key is computed for each instruction:

$$k_0 = (\text{digest of the base image}), \qquad k_i = H\bigl(k_{i-1},\ \sigma(I_i)\bigr)$$

Here $H$ is a cryptographic hash function and $\sigma(I_i)$ is a normalized representation of the instruction: for RUN it is the command string itself, while for COPY and ADD it includes the command string together with the hash of the contents of the files being copied.

Proposition 4.2（The prefix property of the build cache）

Suppose we build using the key computation above, under the rule that “the $i$-th instruction hits the cache exactly when all instructions up to the $(i-1)$-st hit and a layer corresponding to $k_i$ exists locally”. Assume $H$ is collision-free.

Then the set of instructions that hit the cache is a prefix $\{1, 2, \ldots, m\}$ for some $m$. Moreover, if $j$ is the least index at which $\sigma(I_j)$ has changed since the previous build, then $m \le j - 1$, and $I_j, I_{j+1}, \ldots, I_n$ are all re-executed.

Proof（Proposition 4.2）

The first half follows from the rule itself. If the $i$-th instruction hits, then by the first clause of the rule the $(i-1)$-st also hits. Repeating this, all of $1, \ldots, i-1$ hit. Hence the set of hitting indices is closed downwards, that is, a prefix. It suffices to take $m$ to be the largest hitting index (with $m = 0$ if none hits).

For the second half: for $i < j$ the value of $\sigma(I_i)$ has not changed, so as long as $k_0$ is the same, the recurrence for the keys gives inductively that $k_i$ also has its previous value. The layers corresponding to these keys therefore remain locally available and can hit.

For $i \ge j$, on the other hand, the second argument of $k_j = H(k_{j-1}, \sigma(I_j))$ has changed. Since $H$ is assumed collision-free, $k_j$ differs from its previous value. Then the first argument of $k_{j+1} = H(k_j, \sigma(I_{j+1}))$ has changed too, so it also differs, and inductively $k_i \ne k_i^{\text{prev}}$ for all $i \ge j$. No layer exists for a key that has never been generated before, so instructions from the $j$-th on all miss the cache. Hence $m \le j-1$, and combining with the prefix property of the first half, everything from $I_j$ onwards is re-executed.

∎

The practical meaning of this proposition is a single rule: put the instructions that change rarely first and the ones that change often last. Source code changes every time, while the list of dependencies changes a few times a month, so the ordering settles itself.

FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install --no-cache-dir -r requirements.txt
CMD ["python", "-m", "myapp"]

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "-m", "myapp"]

Example 4.3（Instruction order decides 30 minutes a day）

Compare the two Dockerfiles above. Suppose pip install takes 90 seconds and each COPY takes 1 second, that during development we build 20 times a day, and that in 19 of those builds only the source code has changed while in one requirements.txt has changed as well.

In the first Dockerfile, COPY . . is the third instruction. Editing a single character of source code changes $\sigma(I_3)$, so by Proposition 4.2 everything from $I_3$ on is re-executed. That is, pip install runs every time. Per day this is

$$20 \times (1 + 90) = 1820\ \text{s} \approx 30\ \text{minutes}$$

In the second, a source change affects only COPY . . (the fifth instruction) and later. Neither $\sigma(I_3)$ (COPY requirements.txt .) nor $\sigma(I_4)$ (RUN pip install ...) changes, so the cache hits through $I_4$. Per day this is

$$19 \times 1 + 1 \times (1 + 90 + 1) = 111\ \text{s} \approx 2\ \text{minutes}$$

The difference is 28 minutes a day, about nine and a half hours over 20 working days. For a change consisting of swapping two lines in a Dockerfile, that is a large return.

Remark 4.4

Proposition 4.2 is a statement about the classical model of sequential layer caching. BuildKit, the current default builder, analyses the dependencies between instructions as a directed acyclic graph, so independent stages of a multi-stage build are built in parallel and a change in one stage does not invalidate the cache of the other. However, within a single stage the prefix property holds exactly as stated, so the ordering convention above is unchanged.

One more point: the fact that $\sigma(\texttt{RUN ...})$ is determined by the command string alone has a side effect. RUN apt-get update has an unchanging command string, so it keeps hitting the cache forever even after the upstream package index has been updated. If one then adds RUN apt-get install -y foo afterwards, it consults the stale index and fails. Combining apt-get update and apt-get install into a single RUN is how this trap is avoided.

4.2. Deleting does not shrink the image

Proposition 4.5（Non-commutativity of layer deletion）

Measure the total size of an image by the sum $S = \sum_{i=1}^{n} |L_i|$ of the actual data stored in each layer. Suppose a file $f$ of size $s > 0$ is added in layer $L_i$, and that $f$ is deleted by an instruction in a layer $L_j$ with $j > i$.

Then the image size $S_{\text{del}}$ with the deletion and the size $S_{\text{keep}}$ without the deletion instruction satisfy

$$S_{\text{del}} = S_{\text{keep}} + w \ \ge\ S_{\text{keep}}$$

where $w \ge 0$ is the size of the whiteout record. In particular, compared with the size $S_{\text{never}} = S_{\text{keep}} - s$ obtained by never creating $f$ in a persistent layer at all, we have $S_{\text{del}} - S_{\text{never}} \ge s$: the deletion instruction does not make the image smaller by $s$ bytes.

Proof（Proposition 4.5）

By Definition 4.1, layers are immutable. At the time $L_j$ is built, $L_i$ is already fixed and named by its content hash. Rewriting the contents of $L_i$ would change its name and produce a different layer, so by definition a later instruction cannot modify the contents of $L_i$. Therefore $|L_i|$ still contains the $s$ bytes of $f$ and is unchanged.

Now consider $L_j$. In a union file system, deletion is expressed by placing a whiteout record in an upper layer (Definition 4.1). So $L_j$ gains a record representing the deletion of $f$, and writing $w \ge 0$ for its size, $|L_j|$ increases by $w$. No other layer changes.

Hence $S_{\text{del}} = S_{\text{keep}} + w$, and $w \ge 0$ gives $S_{\text{del}} \ge S_{\text{keep}}$. The last claim follows by substituting $S_{\text{never}} = S_{\text{keep}} - s$, which yields $S_{\text{del}} - S_{\text{never}} = s + w \ge s$.

∎

In other words, one must not think of temporary build files as something to “delete later”. There are two remedies.

• Create and delete within the same RUN. One RUN becomes one layer, so a file removed before the layer is finalized is never recorded.
• Use a multi-stage build. Discard the intermediates produced in a build stage and copy only the artefact into a new stage.

FROM golang:1.22 AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/server ./cmd/server

FROM gcr.io/distroless/static-debian12
COPY --from=builder /out/server /server
USER 65532:65532
ENTRYPOINT ["/server"]

Example 4.6（Turning 1.2 GB into 30 MB）

Suppose that in the multi-stage Dockerfile above the sizes are as follows: the golang:1.22 base is 800 MB, the dependencies fetched by go mod download are 350 MB, the intermediates produced by the build are 30 MB, the statically linked binary produced is 30 MB, and gcr.io/distroless/static-debian12 is 2 MB.

If everything were done in a single stage with a final RUN rm -rf /go/pkg /src, then by Proposition 4.5 that rm reduces nothing. The total size would be

$$800 + 350 + 30 + 30 + w \approx 1210\ \text{MB}$$

With the multi-stage build, the final image consists only of the base of the second stage and the binary brought in by COPY --from=builder:

$$2 + 30 = 32\ \text{MB}$$

about one thirty-eighth. Distributing to 50 nodes, the transfer volume drops from 60 GB to 1.6 GB, and the wait when autoscaling adds nodes gets shorter. In addition, since the final image contains no compiler, no shell and no package manager, there is a security benefit: what an intruder can do is drastically reduced.

5. The Dockerfile: what it means to turn an environment into code

So far we have seen that most of the conventions for writing Dockerfiles follow from Proposition 4.2 and Proposition 4.5. The rest concerns what “turning it into code” means.

The Dockerfile lives in the same repository as the source code and goes through the same review process. As a result, “who changed the production environment, when and why” is recorded in the Git history (see the commit graph and the ancestor relation（Definition 4.1）［Git］ in Version Control (Git)). The decisive difference from a setup document is that a Dockerfile is executed. What is executed breaks the build when it is wrong, so it does not rot.

One must nonetheless be careful: “the same Dockerfile” does not mean “the same image”. The following three things break reproducibility.

1. Mutable tags. The image that FROM python:3.12-slim refers to changes its contents when the upstream is updated. If exact reproducibility is required, pin it by digest: FROM python:3.12-slim@sha256:....
2. Fetching over the network. apt-get install and pip install depend on the upstream state at the moment of execution. The more explicitly versions are pinned (pip install -r requirements.lock), the more reproducible the build.
3. Processing that depends on build time. Embedding timestamps or random numbers produces a different image every time.

Here is one more pitfall that everyone eventually falls into. CMD and ENTRYPOINT have two notations.

CMD python -m myapp
CMD ["python", "-m", "myapp"]

The upper, shell form, expands to /bin/sh -c "python -m myapp", so the process that becomes PID 1 is the shell. Most sh implementations do not forward a received SIGTERM to their children. Consequently, an attempt to stop the container does not cause the application to begin its shutdown, and after the grace period (30 seconds by default in Kubernetes) it is killed with SIGKILL. The symptom of in-flight requests being dropped on every deploy is often caused by exactly this. With the lower, exec form, the application itself becomes PID 1 and receives signals directly. The difference is only a pair of brackets, but it decides whether zero-downtime deployment is possible.

Let us also settle the handling of configuration values. The principle is that values that differ per environment, such as database endpoints and log levels, are not baked into the image but passed as environment variables (the third factor of the so-called Twelve-Factor App). This way, development, staging and production run exactly the same image. Being able to ship the very image that was tested is the greatest source of confidence in container-based deployment.

6. When one machine is not enough: the design principles of Kubernetes

If there is one container and one host, docker run is enough. In real operations, however, the following requirements arise at once.

• Make the service redundant across three machines, so that it does not stop when one fails.
• Run three machines at night and twenty at the daytime peak.
• Place containers on nodes that have free memory.
• Roll out a new version gradually, without downtime.
• Reach the service by name, without knowing which node it runs on.

Doing this by hand is impossible. Orchestration means automating this family of decisions, and Kubernetes is one implementation of it. Kubernetes was released by Google in 2014 and incorporates into its design the operational experience of Borg, Google’s internal cluster management system (for Borg see the paper of Verma et al.; for how the design philosophy was inherited see the article by Burns et al., both in the references).

6.1. From imperative to declarative

There is only one central design concept in Kubernetes.

Definition 6.1（Declarative API and reconciliation loop）

The user registers with the API not “which operations to perform in which order” but “what state things should be in” (the desired state). The system keeps running a reconciliation loop. On each iteration it

1. reads the desired state $d$,
2. observes the current state $s$,
3. performs operations that reduce the difference between $d$ and $s$.

These three steps do not depend on the history of how $s$ came about; they are determined by $d$ and $s$ at that moment alone. This property is called being level-triggered.

For contrast, consider imperative operation: a scheme in which one sends the command “add one Pod”. Under this scheme, if the command is lost on the way the count stays short, and if it arrives twice the count is too high. In the middle of an incident, both happen routinely. With a scheme that registers the state “there should be three Pods”, by contrast, whether the command is lost or duplicated, the next loop corrects things on the basis of the currently observed reality.

flowchart LR
U["User: declares the desired state"] --> API["API server / persisted in etcd"]
API --> C["Controller: compares desired with current"]
C -->|operations closing the gap| K["Scheduler and kubelet"]
K --> R["Current state: the Pods actually running"]
R -->|observation| C

The reconciliation loop: continually closing the gap between the desired and the current state

Proposition 6.2（Finite convergence of the reconciliation loop）

Let $\mathcal{O}$ be a finite set of target objects with $|\mathcal{O}| = M$, and for a desired state $d$ and an observed state $s_t$ at time $t$ consider the number of mismatched objects

$$D(s_t) = \#\{o \in \mathcal{O} : s_t(o) \ne d(o)\}$$

Assume the following.

• (A1) The operations of each loop are determined by $(d, s_t)$ alone (the level-triggered property of Definition 6.1).
• (A2) If $D(s_t) > 0$, then one iteration resolves at least one mismatch, and no object that already matched becomes mismatched.
• (A3) No external change of state (node failure, or a rewrite of $d$) occurs while the loop runs.

Then $D(s_{t+1}) \le D(s_t) - 1$ as long as $D(s_t) > 0$, and $s = d$ is reached in at most $D(s_0) \le M$ iterations. After that, by the second half of (A2) the state does not change and no operations are performed (idempotence).

Furthermore, under (A1) the input of a loop is only the current $(d, s_t)$, so the convergence claim is unaffected by dropping past event notifications or by receiving the same notification several times.

Proof（Proposition 6.2）

$D$ is a function with values in the non-negative integers. By the first half of (A2), when $D(s_t) > 0$ at least one mismatch is removed. By the second half of (A2), no new mismatch appears. By (A3), neither $d$ nor $s$ changes in the meantime through external causes. Hence $D(s_{t+1}) \le D(s_t) - 1$.

That $D(s_0) \le M$ follows from the definition of $D$ (the mismatched objects form a subset of $\mathcal{O}$). A sequence of non-negative integers $D(s_0) > D(s_1) > \cdots$ cannot decrease forever, so after at most $D(s_0)$ iterations $D(s_T) = 0$, that is, $s_T = d$.

Once $s_T = d$ is reached, $D = 0$, so there is no mismatch for (A2) to resolve and the set of operations is empty. Running the same loop any number of times therefore leaves the state unchanged. This is idempotence.

Finally we check the last claim. By (A1), the operations at time $t$ are a function of $(d, s_t)$. Dropped or duplicated notifications only change when a loop is triggered; they do not change the input $(d, s_t)$ of a loop that is triggered. Hence, as long as loops continue to be triggered at finite intervals, the argument above applies unchanged and convergence holds.

∎

Remark 6.3

In real operation, the assumptions that break most easily are (A2) and (A3).

The typical failure of (A2) is a container that crashes right after starting. The controller judges that a Pod is missing and creates one, the Pod fails to start, and $D$ does not decrease. The loop keeps running, and the retry interval grows exponentially under the name CrashLoopBackOff. What Proposition 6.2 guarantees is that “if the gap-closing operations actually work, convergence follows”, not that “any configuration will eventually start working”.

As for (A3), in a cluster it is always violated to begin with. Nodes fail and users update manifests. The realistic statement is therefore not “it converges and finishes” but “it converges once the disturbances stop”. When one node fails, $D$ jumps, and the loop again pushes it down towards zero. What is called self-healing is exactly this pushing down.

6.2. The principal objects

Definition 6.4（Pod）

A Pod is the smallest unit of deployment in Kubernetes: a group of one or more containers. Containers in the same Pod share the network namespace (hence the IP address and localhost) and storage volumes, and are always placed on the same node. A Pod is a disposable entity: when it is restarted, both its IP address and its name change.

The point that Pods are disposable is important. It is precisely why mechanisms for holding state on top of them are needed. Here are the principal objects.

Object	What it declares as the desired state
Pod	run one set of these containers
ReplicaSet	keep $N$ copies of this Pod
Deployment	keep $N$ copies of this Pod, updating them by a specified strategy
Service	make the Pods carrying this label reachable through a fixed name and virtual IP
Ingress	route external HTTP traffic to Services
ConfigMap / Secret	inject configuration values and secrets into Pods
StatefulSet	keep $N$ copies, each with a stable name and a persistent volume
Job / CronJob	run this work to completion once, or periodically
HorizontalPodAutoscaler	adjust the replica count so that a metric reaches its target

When a user writes a Deployment, the Deployment controller creates a ReplicaSet, the ReplicaSet controller creates Pods, the scheduler assigns nodes to Pods, and the kubelet on that node calls the container runtime to start the containers. The point is that each stage runs its own reconciliation loop independently: the party responsible at each stage looks only at its own desired state and current state.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: web
      containers:
        - name: web
          image: registry.example.com/web@sha256:0b1f2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "200m"
              memory: "256Mi"
            limits:
              memory: "512Mi"
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 2

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080

Example 6.5（What happens when one node fails）

Let us follow what happens right after the manifests above are applied. Since replicas: 3, the ReplicaSet controller creates three Pods, and topologySpreadConstraints places them on three distinct nodes (nodes with different kubernetes.io/hostname). The setting maxSkew: 1 requires the difference in Pod counts between nodes to stay within one.

Now suppose the hardware of one of those nodes fails.

1. The periodic reports from that node’s kubelet stop arriving. By default, after some time without a response (on the order of tens of seconds) the node is recorded as NotReady.
2. After a further interval, the Pods on that node are marked for deletion. At this point the observed state of the ReplicaSet is “2 running”, so $D = 1$ (in the notation of Proposition 6.2).
3. To close the gap, the ReplicaSet controller creates one Pod. The scheduler picks, among the remaining nodes, one with enough room to satisfy the requests of cpu: 200m and memory: 256Mi and that does not violate maxSkew.
4. The container of the new Pod starts, and once /healthz answers the readiness probe issued every 2 seconds, the Pod is added to the Service’s endpoint list and traffic begins to flow. $D$ returns to $0$.

Nobody lifted a finger. All the human did was write “there should be three”.

Note that without a readinessProbe, the Service starts routing traffic from the instant the container’s process starts. If the application takes 10 seconds to initialize, errors are returned for those 10 seconds. To be free of downtime, one needs a way to declare readiness.

One remark about resources. requests is the reservation the scheduler uses to decide placement, while limits is the upper bound configured in the cgroup. For memory an upper bound is written because exceeding it results in OOMKilled, but a CPU limit does not kill on excess — it throttles — and for latency-sensitive services this can cause unintended latency degradation. That is why no CPU limit is written here.

A Service is the mechanism that reconciles the disposability of Pods with the wish to call a service by a stable name from elsewhere. The cluster’s DNS (the domain name hierarchy and authoritative servers（Definition 6.1）［ネットワーク（TCP/IP）］) resolves names such as web.default.svc.cluster.local to the Service’s virtual IP, and a mechanism on each node forwards packets addressed to that virtual IP to the actual Pod IPs. Understanding which layer performs what in this forwarding becomes much clearer with the material of Networking (TCP/IP), especially the 4-tuple identifying a connection（Definition 5.1）［ネットワーク（TCP/IP）］.

Whether to run stateful software such as a database on Kubernetes is a question to be judged carefully. It is possible with StatefulSets and persistent volumes, but the hard parts of operating a database — backups, failover, schema changes — remain (see Foundations of Database Design). Using a cloud provider’s managed database and putting only stateless applications on Kubernetes is still a strong option (in the terms of the service models (IaaS, PaaS, SaaS)（Definition 2.2）［クラウドコンピューティング］ in Cloud Computing (AWS, GCP), it is a choice of how much operational responsibility to hand to the provider).

7. Scalability and fault tolerance, quantified

One often hears “redundancy makes things stronger” and “it scales automatically”, but how much stronger can be computed. The definition of availability itself, and the composition rules for series and parallel arrangements, are collected in availability of series and parallel configurations（Proposition 6.2）［クラウドコンピューティング］. Below we apply them to the placement of Kubernetes replicas.

7.1. Redundancy and availability

Theorem 7.1（Redundancy under independent failures）

Suppose a service is provided by $N$ replicas, that each replica is unavailable at a given moment with probability $p$ (with $0 < p < 1$), and that the failures of the replicas are mutually independent. If the service is available whenever at least one replica is available, then its availability is

$$A(N) = 1 - p^{N}$$

Consequently the number of replicas needed to meet a target availability $A^{*}$ (with $0 < A^{*} < 1$) is the least integer satisfying

$$N \ \ge\ \frac{\ln(1 - A^{*})}{\ln p}$$

Proof（Theorem 7.1）

The service is unavailable exactly when all $N$ replicas are unavailable. Writing $F_i$ for the event that replica $i$ is unavailable, independence gives

$$\Pr\Bigl[\bigcap_{i=1}^{N} F_i\Bigr] = \prod_{i=1}^{N} \Pr[F_i] = p^{N}$$

Hence the availability is $A(N) = 1 - p^{N}$.

Now for the required number of replicas. The condition $1 - p^{N} \ge A^{*}$ is equivalent to $p^{N} \le 1 - A^{*}$. Both sides are positive, so taking natural logarithms gives $N \ln p \le \ln(1 - A^{*})$. Since $0 < p < 1$ we have $\ln p < 0$, so dividing by $\ln p$ reverses the inequality and yields $N \ge \ln(1-A^{*}) / \ln p$.

∎

Example 7.2（How many minutes of downtime one extra replica removes）

Suppose a single replica has availability 99 %, that is, $p = 0.01$. Taking 30 days to be 43,200 minutes, we compute the downtime.

• $N = 1$: $A = 0.99$, downtime per month $43{,}200 \times 0.01 = 432$ minutes (about 7.2 hours).
• $N = 2$: $A = 1 - 10^{-4} = 0.9999$, downtime $43{,}200 \times 10^{-4} = 4.32$ minutes.
• $N = 3$: $A = 1 - 10^{-6} = 0.999999$, downtime $43{,}200 \times 10^{-6} \approx 2.6$ seconds.

Going from one replica to two turns 7 hours of monthly downtime into 4 minutes. In the other direction, computing the number required for a target $A^{*} = 0.99999$ (five nines) via Theorem 7.1 gives

$$N \ \ge\ \frac{\ln(10^{-5})}{\ln(10^{-2})} = \frac{-11.5129}{-4.6052} = 2.5$$

so $N = 3$.

Everything so far depends entirely on the assumption of independence. Let us see what happens when it is dropped.

Corollary 7.3（The ceiling on availability imposed by correlated failure）

In addition to the setting of Theorem 7.1, suppose all $N$ replicas are placed on the same node. Let $q$ (with $0 < q < 1$) be the probability that the whole node is unavailable, and suppose that, conditionally on the node being healthy, each replica is unavailable independently with probability $p$. Then

$$A(N) = 1 - \bigl(q + (1-q)\,p^{N}\bigr) \ \le\ 1 - q$$

so that increasing the number of replicas without bound never raises the availability above $1 - q$.

Proof（Corollary 7.3）

Split the event that the service is unavailable according to whether the node has failed. If the node has failed (probability $q$), all replicas on it are unavailable, so the service is unavailable. If the node is healthy (probability $1-q$), each replica is conditionally unavailable independently with probability $p$, so by the same computation as in the proof of Theorem 7.1 the conditional probability that all of them fail is $p^{N}$. By the law of total probability,

$$\Pr[\text{unavailable}] = q \cdot 1 + (1-q)\, p^{N} = q + (1-q)p^{N}$$

Since $(1-q)p^{N} \ge 0$ we get $\Pr[\text{unavailable}] \ge q$, that is, $A(N) \le 1-q$. As $N \to \infty$ we have $p^{N} \to 0$ (because $0<p<1$), so $A(N) \to 1-q$ and the bound cannot be improved.

∎

Concretely, suppose a node’s monthly failure probability is $q = 10^{-3}$. In Example 7.2 we computed that three replicas give five nines, but if those three sit on the same node, the availability is capped at $1 - 10^{-3} = 0.999$. That is 43.2 minutes of downtime per month, and no number of replicas changes it. The benefit of redundancy is determined by where the correlations in failure lie.

The topologySpreadConstraints of Example 6.5 is precisely the setting that breaks this correlation. Changing topologyKey from the node to the availability zone (topology.kubernetes.io/zone) also guards against the coarser correlation of an entire zone failing. At the same time, communication across zones incurs additional latency. The right order is to decide what one wants to be independent, and only then write the placement constraints.

7.2. Horizontal autoscaling

Proposition 7.4（One-step convergence of the HorizontalPodAutoscaler）

Let the current replica count be $n \ge 1$, the average value of the metric per Pod be $m > 0$, and the target be $T > 0$, and update the replica count by the same rule as the Kubernetes HorizontalPodAutoscaler:

$$n' = \Bigl\lceil\, n \cdot \frac{m}{T} \,\Bigr\rceil, \qquad \text{except that } n' = n \text{ when } \Bigl|\frac{m}{T} - 1\Bigr| \le \tau$$

where $\tau$ is a tolerance, by default $0.1$.

Now suppose the total load on the system, $\lambda > 0$, is constant, is distributed evenly across all Pods, and that the per-Pod metric can be written $m = \lambda / n$. Then, whatever the value of $n$, a single update gives

$$n' = \Bigl\lceil \frac{\lambda}{T} \Bigr\rceil =: n^{*}$$

Moreover $n^{*}$ is a fixed point: as long as $\lambda / (n^{*} T) \ge 1 - \tau$ holds, no change in the downward direction occurs either.

Proof（Proposition 7.4）

Substitute $m = \lambda/n$ into the update formula:

$$n \cdot \frac{m}{T} = n \cdot \frac{\lambda/n}{T} = \frac{\lambda}{T}$$

so $n$ cancels. Hence $n' = \lceil \lambda/T \rceil = n^{*}$, which does not depend on the replica count $n$ before the update.

Next we show $n^{*}$ is a fixed point. When the replica count is $n^{*}$, the metric is $m^{*} = \lambda/n^{*}$. By definition of the ceiling, $n^{*} \ge \lambda/T$, so $m^{*} = \lambda/n^{*} \le T$, that is, the ratio $r := m^{*}/T$ satisfies $r \le 1$. The assumption $\lambda/(n^{*}T) \ge 1-\tau$ says exactly that $r \ge 1-\tau$, so $1-\tau \le r \le 1$, that is, $|r - 1| \le \tau$. This is precisely the tolerance condition of the update rule, so $n' = n^{*}$ and no change occurs.

∎

Example 7.5（An actual autoscaling computation）

Take the target to be “CPU usage per Pod equal to 60 % of requests”, and suppose 6 Pods are currently running at an average of 85 %. By the update rule of Proposition 7.4,

$$n' = \Bigl\lceil 6 \times \frac{85}{60} \Bigr\rceil = \lceil 8.5 \rceil = 9$$

After the increase to 9, the average utilization, assuming the total load is unchanged, is

$$\frac{6 \times 85}{9} \approx 56.7\ \%$$

The ratio is $56.7/60 = 0.944$, and $|0.944 - 1| = 0.056 \le 0.1$, so it falls within the tolerance and no further change occurs. The system settles in a single adjustment, with no oscillation between growing and shrinking.

It is worth seeing what would happen without the tolerance. Applying the update rule in the state of 9 Pods at 56.7 % gives $\lceil 9 \times 56.7/60 \rceil = \lceil 8.5 \rceil = 9$, so in this example there happens to be no oscillation. Because of the rounding up in the ceiling, however, it is easy to construct settings where a ratio that stays slightly below 1 makes the count grow and shrink repeatedly. The tolerance $\tau$ and the stabilization window for scaling down (5 minutes by default) exist to stop oscillations of this kind.

Remark 7.6

The crux of Proposition 7.4 is the assumption $m = \lambda/n$, that is, “adding Pods reduces the load per machine in inverse proportion” (for the response time when a load is spread evenly across $n$ machines see response time when load is spread evenly across n machines（Corollary 5.3）［クラウドコンピューティング］). The standard case in which this assumption breaks is a saturated downstream database. Adding Pods not only fails to shorten the waiting time; the number of connections grows, the database gets even slower, CPU utilization does not fall, and so the HPA adds still more Pods — a vicious circle. Autoscaling does not remove the rate-limiting stage. Determine where the bottleneck is before configuring it.

7.3. How long a zero-downtime update takes

The behaviour of a Deployment’s RollingUpdate is determined by two numbers. With $N$ the desired replica count, maxUnavailable $= u$ means “keep at least $N - u$ Pods available” and maxSurge $= s$ means “keep the total number of Pods at most $N + s$” (both default to 25 %).

Proposition 7.7（Lower bound on the duration of a rolling update）

Suppose a Deployment with $N$ replicas is rolled out with maxUnavailable $= u \ge 0$ and maxSurge $= s \ge 0$ (with $u + s \ge 1$). Assume the following.

• Throughout the update, the number $R$ of available Pods satisfies $R \ge N - u$ and the total number $P$ of Pods satisfies $P \le N + s$.
• A new Pod requires at least a time $t > 0$ between being created and becoming available.
• At the completion of the update, $N$ new Pods are available.

Then the duration $\Theta$ of the update satisfies

$$\Theta \ \ge\ \frac{N t}{u + s}$$

Proof（Proposition 7.7）

Let $g(x) = P(x) - R(x)$ be the number of Pods at time $x$ that have been created but are not yet available. By assumption $P(x) \le N+s$ and $R(x) \ge N-u$, so

$$g(x) \le (N+s) - (N-u) = u + s$$

holds at all times. That is, at most $u+s$ Pods can be in preparation simultaneously.

On the other hand, by the completion of the update $N$ new Pods have been created, and each of them stays in the “created but not available” state for at least $t$ (the second assumption). Therefore the integral of $g$ over the update interval $[0, \Theta]$, counting only the new Pods, satisfies

$$\int_{0}^{\Theta} g(x)\, dx \ \ge\ N t$$

On the other hand, from $g(x) \le u+s$,

$$\int_{0}^{\Theta} g(x)\, dx \ \le\ (u+s)\,\Theta$$

Combining the two gives $(u+s)\Theta \ge Nt$, that is, $\Theta \ge Nt/(u+s)$.

∎

Example 7.8（Why the safe setting takes 20 minutes）

Consider a service with $N = 60$ for which a new Pod takes $t = 20$ seconds to become available.

• With the defaults $u = s = 25\ \%$, that is $u = s = 15$: $\Theta \ge 60 \times 20 / 30 = 40$ seconds.
• With “we absolutely must not lose capacity”, $u = 0$ and $s = 1$: $\Theta \ge 60 \times 20 / 1 = 1200$ seconds, that is 20 minutes.

A factor of 30 for the same service. The setting $u = 0, s = 1$ — “always keep 60 Pods available and replace them one at a time” — is intuitively the safest, but if an update takes 20 minutes, so does an emergency rollback. During an incident that is a heavy constraint.

If capacity must not be reduced, it is better to keep $u = 0$ and increase $s$ instead (for example $s = 10$ gives $\Theta \ge 120$ seconds), at the price of 10 Pods’ worth of extra compute during the update. Proposition 7.7 is a lower bound, so the actual duration will be longer. Leave margin when choosing the settings.

8. Exercises

Exercise 8.1Easy

Answer the following about this Dockerfile. Assume npm ci takes 90 seconds and each COPY takes 1 second.

FROM node:22-slim
WORKDIR /app
COPY . .
RUN npm ci
RUN npm run build
CMD ["node", "dist/main.js"]

(1) When one file under src/ is edited and the image is rebuilt, which instructions hit the cache? Explain using Proposition 4.2.

(2) Rewrite the Dockerfile so as to minimize the daily time spent waiting for npm ci, for a developer who builds 20 times a day editing only src/.

(3) With the rewritten Dockerfile, what kind of change causes npm ci to be re-executed?

Solution

(1) COPY . . is the third instruction, and $\sigma(I_3)$ includes the hash of the contents of the copied files. Editing src/ changes $\sigma(I_3)$, so by the second half of Proposition 4.2 we get $m \le 2$, and everything from $I_3$ on (COPY . ., npm ci, npm run build) is re-executed. Only $I_1$ and $I_2$, corresponding to FROM and WORKDIR, hit.

(2) Copy only the dependency manifests first.

FROM node:22-slim
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
CMD ["node", "dist/main.js"]

Editing src/ changes neither $\sigma(I_3)$ (COPY package.json package-lock.json ./) nor $\sigma(I_4)$ (RUN npm ci), so by Proposition 4.2 the cache hits through $I_4$ and npm ci is not re-executed. The daily waiting time for npm ci drops from $20 \times 90 = 1800$ seconds to zero.

(3) When the contents of package.json or package-lock.json change. Then $\sigma(I_3)$ changes, so everything from $I_3$ on is re-executed and npm ci runs. Reinstalling when dependencies are added or updated is the correct behaviour, and exactly what we intended.

Exercise 8.2Standard

Build the following Dockerfile. Assume the amounts of data newly written by each step are: base image 120 MB, instruction 2: 400 MB, instruction 3: 120 MB, the source unpacked by instruction 4: 350 MB, and the build intermediates plus the binary in instruction 5: 240 MB in total. The size of whiteout records is negligible.

FROM debian:12
RUN apt-get update && apt-get install -y build-essential
RUN curl -sL https://example.com/src.tar.gz -o /tmp/src.tar.gz
RUN tar xf /tmp/src.tar.gz -C /opt && rm /tmp/src.tar.gz
RUN make -C /opt/src && make -C /opt/src install

(1) Compute the size of the final image. How does the rm in instruction 4 affect it?

(2) Suppose the resulting binary is 40 MB and is all that is needed to run. Using debian:12-slim (80 MB) as the runtime base, write a Dockerfile that minimizes the final image, and compute its size.

Solution

(1) By Proposition 4.5, the rm /tmp/src.tar.gz of instruction 4 does not delete the 120 MB written in the layer of instruction 3. The layer of instruction 3 is immutable and named by its content hash, so no later instruction can rewrite it. The deletion appears only as a whiteout record in the layer of instruction 4, whose size we agreed to neglect. Hence

$$120 + 400 + 120 + 350 + 240 = 1230\ \text{MB}$$

The rm reduced the image by not a single byte; what it reduced is only “the files visible from a running container”.

(2) Use a multi-stage build.

FROM debian:12 AS builder
RUN apt-get update && apt-get install -y build-essential curl
RUN curl -sL https://example.com/src.tar.gz -o /tmp/src.tar.gz \
    && tar xf /tmp/src.tar.gz -C /opt \
    && rm /tmp/src.tar.gz \
    && make -C /opt/src \
    && make -C /opt/src install

FROM debian:12-slim
COPY --from=builder /usr/local/bin/myapp /usr/local/bin/myapp
USER 1000:1000
ENTRYPOINT ["/usr/local/bin/myapp"]

The final image contains only the base of the second stage and the copied binary, so

$$80 + 40 = 120\ \text{MB}$$

About one tenth of 1230 MB. Fetching, unpacking, building and deleting are combined into one RUN in the first stage in order to keep that stage’s layers small and save space in the intermediate cache; it has no effect on the size of the final image (the layers of the first stage are not part of it).

Exercise 8.3Standard

A single replica of a service has availability 99 % ($p = 0.01$), and the probability that a node carrying replicas fails entirely is $q = 0.002$ per month.

(1) Using Corollary 7.3, find the ceiling on availability when 4 replicas are all placed on a single node. Convert it into monthly downtime, taking 30 days to be 43,200 minutes.

(2) Now spread the 4 replicas over 4 distinct nodes, and suppose node failures and replica failures may all be treated as independent. Find the probability that a single replica is unavailable because either “its node fails” or “the replica itself fails”, and then find the availability of the whole service.

(3) From the result of (2), how many replicas are needed for a target of 99.999 %?

Solution

(1) By Corollary 7.3, $A \le 1 - q = 1 - 0.002 = 0.998$. The exact value is

$$A = 1 - \bigl(0.002 + 0.998 \times 0.01^{4}\bigr) = 1 - (0.002 + 9.98 \times 10^{-9}) \approx 0.998$$

where the replica-side contribution of order $10^{-8}$ is completely buried in the node failure term $2\times10^{-3}$. The downtime is $43{,}200 \times 0.002 = 86.4$ minutes. Going from 4 replicas to 40 does not change these 86.4 minutes.

(2) A single replica is available when its node is healthy and the replica itself is healthy, with probability $(1-q)(1-p) = 0.998 \times 0.99 = 0.98802$. So the probability that one replica is unavailable is

$$p' = 1 - 0.98802 = 0.01198$$

The four nodes are independent, so Theorem 7.1 applies directly:

$$A(4) = 1 - (0.01198)^{4} = 1 - 2.06 \times 10^{-8} \approx 0.99999998$$

The downtime is $43{,}200 \times 2.06\times10^{-8} \approx 0.0009$ minutes per month, that is, about 0.05 seconds. Compare this with the 86.4 minutes of (1). With the same 4 replicas, changing only where they are placed makes a difference of more than five orders of magnitude.

(3) Substitute $p' = 0.01198$ and $A^{*} = 0.99999$ into the formula of Theorem 7.1:

$$N \ \ge\ \frac{\ln(10^{-5})}{\ln(0.01198)} = \frac{-11.5129}{-4.4234} = 2.60$$

so $N = 3$. This holds under the assumption that the three nodes are independent, however; if the three sit in the same rack or on the same power supply, that correlation imposes a new ceiling. Re-instantiate the $q$ of Corollary 7.3 for each granularity of failure under consideration and evaluate accordingly.

Exercise 8.4Hard

A service has $N = 200$ replicas. A new Pod takes $t = 45$ seconds to become available. There are two operational requirements.

• During an update, keep the available replicas at all times at 90 % of $N$ or above.
• Complete an update (and a rollback) within 10 minutes.

(1) Write the conditions on $u$ (maxUnavailable) and $s$ (maxSurge) meeting the requirements, as inequalities.

(2) Find the $(u, s)$ minimizing the extra compute needed during the update.

(3) If $u = 0$ is required, what should $s$ be? With 10 Pods per node, how many extra nodes are then needed during the update?

Solution

(1) The capacity requirement says that the lower bound $N - u$ on the available count during the update is at least $0.9N$:

$$200 - u \ \ge\ 180 \quad\Longleftrightarrow\quad u \le 20$$

The time requirement makes it necessary that the lower bound of Proposition 7.7 be at most 600 seconds:

$$\frac{Nt}{u+s} = \frac{200 \times 45}{u+s} = \frac{9000}{u+s} \ \le\ 600 \quad\Longleftrightarrow\quad u + s \ \ge\ 15$$

Together: $u \le 20$ and $u + s \ge 15$ (with $u, s \ge 0$).

(2) The extra resources needed during the update amount to $s$ Pods (since the total is capped at $N+s$). We want $s$ minimal, so take $s = 0$, which requires $u \ge 15$. Combined with $u \le 20$ this gives $15 \le u \le 20$, so for instance $(u, s) = (20, 0)$ meets both requirements with zero additional resources. The lower bound is then $9000/20 = 450$ seconds.

Note, however, that Proposition 7.7 is a lower bound, and the actual duration is longer by the time spent on Pod termination and image pulls. Against a requirement of 600 seconds, a lower bound of 450 seconds leaves only 25 % of margin, so in practice it would be better to allow more room, for example $u = 20, s = 10$ (lower bound 300 seconds).

(3) If $u = 0$, the conditions of (1) give $s \ge 15$. With $s = 15$ the lower bound is $9000/15 = 600$ seconds, exactly on the boundary of the requirement, so taking $s = 30$ for margin gives a lower bound of 300 seconds.

With $s = 30$, the total number of Pods during the update is at most $200 + 30 = 230$. The extra 30 Pods correspond, at 10 Pods per node, to 3 nodes. In other words, the price of the requirement “update in 5 minutes without losing any capacity” is 3 nodes temporarily, during the update. In an environment with a cluster autoscaler, these 3 nodes appear only during the update and disappear afterwards, so the cost is only what is proportional to the update time.

References

• Kubernetes Documentation, “Concepts” and “Horizontal Pod Autoscaling”. https://kubernetes.io/docs/concepts/ — the primary source on Pods, Deployments, Services and the HPA. The HPA update formula and the default tolerance and stabilization window are documented at https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/.
• Docker Docs, “Building best practices”. https://docs.docker.com/build/building/best-practices/ — the official guide to layers, caching and multi-stage builds.
• Open Container Initiative, Image Format Specification. https://github.com/opencontainers/image-spec — the specification of layer stacking and whiteout records. The definition of an image in §4 is based on this.
• B. Burns, B. Grant, D. Oppenheimer, E. Brewer, J. Wilkes, “Borg, Omega, and Kubernetes”, Communications of the ACM 59(5) (2016), 50–57. https://doi.org/10.1145/2890784 — how the design arrived at a declarative API and a reconciliation loop.
• A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, J. Wilkes, “Large-scale cluster management at Google with Borg”, Proceedings of EuroSys 2015. https://doi.org/10.1145/2741948.2741964 — the cluster management system that was the prototype for Kubernetes.
• B. Burns, J. Beda, K. Hightower, L. Evenson, Kubernetes: Up and Running, 3rd ed., O’Reilly Media, 2022 — a practical introduction. The chapters on Deployments and rolling updates correspond to the material of §7.

Appendix: What is better left out of containers

Some things do not lend themselves to containerization. Software strongly dependent on kernel modules or on particular device drivers (since containers share the kernel, separate preparation on the host’s kernel side is required), desktop applications with a GUI, and software whose license is tied to a hardware identifier of the host. Also, introducing Kubernetes into a small system that fits on a single server tends to cost more in operational complexity than it returns in availability. Maintaining the cluster itself — version upgrades, certificate renewal, management of network plugins — is by no means light work.

If you do adopt it, respect the order. The sequence that works in practice is: (1) containerize the development environment first, to eliminate “it works on my machine”; (2) run CI on containers, to gain reproducible builds; (3) containerize stateless applications in production; (4) introduce Kubernetes once the number of machines and the update frequency reach the level that actually requires orchestration. Doing (4) first means taking on the complexity without having the problem it solves.

Consider stateful components last. For databases and message queues, the correctness of backup, replication and failover depends on knowledge specific to each product. If a managed service will do, using it first is the sensible judgement, I think. If you do operate them on Kubernetes, you will find yourself considering, in addition to StatefulSets and persistent volumes, an operator specific to that product — a controller that extends the reconciliation loop with product-specific knowledge.