# Docker and Kubernetes: Why Containers Are Light and Clusters Heal Themselves > Namespaces and cgroups as the real content of a container, why the build cache hits only on a prefix, and how the reconciliation loop of Kubernetes makes a cluster heal itself. > https://rikai.mugen-giken.com/en/computer-science/software-engineering/containers-and-kubernetes ## 0. Key points - A container is not a small virtual machine. It is an ordinary process on the host, with the kernel's **namespaces** restricting *what it can see* and **cgroups** restricting *how much it can use*. There is only one kernel, and it belongs to the host. - The real content of the "it works on my machine" problem is that a program's true dependencies are far larger than its `requirements.txt` — they include the operating system's libraries, its file layout, and its environment variables — and that all of this is left implicit. A Dockerfile turns those implicit dependencies into **code** whose changes Git can track. - An image is a sequence of immutable layers. From that structure alone we obtain two facts: the build cache can only hit on a **prefix** of the instruction sequence (), and deleting a file in a later layer does **not** shrink the image (). Nearly all the conventional wisdom about writing Dockerfiles is a consequence of these two. - The core of Kubernetes is "a declarative API plus a reconciliation loop". The user writes down the desired state, and controllers keep closing the gap between it and the current state. We show in that this design converges in finitely many steps and that it survives dropped notifications. - The benefit of redundancy can be computed. Under independent failures the availability is $1 - p^{N}$, but if all replicas sit on the same node, adding replicas hits a ceiling set by the node's own availability (, ). - The speed of a zero-downtime update is likewise fixed by configuration. The duration of a rolling update has the lower bound $Nt/(u+s)$ (), and the accident of "we played it safe and it took 20 minutes" can be predicted from this formula in advance. ## 1. Motivation: what exactly is the problem with "it works on my machine" Someone joins the team, clones the repository, follows the README, and nothing runs. Half a day of investigation later, the cause turns out to be that developer A's macOS had `libjpeg` installed through Homebrew while the newcomer's machine did not. In software development, time disappears into this sort of thing without limit. Failures that occur only on the production server, or only in CI, are other symptoms of the same disease. Naively this is puzzling. The source code is shared in full through Git; not a single byte differs. The behaviour differs anyway because **the source code is not the only thing the program depends on**. Enumerating the dependencies shows that there are more of them than one expects. | Layer of dependency | Examples | How it used to be shared | |---|---|---| | Language packages | `requirements.txt`, `package-lock.json` | Shared through Git | | OS shared libraries | versions of `libssl`, `libjpeg`, `glibc` | "Please install these" in a setup document | | OS configuration | locale, time zone, file descriptor limits | Nobody wrote it down | | File layout | absolute paths of config files, location of certificates | Different for each person | | Runtime environment variables | `DATABASE_URL`, `PATH` | A `.env` file sent over Slack | | Kernel and architecture | Linux 5.x or 6.x, amd64 or arm64 | Not even consciously considered | Only the top row was shared through Git. The five rows below it were shared through a setup document — a program written in natural language that is never executed. A document that is never executed cannot be tested, and what cannot be tested rots. Historically, the answers to this problem were refined roughly in the following order. 1. **Setup documents** (a wiki, or a Word file). They are never executed, so they inevitably drift away from reality. 2. **Configuration management tools** (CFEngine, Puppet, Chef, Ansible). These turned setup procedures into code. That was a large step forward, but their target is "an existing server on which something is already installed", so the result depends on the previous state. Running the same playbook against a machine that has been in production for ten years and against a brand-new one can give different results. 3. **Virtual machines** (VMware, VirtualBox, Vagrant). Shipping the whole operating system raised reproducibility dramatically. The price was weight: images of several gigabytes, boot times of tens of seconds, and at best a handful running simultaneously on one laptop. 4. **Containers.** These achieve the same "ship the whole environment" as virtual machines without duplicating an entire operating system. One historical remark. The component technologies of containers were not invented by Docker. `chroot`, which replaces the root of the file system a process sees, was in Version 7 Unix in 1979; FreeBSD jail appeared in 2000, Solaris Zones in 2005, and Linux cgroups — developed at Google as "process containers" — entered kernel 2.6.24 in 2008. The ingredients were all on the table by 2008. What Docker changed when it appeared in 2013 was not the isolation mechanism but **the format of the image as a distributable artefact** and **the procedure for building it reproducibly from a single-file recipe**. Once the whole chain was in place — put the recipe (the Dockerfile) in Git, distribute the artefact (the image) through a registry, run it identically anywhere — the environment became, for the first time, an object under version control. It is an example of a technology whose value lay not in the parts but in the way the parts were combined. In what follows we first define precisely what a container is (§3), then derive the Dockerfile conventions as theorems (§4, §5), then look at what Kubernetes does in a world where one machine is no longer enough (§6), and finally evaluate the effects of redundancy, autoscaling and zero-downtime updates numerically (§7). ## 2. Preliminaries: the runtime environment as a 4-tuple To make the discussion precise, we regard "the environment in which a program runs" as the following 4-tuple. $$ E = (K,\ R,\ V,\ Q) $$ - $K$: the **kernel**. The implementation and ABI of the system calls, and the CPU architecture. - $R$: the **root file system**. Which shared libraries live in `/usr/lib`, what is written in `/etc`. - $V$: the **visible scope**. The list of processes, network interfaces, mounts, host name and user-ID mapping that the process can see. - $Q$: the **available resources**. CPU time, memory, number of PIDs, I/O bandwidth. "It works on my machine" arises because $R$, $V$ and $Q$ differ from person to person. A dynamically linked executable, for instance, searches $R$ for its shared libraries at startup. ```bash $ ldd ./myapp linux-vdso.so.1 (0x00007ffd8b7f0000) libjpeg.so.62 => not found libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2e4c000000) ``` That single `not found` line is what costs half a day. ## 3. What a container really is: namespaces and cgroups A container on Linux is a collection of one or more processes that have been given the following three things. 1. **Namespaces**: the view of resources such as process IDs, mounts, networking, host name, inter-process communication and user IDs is separated from that of the host and of other containers. In the notation of §2 this is a restriction of $V$. 2. **cgroups (control groups)**: upper bounds are imposed on the use of CPU, memory, number of PIDs and so on. This is a restriction of $Q$. 3. **A root file system**: by means of `pivot_root` or similar, the processes see a file system prepared for the container as their root. This is a replacement of $R$. The kernel $K$ is the host's, used as it is. Consequently a process inside a container is, seen from the host, an ordinary process appearing in `ps`. Namespaces are independent by kind, so one can separate only what needs separating. The principal ones are these. | Namespace | What is separated | What happens without it | |---|---|---| | mount | the set of mount points | files of another container are visible | | PID | the process-ID space | one can `kill` processes of another container | | network | interfaces, routing, ports | two containers cannot both use port 80 | | IPC | shared memory, semaphores | name collisions occur | | UTS | host name, domain name | everything has the same host name | | user | mapping of user and group IDs | root inside the container is root on the host | | cgroup | the view of the cgroup hierarchy | one can see outside one's own resource limits | The effect of the PID namespace is an easy illustration. Running `ps` inside a container shows one's own process first, as PID 1. Looking at the same process from the host, the PID may be 24187. The same process carries different numbers depending on where you look at it — that is the content of "isolation", and it is a rather modest contraption for something called virtualization. cgroups, by contrast, are about upper bounds. If a memory limit of 512 MiB is set in cgroup v2, then when the processes in that group reach the limit the kernel attempts reclamation and, if that is not enough, stops a process in the group with the OOM killer. A Kubernetes Pod that suddenly becomes `OOMKilled` is the result of this mechanism at work.

The difference between the two halves of the figure comes down to the number of kernels, drawn in the accent colour. With virtual machines, each additional machine brings an additional kernel, and that kernel needs memory and boot time of its own. With containers the kernel count does not grow. This difference shows up as a difference in practical properties. | | Virtual machine | Container | |---|---|---| | Kernel | one per guest | one on the host (shared) | | Boot time | tens of seconds (including OS boot) | tens to hundreds of milliseconds | | Image size | several GB | tens to hundreds of MB | | Number per host | a few to a dozen or so | dozens to hundreds | | Strength of isolation | strong isolation by the hypervisor | isolation by kernel features | | Running a different OS | possible (Windows on Linux) | not possible (same kernel only) | "Starts fast" and "many at once" are the preconditions for the autoscaling and rolling updates we look at later. A unit that takes tens of seconds to start cannot follow a sudden surge in load, and cannot be deployed dozens of times a day. Sharing the kernel means that **a kernel vulnerability is directly a way through the isolation**. Containers were not designed to be the primary barrier in a multi-tenant environment. In practice, observe at least the following. - Do not run as root inside the container (specify `USER` in the Dockerfile; see the example in §4). - Enable the user namespace, so that uid 0 inside the container maps to an unprivileged uid on the host. - Avoid `--privileged`, unnecessary capabilities, and mounts of the host file system. - If untrusted code must run, consider execution environments that interpose a virtualization boundary, such as gVisor, Kata Containers or Firecracker. Note also that "Docker" is no longer the only implementation. The image and runtime specifications are standardized under the OCI (Open Container Initiative), and since v1.24 Kubernetes does not call Docker Engine directly but runs containers through containerd or CRI-O, which implement the CRI. It is thanks to this standardization that an image built from a Dockerfile simply runs. ## 4. Images and layers: two consequences of immutability An **image** is a pair consisting of an ordered sequence of **layers** $L_1, L_2, \ldots, L_n$ together with metadata describing the startup configuration (environment variables, working directory, default command, and so on). Each layer is **immutable data, named uniquely by its content**, recording a difference (addition, modification or deletion of files) against the file system produced by the layers before it. When a container is started, the runtime stacks $L_1$ through $L_n$ (a union file system) and places one thin writable layer on top. Deleting a file in $L_i$ does not erase the actual data in $L_{i-1}$ or earlier; it only records a mark in $L_i$ (a whiteout) saying "this path has been deleted". That layers are immutable and content-addressed — the same idea by which Git names objects by the hash of their content () — yields the pleasant property that an identical layer need be transferred and stored only once. If ten images use the same base image, the base exists physically once on the host. On the other hand, immutability also creates two constraints. Both bear directly on how Dockerfiles should be written, so we state them as theorems. ### 4.1. The cache only helps on a prefix Let the instructions of a Dockerfile be $I_1, \ldots, I_n$. At build time a cache key is computed for each instruction: $$ k_0 = (\text{digest of the base image}), \qquad k_i = H\bigl(k_{i-1},\ \sigma(I_i)\bigr) $$ Here $H$ is a cryptographic hash function and $\sigma(I_i)$ is a normalized representation of the instruction: for `RUN` it is **the command string itself**, while for `COPY` and `ADD` it includes the command string together with **the hash of the contents of the files being copied**. Suppose we build using the key computation above, under the rule that "the $i$-th instruction hits the cache exactly when all instructions up to the $(i-1)$-st hit and a layer corresponding to $k_i$ exists locally". Assume $H$ is collision-free. Then the set of instructions that hit the cache is a **prefix** $\{1, 2, \ldots, m\}$ for some $m$. Moreover, if $j$ is the least index at which $\sigma(I_j)$ has changed since the previous build, then $m \le j - 1$, and $I_j, I_{j+1}, \ldots, I_n$ are all re-executed. The first half follows from the rule itself. If the $i$-th instruction hits, then by the first clause of the rule the $(i-1)$-st also hits. Repeating this, all of $1, \ldots, i-1$ hit. Hence the set of hitting indices is closed downwards, that is, a prefix. It suffices to take $m$ to be the largest hitting index (with $m = 0$ if none hits). For the second half: for $i < j$ the value of $\sigma(I_i)$ has not changed, so as long as $k_0$ is the same, the recurrence for the keys gives inductively that $k_i$ also has its previous value. The layers corresponding to these keys therefore remain locally available and can hit. For $i \ge j$, on the other hand, the second argument of $k_j = H(k_{j-1}, \sigma(I_j))$ has changed. Since $H$ is assumed collision-free, $k_j$ differs from its previous value. Then the first argument of $k_{j+1} = H(k_j, \sigma(I_{j+1}))$ has changed too, so it also differs, and inductively $k_i \ne k_i^{\text{prev}}$ for all $i \ge j$. No layer exists for a key that has never been generated before, so instructions from the $j$-th on all miss the cache. Hence $m \le j-1$, and combining with the prefix property of the first half, everything from $I_j$ onwards is re-executed. The practical meaning of this proposition is a single rule: **put the instructions that change rarely first and the ones that change often last**. Source code changes every time, while the list of dependencies changes a few times a month, so the ordering settles itself. ```dockerfile FROM python:3.12-slim WORKDIR /app COPY . . RUN pip install --no-cache-dir -r requirements.txt CMD ["python", "-m", "myapp"] ``` ```dockerfile FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . CMD ["python", "-m", "myapp"] ``` Compare the two Dockerfiles above. Suppose `pip install` takes 90 seconds and each `COPY` takes 1 second, that during development we build 20 times a day, and that in 19 of those builds only the source code has changed while in one `requirements.txt` has changed as well. In the first Dockerfile, `COPY . .` is the third instruction. Editing a single character of source code changes $\sigma(I_3)$, so by everything from $I_3$ on is re-executed. That is, `pip install` runs every time. Per day this is $$ 20 \times (1 + 90) = 1820\ \text{s} \approx 30\ \text{minutes} $$ In the second, a source change affects only `COPY . .` (the fifth instruction) and later. Neither $\sigma(I_3)$ (`COPY requirements.txt .`) nor $\sigma(I_4)$ (`RUN pip install ...`) changes, so the cache hits through $I_4$. Per day this is $$ 19 \times 1 + 1 \times (1 + 90 + 1) = 111\ \text{s} \approx 2\ \text{minutes} $$ The difference is 28 minutes a day, about nine and a half hours over 20 working days. For a change consisting of swapping two lines in a Dockerfile, that is a large return. is a statement about the classical model of sequential layer caching. BuildKit, the current default builder, analyses the dependencies between instructions as a directed acyclic graph, so independent stages of a multi-stage build are built in parallel and a change in one stage does not invalidate the cache of the other. However, **within a single stage the prefix property holds exactly as stated**, so the ordering convention above is unchanged. One more point: the fact that $\sigma(\texttt{RUN ...})$ is determined by the command string alone has a side effect. `RUN apt-get update` has an unchanging command string, so it keeps hitting the cache forever even after the upstream package index has been updated. If one then adds `RUN apt-get install -y foo` afterwards, it consults the stale index and fails. Combining `apt-get update` and `apt-get install` into a single `RUN` is how this trap is avoided. ### 4.2. Deleting does not shrink the image Measure the total size of an image by the sum $S = \sum_{i=1}^{n} |L_i|$ of the actual data stored in each layer. Suppose a file $f$ of size $s > 0$ is added in layer $L_i$, and that $f$ is deleted by an instruction in a layer $L_j$ with $j > i$. Then the image size $S_{\text{del}}$ with the deletion and the size $S_{\text{keep}}$ without the deletion instruction satisfy $$ S_{\text{del}} = S_{\text{keep}} + w \ \ge\ S_{\text{keep}} $$ where $w \ge 0$ is the size of the whiteout record. In particular, compared with the size $S_{\text{never}} = S_{\text{keep}} - s$ obtained by never creating $f$ in a persistent layer at all, we have $S_{\text{del}} - S_{\text{never}} \ge s$: the deletion instruction does not make the image smaller by $s$ bytes. By , layers are immutable. At the time $L_j$ is built, $L_i$ is already fixed and named by its content hash. Rewriting the contents of $L_i$ would change its name and produce a different layer, so by definition a later instruction cannot modify the contents of $L_i$. Therefore $|L_i|$ still contains the $s$ bytes of $f$ and is unchanged. Now consider $L_j$. In a union file system, deletion is expressed by placing a whiteout record in an upper layer (). So $L_j$ gains a record representing the deletion of $f$, and writing $w \ge 0$ for its size, $|L_j|$ increases by $w$. No other layer changes. Hence $S_{\text{del}} = S_{\text{keep}} + w$, and $w \ge 0$ gives $S_{\text{del}} \ge S_{\text{keep}}$. The last claim follows by substituting $S_{\text{never}} = S_{\text{keep}} - s$, which yields $S_{\text{del}} - S_{\text{never}} = s + w \ge s$. In other words, one must not think of temporary build files as something to "delete later". There are two remedies. - **Create and delete within the same `RUN`.** One `RUN` becomes one layer, so a file removed before the layer is finalized is never recorded. - **Use a multi-stage build.** Discard the intermediates produced in a build stage and copy only the artefact into a new stage. ```dockerfile FROM golang:1.22 AS builder WORKDIR /src COPY go.mod go.sum ./ RUN go mod download COPY . . RUN CGO_ENABLED=0 go build -o /out/server ./cmd/server FROM gcr.io/distroless/static-debian12 COPY --from=builder /out/server /server USER 65532:65532 ENTRYPOINT ["/server"] ``` Suppose that in the multi-stage Dockerfile above the sizes are as follows: the `golang:1.22` base is 800 MB, the dependencies fetched by `go mod download` are 350 MB, the intermediates produced by the build are 30 MB, the statically linked binary produced is 30 MB, and `gcr.io/distroless/static-debian12` is 2 MB. If everything were done in a single stage with a final `RUN rm -rf /go/pkg /src`, then by that `rm` reduces nothing. The total size would be $$ 800 + 350 + 30 + 30 + w \approx 1210\ \text{MB} $$ With the multi-stage build, the final image consists only of the base of the second stage and the binary brought in by `COPY --from=builder`: $$ 2 + 30 = 32\ \text{MB} $$ about one thirty-eighth. Distributing to 50 nodes, the transfer volume drops from 60 GB to 1.6 GB, and the wait when autoscaling adds nodes gets shorter. In addition, since the final image contains no compiler, no shell and no package manager, there is a security benefit: what an intruder can do is drastically reduced. ## 5. The Dockerfile: what it means to turn an environment into code So far we have seen that most of the conventions for writing Dockerfiles follow from and . The rest concerns what "turning it into code" means. The Dockerfile lives in the same repository as the source code and goes through the same review process. As a result, "who changed the production environment, when and why" is recorded in the Git history (see in [Version Control (Git)](/en/computer-science/software-engineering/version-control-git)). The decisive difference from a setup document is that a Dockerfile is **executed**. What is executed breaks the build when it is wrong, so it does not rot. One must nonetheless be careful: "the same Dockerfile" does not mean "the same image". The following three things break reproducibility. 1. **Mutable tags.** The image that `FROM python:3.12-slim` refers to changes its contents when the upstream is updated. If exact reproducibility is required, pin it by digest: `FROM python:3.12-slim@sha256:...`. 2. **Fetching over the network.** `apt-get install` and `pip install` depend on the upstream state at the moment of execution. The more explicitly versions are pinned (`pip install -r requirements.lock`), the more reproducible the build. 3. **Processing that depends on build time.** Embedding timestamps or random numbers produces a different image every time. Here is one more pitfall that everyone eventually falls into. `CMD` and `ENTRYPOINT` have two notations. ```dockerfile CMD python -m myapp CMD ["python", "-m", "myapp"] ``` The upper, **shell form**, expands to `/bin/sh -c "python -m myapp"`, so the process that becomes PID 1 is the shell. Most `sh` implementations do not forward a received `SIGTERM` to their children. Consequently, an attempt to stop the container does not cause the application to begin its shutdown, and after the grace period (30 seconds by default in Kubernetes) it is killed with `SIGKILL`. The symptom of in-flight requests being dropped on every deploy is often caused by exactly this. With the lower, **exec form**, the application itself becomes PID 1 and receives signals directly. The difference is only a pair of brackets, but it decides whether zero-downtime deployment is possible. Let us also settle the handling of configuration values. The principle is that **values that differ per environment, such as database endpoints and log levels, are not baked into the image but passed as environment variables** (the third factor of the so-called Twelve-Factor App). This way, development, staging and production run **exactly the same image**. Being able to ship the very image that was tested is the greatest source of confidence in container-based deployment. ## 6. When one machine is not enough: the design principles of Kubernetes If there is one container and one host, `docker run` is enough. In real operations, however, the following requirements arise at once. - Make the service redundant across three machines, so that it does not stop when one fails. - Run three machines at night and twenty at the daytime peak. - Place containers on nodes that have free memory. - Roll out a new version gradually, without downtime. - Reach the service by name, without knowing which node it runs on. Doing this by hand is impossible. **Orchestration** means automating this family of decisions, and Kubernetes is one implementation of it. Kubernetes was released by Google in 2014 and incorporates into its design the operational experience of Borg, Google's internal cluster management system (for Borg see the paper of Verma et al.; for how the design philosophy was inherited see the article by Burns et al., both in the references). ### 6.1. From imperative to declarative There is only one central design concept in Kubernetes. The user registers with the API not "which operations to perform in which order" but "**what state things should be in**" (the desired state). The system keeps running a **reconciliation loop**. On each iteration it 1. reads the desired state $d$, 2. observes the current state $s$, 3. performs operations that reduce the difference between $d$ and $s$. These three steps do not depend on the history of how $s$ came about; they are determined by **$d$ and $s$ at that moment alone**. This property is called being **level-triggered**. For contrast, consider imperative operation: a scheme in which one sends the command "add one Pod". Under this scheme, if the command is lost on the way the count stays short, and if it arrives twice the count is too high. In the middle of an incident, both happen routinely. With a scheme that registers the state "there should be three Pods", by contrast, whether the command is lost or duplicated, the next loop corrects things on the basis of the currently observed reality.

API["API server / persisted in etcd"] API --> C["Controller: compares desired with current"] C -->|operations closing the gap| K["Scheduler and kubelet"] K --> R["Current state: the Pods actually running"] R -->|observation| C`} /> Let $\mathcal{O}$ be a finite set of target objects with $|\mathcal{O}| = M$, and for a desired state $d$ and an observed state $s_t$ at time $t$ consider the number of mismatched objects $$ D(s_t) = \#\{o \in \mathcal{O} : s_t(o) \ne d(o)\} $$ Assume the following. - **(A1)** The operations of each loop are determined by $(d, s_t)$ alone (the level-triggered property of ). - **(A2)** If $D(s_t) > 0$, then one iteration resolves at least one mismatch, and no object that already matched becomes mismatched. - **(A3)** No external change of state (node failure, or a rewrite of $d$) occurs while the loop runs. Then $D(s_{t+1}) \le D(s_t) - 1$ as long as $D(s_t) > 0$, and $s = d$ is reached in at most $D(s_0) \le M$ iterations. After that, by the second half of (A2) the state does not change and no operations are performed (idempotence). Furthermore, under (A1) the input of a loop is only the current $(d, s_t)$, so **the convergence claim is unaffected by dropping past event notifications or by receiving the same notification several times**. $D$ is a function with values in the non-negative integers. By the first half of (A2), when $D(s_t) > 0$ at least one mismatch is removed. By the second half of (A2), no new mismatch appears. By (A3), neither $d$ nor $s$ changes in the meantime through external causes. Hence $D(s_{t+1}) \le D(s_t) - 1$. That $D(s_0) \le M$ follows from the definition of $D$ (the mismatched objects form a subset of $\mathcal{O}$). A sequence of non-negative integers $D(s_0) > D(s_1) > \cdots$ cannot decrease forever, so after at most $D(s_0)$ iterations $D(s_T) = 0$, that is, $s_T = d$. Once $s_T = d$ is reached, $D = 0$, so there is no mismatch for (A2) to resolve and the set of operations is empty. Running the same loop any number of times therefore leaves the state unchanged. This is idempotence. Finally we check the last claim. By (A1), the operations at time $t$ are a function of $(d, s_t)$. Dropped or duplicated notifications only change *when* a loop is triggered; they do not change the input $(d, s_t)$ of a loop that is triggered. Hence, as long as loops continue to be triggered at finite intervals, the argument above applies unchanged and convergence holds. In real operation, the assumptions that break most easily are (A2) and (A3). The typical failure of (A2) is a container that crashes right after starting. The controller judges that a Pod is missing and creates one, the Pod fails to start, and $D$ does not decrease. The loop keeps running, and the retry interval grows exponentially under the name `CrashLoopBackOff`. What guarantees is that "if the gap-closing operations actually work, convergence follows", not that "any configuration will eventually start working". As for (A3), in a cluster it is always violated to begin with. Nodes fail and users update manifests. The realistic statement is therefore not "it converges and finishes" but "**it converges once the disturbances stop**". When one node fails, $D$ jumps, and the loop again pushes it down towards zero. What is called self-healing is exactly this pushing down. ### 6.2. The principal objects A **Pod** is the smallest unit of deployment in Kubernetes: a group of one or more containers. Containers in the same Pod share the network namespace (hence the IP address and `localhost`) and storage volumes, and are always placed on the same node. A Pod is a disposable entity: when it is restarted, both its IP address and its name change. The point that Pods are disposable is important. It is precisely why mechanisms for holding state on top of them are needed. Here are the principal objects. | Object | What it declares as the desired state | |---|---| | Pod | run one set of these containers | | ReplicaSet | keep $N$ copies of this Pod | | Deployment | keep $N$ copies of this Pod, updating them by a specified strategy | | Service | make the Pods carrying this label reachable through a fixed name and virtual IP | | Ingress | route external HTTP traffic to Services | | ConfigMap / Secret | inject configuration values and secrets into Pods | | StatefulSet | keep $N$ copies, each with a stable name and a persistent volume | | Job / CronJob | run this work to completion once, or periodically | | HorizontalPodAutoscaler | adjust the replica count so that a metric reaches its target | When a user writes a Deployment, the Deployment controller creates a ReplicaSet, the ReplicaSet controller creates Pods, the scheduler assigns nodes to Pods, and the kubelet on that node calls the container runtime to start the containers. The point is that **each stage runs its own reconciliation loop independently**: the party responsible at each stage looks only at its own desired state and current state. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: web spec: replicas: 3 selector: matchLabels: app: web strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 template: metadata: labels: app: web spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: web containers: - name: web image: registry.example.com/web@sha256:0b1f2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b ports: - containerPort: 8080 resources: requests: cpu: "200m" memory: "256Mi" limits: memory: "512Mi" readinessProbe: httpGet: path: /healthz port: 8080 periodSeconds: 2 ``` ```yaml apiVersion: v1 kind: Service metadata: name: web spec: selector: app: web ports: - port: 80 targetPort: 8080 ``` Let us follow what happens right after the manifests above are applied. Since `replicas: 3`, the ReplicaSet controller creates three Pods, and `topologySpreadConstraints` places them on three distinct nodes (nodes with different `kubernetes.io/hostname`). The setting `maxSkew: 1` requires the difference in Pod counts between nodes to stay within one. Now suppose the hardware of one of those nodes fails. 1. The periodic reports from that node's kubelet stop arriving. By default, after some time without a response (on the order of tens of seconds) the node is recorded as `NotReady`. 2. After a further interval, the Pods on that node are marked for deletion. At this point the observed state of the ReplicaSet is "2 running", so $D = 1$ (in the notation of ). 3. To close the gap, the ReplicaSet controller creates one Pod. The scheduler picks, among the remaining nodes, one with enough room to satisfy the `requests` of `cpu: 200m` and `memory: 256Mi` and that does not violate `maxSkew`. 4. The container of the new Pod starts, and once `/healthz` answers the readiness probe issued every 2 seconds, the Pod is added to the Service's endpoint list and traffic begins to flow. $D$ returns to $0$. Nobody lifted a finger. All the human did was write "there should be three". Note that without a `readinessProbe`, the Service starts routing traffic from the instant the container's process starts. If the application takes 10 seconds to initialize, errors are returned for those 10 seconds. To be free of downtime, one needs a way to declare readiness. One remark about `resources`. `requests` is the reservation the scheduler uses to decide placement, while `limits` is the upper bound configured in the cgroup. For memory an upper bound is written because exceeding it results in `OOMKilled`, but a CPU `limit` does not kill on excess — it throttles — and for latency-sensitive services this can cause unintended latency degradation. That is why no CPU `limit` is written here. A Service is the mechanism that reconciles the disposability of Pods with the wish to call a service by a stable name from elsewhere. The cluster's DNS () resolves names such as `web.default.svc.cluster.local` to the Service's virtual IP, and a mechanism on each node forwards packets addressed to that virtual IP to the actual Pod IPs. Understanding which layer performs what in this forwarding becomes much clearer with the material of [Networking (TCP/IP)](/computer-science/software-engineering/networking-tcp-ip), especially . Whether to run stateful software such as a database on Kubernetes is a question to be judged carefully. It is possible with StatefulSets and persistent volumes, but the hard parts of operating a database — backups, failover, schema changes — remain (see [Foundations of Database Design](/en/computer-science/software-engineering/database-design)). Using a cloud provider's managed database and putting only stateless applications on Kubernetes is still a strong option (in the terms of in [Cloud Computing (AWS, GCP)](/computer-science/software-engineering/cloud-computing), it is a choice of how much operational responsibility to hand to the provider). ## 7. Scalability and fault tolerance, quantified One often hears "redundancy makes things stronger" and "it scales automatically", but how much stronger can be computed. The definition of availability itself, and the composition rules for series and parallel arrangements, are collected in . Below we apply them to the placement of Kubernetes replicas. ### 7.1. Redundancy and availability Suppose a service is provided by $N$ replicas, that each replica is unavailable at a given moment with probability $p$ (with $0 < p < 1$), and that the failures of the replicas are mutually independent. If the service is available whenever at least one replica is available, then its availability is $$ A(N) = 1 - p^{N} $$ Consequently the number of replicas needed to meet a target availability $A^{*}$ (with $0 < A^{*} < 1$) is the least integer satisfying $$ N \ \ge\ \frac{\ln(1 - A^{*})}{\ln p} $$ The service is unavailable exactly when **all** $N$ replicas are unavailable. Writing $F_i$ for the event that replica $i$ is unavailable, independence gives $$ \Pr\Bigl[\bigcap_{i=1}^{N} F_i\Bigr] = \prod_{i=1}^{N} \Pr[F_i] = p^{N} $$ Hence the availability is $A(N) = 1 - p^{N}$. Now for the required number of replicas. The condition $1 - p^{N} \ge A^{*}$ is equivalent to $p^{N} \le 1 - A^{*}$. Both sides are positive, so taking natural logarithms gives $N \ln p \le \ln(1 - A^{*})$. Since $0 < p < 1$ we have $\ln p < 0$, so dividing by $\ln p$ reverses the inequality and yields $N \ge \ln(1-A^{*}) / \ln p$. Suppose a single replica has availability 99 %, that is, $p = 0.01$. Taking 30 days to be 43,200 minutes, we compute the downtime. - $N = 1$: $A = 0.99$, downtime per month $43{,}200 \times 0.01 = 432$ minutes (about 7.2 hours). - $N = 2$: $A = 1 - 10^{-4} = 0.9999$, downtime $43{,}200 \times 10^{-4} = 4.32$ minutes. - $N = 3$: $A = 1 - 10^{-6} = 0.999999$, downtime $43{,}200 \times 10^{-6} \approx 2.6$ seconds. Going from one replica to two turns 7 hours of monthly downtime into 4 minutes. In the other direction, computing the number required for a target $A^{*} = 0.99999$ (five nines) via gives $$ N \ \ge\ \frac{\ln(10^{-5})}{\ln(10^{-2})} = \frac{-11.5129}{-4.6052} = 2.5 $$ so $N = 3$. Everything so far depends entirely on the assumption of independence. Let us see what happens when it is dropped. In addition to the setting of , suppose all $N$ replicas are placed on the same node. Let $q$ (with $0 < q < 1$) be the probability that the whole node is unavailable, and suppose that, conditionally on the node being healthy, each replica is unavailable independently with probability $p$. Then $$ A(N) = 1 - \bigl(q + (1-q)\,p^{N}\bigr) \ \le\ 1 - q $$ so that increasing the number of replicas without bound never raises the availability above $1 - q$. Split the event that the service is unavailable according to whether the node has failed. If the node has failed (probability $q$), all replicas on it are unavailable, so the service is unavailable. If the node is healthy (probability $1-q$), each replica is conditionally unavailable independently with probability $p$, so by the same computation as in the proof of the conditional probability that all of them fail is $p^{N}$. By the law of total probability, $$ \Pr[\text{unavailable}] = q \cdot 1 + (1-q)\, p^{N} = q + (1-q)p^{N} $$ Since $(1-q)p^{N} \ge 0$ we get $\Pr[\text{unavailable}] \ge q$, that is, $A(N) \le 1-q$. As $N \to \infty$ we have $p^{N} \to 0$ (because $0 Concretely, suppose a node's monthly failure probability is $q = 10^{-3}$. In we computed that three replicas give five nines, but if those three sit on the same node, the availability is capped at $1 - 10^{-3} = 0.999$. That is 43.2 minutes of downtime per month, and no number of replicas changes it. **The benefit of redundancy is determined by where the correlations in failure lie.** The `topologySpreadConstraints` of is precisely the setting that breaks this correlation. Changing `topologyKey` from the node to the availability zone (`topology.kubernetes.io/zone`) also guards against the coarser correlation of an entire zone failing. At the same time, communication across zones incurs additional latency. The right order is to decide what one wants to be independent, and only then write the placement constraints. ### 7.2. Horizontal autoscaling Let the current replica count be $n \ge 1$, the average value of the metric per Pod be $m > 0$, and the target be $T > 0$, and update the replica count by the same rule as the Kubernetes HorizontalPodAutoscaler: $$ n' = \Bigl\lceil\, n \cdot \frac{m}{T} \,\Bigr\rceil, \qquad \text{except that } n' = n \text{ when } \Bigl|\frac{m}{T} - 1\Bigr| \le \tau $$ where $\tau$ is a tolerance, by default $0.1$. Now suppose the total load on the system, $\lambda > 0$, is constant, is distributed evenly across all Pods, and that the per-Pod metric can be written $m = \lambda / n$. Then, whatever the value of $n$, a single update gives $$ n' = \Bigl\lceil \frac{\lambda}{T} \Bigr\rceil =: n^{*} $$ Moreover $n^{*}$ is a fixed point: as long as $\lambda / (n^{*} T) \ge 1 - \tau$ holds, no change in the downward direction occurs either. Substitute $m = \lambda/n$ into the update formula: $$ n \cdot \frac{m}{T} = n \cdot \frac{\lambda/n}{T} = \frac{\lambda}{T} $$ so $n$ cancels. Hence $n' = \lceil \lambda/T \rceil = n^{*}$, which does not depend on the replica count $n$ before the update. Next we show $n^{*}$ is a fixed point. When the replica count is $n^{*}$, the metric is $m^{*} = \lambda/n^{*}$. By definition of the ceiling, $n^{*} \ge \lambda/T$, so $m^{*} = \lambda/n^{*} \le T$, that is, the ratio $r := m^{*}/T$ satisfies $r \le 1$. The assumption $\lambda/(n^{*}T) \ge 1-\tau$ says exactly that $r \ge 1-\tau$, so $1-\tau \le r \le 1$, that is, $|r - 1| \le \tau$. This is precisely the tolerance condition of the update rule, so $n' = n^{*}$ and no change occurs. Take the target to be "CPU usage per Pod equal to 60 % of `requests`", and suppose 6 Pods are currently running at an average of 85 %. By the update rule of , $$ n' = \Bigl\lceil 6 \times \frac{85}{60} \Bigr\rceil = \lceil 8.5 \rceil = 9 $$ After the increase to 9, the average utilization, assuming the total load is unchanged, is $$ \frac{6 \times 85}{9} \approx 56.7\ \% $$ The ratio is $56.7/60 = 0.944$, and $|0.944 - 1| = 0.056 \le 0.1$, so it falls within the tolerance and no further change occurs. The system settles in a single adjustment, with no oscillation between growing and shrinking. It is worth seeing what would happen without the tolerance. Applying the update rule in the state of 9 Pods at 56.7 % gives $\lceil 9 \times 56.7/60 \rceil = \lceil 8.5 \rceil = 9$, so in this example there happens to be no oscillation. Because of the rounding up in the ceiling, however, it is easy to construct settings where a ratio that stays slightly below 1 makes the count grow and shrink repeatedly. The tolerance $\tau$ and the stabilization window for scaling down (5 minutes by default) exist to stop oscillations of this kind. The crux of is the assumption $m = \lambda/n$, that is, "adding Pods reduces the load per machine in inverse proportion" (for the response time when a load is spread evenly across $n$ machines see ). The standard case in which this assumption breaks is a saturated downstream database. Adding Pods not only fails to shorten the waiting time; the number of connections grows, the database gets even slower, CPU utilization does not fall, and so the HPA adds still more Pods — a vicious circle. Autoscaling does not remove the rate-limiting stage. Determine where the bottleneck is before configuring it. ### 7.3. How long a zero-downtime update takes The behaviour of a Deployment's `RollingUpdate` is determined by two numbers. With $N$ the desired replica count, `maxUnavailable` $= u$ means "keep at least $N - u$ Pods available" and `maxSurge` $= s$ means "keep the total number of Pods at most $N + s$" (both default to 25 %). Suppose a Deployment with $N$ replicas is rolled out with `maxUnavailable` $= u \ge 0$ and `maxSurge` $= s \ge 0$ (with $u + s \ge 1$). Assume the following. - Throughout the update, the number $R$ of available Pods satisfies $R \ge N - u$ and the total number $P$ of Pods satisfies $P \le N + s$. - A new Pod requires at least a time $t > 0$ between being created and becoming available. - At the completion of the update, $N$ new Pods are available. Then the duration $\Theta$ of the update satisfies $$ \Theta \ \ge\ \frac{N t}{u + s} $$ Let $g(x) = P(x) - R(x)$ be the number of Pods at time $x$ that have been created but are not yet available. By assumption $P(x) \le N+s$ and $R(x) \ge N-u$, so $$ g(x) \le (N+s) - (N-u) = u + s $$ holds at all times. That is, at most $u+s$ Pods can be in preparation simultaneously. On the other hand, by the completion of the update $N$ new Pods have been created, and each of them stays in the "created but not available" state for at least $t$ (the second assumption). Therefore the integral of $g$ over the update interval $[0, \Theta]$, counting only the new Pods, satisfies $$ \int_{0}^{\Theta} g(x)\, dx \ \ge\ N t $$ On the other hand, from $g(x) \le u+s$, $$ \int_{0}^{\Theta} g(x)\, dx \ \le\ (u+s)\,\Theta $$ Combining the two gives $(u+s)\Theta \ge Nt$, that is, $\Theta \ge Nt/(u+s)$. Consider a service with $N = 60$ for which a new Pod takes $t = 20$ seconds to become available. - With the defaults $u = s = 25\ \%$, that is $u = s = 15$: $\Theta \ge 60 \times 20 / 30 = 40$ seconds. - With "we absolutely must not lose capacity", $u = 0$ and $s = 1$: $\Theta \ge 60 \times 20 / 1 = 1200$ seconds, that is 20 minutes. A factor of 30 for the same service. The setting $u = 0, s = 1$ — "always keep 60 Pods available and replace them one at a time" — is intuitively the safest, but if an update takes 20 minutes, so does an emergency rollback. During an incident that is a heavy constraint. If capacity must not be reduced, it is better to keep $u = 0$ and increase $s$ instead (for example $s = 10$ gives $\Theta \ge 120$ seconds), at the price of 10 Pods' worth of extra compute during the update. is a lower bound, so the actual duration will be longer. Leave margin when choosing the settings. ## 8. Exercises Answer the following about this Dockerfile. Assume `npm ci` takes 90 seconds and each `COPY` takes 1 second. ```dockerfile FROM node:22-slim WORKDIR /app COPY . . RUN npm ci RUN npm run build CMD ["node", "dist/main.js"] ``` (1) When one file under `src/` is edited and the image is rebuilt, which instructions hit the cache? Explain using . (2) Rewrite the Dockerfile so as to minimize the daily time spent waiting for `npm ci`, for a developer who builds 20 times a day editing only `src/`. (3) With the rewritten Dockerfile, what kind of change causes `npm ci` to be re-executed? (1) `COPY . .` is the third instruction, and $\sigma(I_3)$ includes the hash of the contents of the copied files. Editing `src/` changes $\sigma(I_3)$, so by the second half of we get $m \le 2$, and everything from $I_3$ on (`COPY . .`, `npm ci`, `npm run build`) is re-executed. Only $I_1$ and $I_2$, corresponding to `FROM` and `WORKDIR`, hit. (2) Copy only the dependency manifests first. ```dockerfile FROM node:22-slim WORKDIR /app COPY package.json package-lock.json ./ RUN npm ci COPY . . RUN npm run build CMD ["node", "dist/main.js"] ``` Editing `src/` changes neither $\sigma(I_3)$ (`COPY package.json package-lock.json ./`) nor $\sigma(I_4)$ (`RUN npm ci`), so by the cache hits through $I_4$ and `npm ci` is not re-executed. The daily waiting time for `npm ci` drops from $20 \times 90 = 1800$ seconds to zero. (3) When the contents of `package.json` or `package-lock.json` change. Then $\sigma(I_3)$ changes, so everything from $I_3$ on is re-executed and `npm ci` runs. Reinstalling when dependencies are added or updated is the correct behaviour, and exactly what we intended. Build the following Dockerfile. Assume the amounts of data newly written by each step are: base image 120 MB, instruction 2: 400 MB, instruction 3: 120 MB, the source unpacked by instruction 4: 350 MB, and the build intermediates plus the binary in instruction 5: 240 MB in total. The size of whiteout records is negligible. ```dockerfile FROM debian:12 RUN apt-get update && apt-get install -y build-essential RUN curl -sL https://example.com/src.tar.gz -o /tmp/src.tar.gz RUN tar xf /tmp/src.tar.gz -C /opt && rm /tmp/src.tar.gz RUN make -C /opt/src && make -C /opt/src install ``` (1) Compute the size of the final image. How does the `rm` in instruction 4 affect it? (2) Suppose the resulting binary is 40 MB and is all that is needed to run. Using `debian:12-slim` (80 MB) as the runtime base, write a Dockerfile that minimizes the final image, and compute its size. (1) By , the `rm /tmp/src.tar.gz` of instruction 4 does not delete the 120 MB written in the layer of instruction 3. The layer of instruction 3 is immutable and named by its content hash, so no later instruction can rewrite it. The deletion appears only as a whiteout record in the layer of instruction 4, whose size we agreed to neglect. Hence $$ 120 + 400 + 120 + 350 + 240 = 1230\ \text{MB} $$ The `rm` reduced the image by not a single byte; what it reduced is only "the files visible from a running container". (2) Use a multi-stage build. ```dockerfile FROM debian:12 AS builder RUN apt-get update && apt-get install -y build-essential curl RUN curl -sL https://example.com/src.tar.gz -o /tmp/src.tar.gz \ && tar xf /tmp/src.tar.gz -C /opt \ && rm /tmp/src.tar.gz \ && make -C /opt/src \ && make -C /opt/src install FROM debian:12-slim COPY --from=builder /usr/local/bin/myapp /usr/local/bin/myapp USER 1000:1000 ENTRYPOINT ["/usr/local/bin/myapp"] ``` The final image contains only the base of the second stage and the copied binary, so $$ 80 + 40 = 120\ \text{MB} $$ About one tenth of 1230 MB. Fetching, unpacking, building and deleting are combined into one `RUN` in the first stage in order to keep that stage's layers small and save space in the intermediate cache; it has no effect on the size of the final image (the layers of the first stage are not part of it). A single replica of a service has availability 99 % ($p = 0.01$), and the probability that a node carrying replicas fails entirely is $q = 0.002$ per month. (1) Using , find the ceiling on availability when 4 replicas are all placed on a single node. Convert it into monthly downtime, taking 30 days to be 43,200 minutes. (2) Now spread the 4 replicas over 4 distinct nodes, and suppose node failures and replica failures may all be treated as independent. Find the probability that a single replica is unavailable because either "its node fails" or "the replica itself fails", and then find the availability of the whole service. (3) From the result of (2), how many replicas are needed for a target of 99.999 %? (1) By , $A \le 1 - q = 1 - 0.002 = 0.998$. The exact value is $$ A = 1 - \bigl(0.002 + 0.998 \times 0.01^{4}\bigr) = 1 - (0.002 + 9.98 \times 10^{-9}) \approx 0.998 $$ where the replica-side contribution of order $10^{-8}$ is completely buried in the node failure term $2\times10^{-3}$. The downtime is $43{,}200 \times 0.002 = 86.4$ minutes. Going from 4 replicas to 40 does not change these 86.4 minutes. (2) A single replica is available when its node is healthy and the replica itself is healthy, with probability $(1-q)(1-p) = 0.998 \times 0.99 = 0.98802$. So the probability that one replica is unavailable is $$ p' = 1 - 0.98802 = 0.01198 $$ The four nodes are independent, so applies directly: $$ A(4) = 1 - (0.01198)^{4} = 1 - 2.06 \times 10^{-8} \approx 0.99999998 $$ The downtime is $43{,}200 \times 2.06\times10^{-8} \approx 0.0009$ minutes per month, that is, about 0.05 seconds. Compare this with the 86.4 minutes of (1). **With the same 4 replicas, changing only where they are placed makes a difference of more than five orders of magnitude.** (3) Substitute $p' = 0.01198$ and $A^{*} = 0.99999$ into the formula of : $$ N \ \ge\ \frac{\ln(10^{-5})}{\ln(0.01198)} = \frac{-11.5129}{-4.4234} = 2.60 $$ so $N = 3$. This holds under the assumption that the three nodes are independent, however; if the three sit in the same rack or on the same power supply, that correlation imposes a new ceiling. Re-instantiate the $q$ of for each granularity of failure under consideration and evaluate accordingly. A service has $N = 200$ replicas. A new Pod takes $t = 45$ seconds to become available. There are two operational requirements. - During an update, keep the available replicas at all times at 90 % of $N$ or above. - Complete an update (and a rollback) within 10 minutes. (1) Write the conditions on $u$ (`maxUnavailable`) and $s$ (`maxSurge`) meeting the requirements, as inequalities. (2) Find the $(u, s)$ minimizing the extra compute needed during the update. (3) If $u = 0$ is required, what should $s$ be? With 10 Pods per node, how many extra nodes are then needed during the update? (1) The capacity requirement says that the lower bound $N - u$ on the available count during the update is at least $0.9N$: $$ 200 - u \ \ge\ 180 \quad\Longleftrightarrow\quad u \le 20 $$ The time requirement makes it **necessary** that the lower bound of be at most 600 seconds: $$ \frac{Nt}{u+s} = \frac{200 \times 45}{u+s} = \frac{9000}{u+s} \ \le\ 600 \quad\Longleftrightarrow\quad u + s \ \ge\ 15 $$ Together: $u \le 20$ and $u + s \ge 15$ (with $u, s \ge 0$). (2) The extra resources needed during the update amount to $s$ Pods (since the total is capped at $N+s$). We want $s$ minimal, so take $s = 0$, which requires $u \ge 15$. Combined with $u \le 20$ this gives $15 \le u \le 20$, so for instance $(u, s) = (20, 0)$ meets both requirements with zero additional resources. The lower bound is then $9000/20 = 450$ seconds. Note, however, that is a **lower** bound, and the actual duration is longer by the time spent on Pod termination and image pulls. Against a requirement of 600 seconds, a lower bound of 450 seconds leaves only 25 % of margin, so in practice it would be better to allow more room, for example $u = 20, s = 10$ (lower bound 300 seconds). (3) If $u = 0$, the conditions of (1) give $s \ge 15$. With $s = 15$ the lower bound is $9000/15 = 600$ seconds, exactly on the boundary of the requirement, so taking $s = 30$ for margin gives a lower bound of 300 seconds. With $s = 30$, the total number of Pods during the update is at most $200 + 30 = 230$. The extra 30 Pods correspond, at 10 Pods per node, to **3 nodes**. In other words, the price of the requirement "update in 5 minutes without losing any capacity" is 3 nodes temporarily, during the update. In an environment with a cluster autoscaler, these 3 nodes appear only during the update and disappear afterwards, so the cost is only what is proportional to the update time. ## References - Kubernetes Documentation, "Concepts" and "Horizontal Pod Autoscaling". [https://kubernetes.io/docs/concepts/](https://kubernetes.io/docs/concepts/) — the primary source on Pods, Deployments, Services and the HPA. The HPA update formula and the default tolerance and stabilization window are documented at [https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). - Docker Docs, "Building best practices". [https://docs.docker.com/build/building/best-practices/](https://docs.docker.com/build/building/best-practices/) — the official guide to layers, caching and multi-stage builds. - Open Container Initiative, *Image Format Specification*. [https://github.com/opencontainers/image-spec](https://github.com/opencontainers/image-spec) — the specification of layer stacking and whiteout records. The definition of an image in §4 is based on this. - B. Burns, B. Grant, D. Oppenheimer, E. Brewer, J. Wilkes, "Borg, Omega, and Kubernetes", *Communications of the ACM* 59(5) (2016), 50–57. [https://doi.org/10.1145/2890784](https://doi.org/10.1145/2890784) — how the design arrived at a declarative API and a reconciliation loop. - A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, J. Wilkes, "Large-scale cluster management at Google with Borg", *Proceedings of EuroSys 2015*. [https://doi.org/10.1145/2741948.2741964](https://doi.org/10.1145/2741948.2741964) — the cluster management system that was the prototype for Kubernetes. - B. Burns, J. Beda, K. Hightower, L. Evenson, *Kubernetes: Up and Running*, 3rd ed., O'Reilly Media, 2022 — a practical introduction. The chapters on Deployments and rolling updates correspond to the material of §7. ## Appendix: What is better left out of containers **Some things do not lend themselves to containerization.** Software strongly dependent on kernel modules or on particular device drivers (since containers share the kernel, separate preparation on the host's kernel side is required), desktop applications with a GUI, and software whose license is tied to a hardware identifier of the host. Also, introducing Kubernetes into a small system that fits on a single server tends to cost more in operational complexity than it returns in availability. Maintaining the cluster itself — version upgrades, certificate renewal, management of network plugins — is by no means light work. **If you do adopt it, respect the order.** The sequence that works in practice is: (1) containerize the development environment first, to eliminate "it works on my machine"; (2) run CI on containers, to gain reproducible builds; (3) containerize stateless applications in production; (4) introduce Kubernetes once the number of machines and the update frequency reach the level that actually requires orchestration. Doing (4) first means taking on the complexity without having the problem it solves. **Consider stateful components last.** For databases and message queues, the correctness of backup, replication and failover depends on knowledge specific to each product. If a managed service will do, using it first is the sensible judgement, I think. If you do operate them on Kubernetes, you will find yourself considering, in addition to StatefulSets and persistent volumes, an operator specific to that product — a controller that extends the reconciliation loop with product-specific knowledge.