A Survival Strategy for Software Engineers in the AI Era: Estimating the Ceiling of Automation

0. Key points

• The only stage of development that LLM code generation makes faster is implementation. By Amdahl’s law, if implementation accounts for a fraction $p$ of the total, then even an infinitely fast implementation stage caps the overall speedup at $1/(1-p)$ (Corollary 3.2).
• As generation becomes cheap, what becomes relatively expensive is verification. Even with the cost of generation driven to $0$, the expected total cost cannot fall below the verification cost divided by the acceptance probability (Corollary 4.3). This is the source of the value of people who can review.
• Verification cannot be handed over to a machine wholesale. Every non-trivial property of the meaning of a program is undecidable (Theorem 5.1).
• Nor can one leave “what to build” vague and let the model generate it. Selecting one behaviour out of $N$ candidates requires at least $\log_2 N$ bits of instruction (Proposition 6.2). This encoding is what prompt engineering really is.
• The value of architecture can be quantified too. Splitting $n$ elements suitably into $m$ modules cuts the number of interactions to be inspected from $\Theta(n^2)$ to $\Theta(n^{4/3})$ (Proposition 7.1).
• The conclusion is that what one should learn is problem framing, architecture and the ability to verify, together with the low-level knowledge and the mathematics that support them.

1. Motivation: what did code generation change?

When OpenAI published its evaluation of Codex in 2021, the shock felt by many engineers came down to a single point: working code emerges from a description of a function written in natural language [5]. Completion tools such as GitHub Copilot then entered day-to-day practice, and by 2023 there were measurements of productivity from a randomised controlled trial (RCT) [6].

Reactions to this change split into two camps. One says that engineering jobs will disappear; the other says the tools are toys and nothing essential has changed. I regard both as poorly grounded. The former extrapolates measured numbers far beyond what was measured; the latter ignores the measured numbers altogether.

This article takes a different position: estimate the size of the change stage by stage. Fortunately, software engineering already has a classical instrument for answering “if only part of the work gets faster, how much faster does the whole get?” — Amdahl’s 1967 argument [1], together with Brooks’s 1987 distinction between essential and accidental complexity [2]. In Brooks’s vocabulary, code generation attacks mainly the accidental side: not the problem itself, but the drudgery of expressing it to a machine. Essential complexity, that is, deciding what ought to be built, survives a change of tools.

Yet merely chanting that the essence remains does not tell anyone what to learn. So this article answers four questions with equations.

1. If implementation gets faster, how much faster does the whole get?
2. If generation gets cheaper, what becomes relatively expensive?
3. Can the thing that becomes expensive be automated?
4. What should be designed so that it does not become expensive?

2. Preliminaries: decomposing development into stages

Let us fix the object of discussion. We split the work of building software into the following six stages. Names differ between organisations, but the ordering and the roles are broadly common.

Definition 2.1（Development stages and the automation ratio）

Suppose the total working time $T$ of software development decomposes as a sum of the times of six stages:

$$T = T_{\text{framing}} + T_{\text{spec}} + T_{\text{design}} + T_{\text{impl}} + T_{\text{verify}} + T_{\text{ops}}$$

Here framing is the stage of deciding what problem is to be solved at all; specification is the stage of describing what must be solved without ambiguity; design is the stage of fixing the structure of the solution (module decomposition, interfaces, data representation); implementation is the stage of turning the design into code; verification is the stage of confirming that the code meets the specification; and operation is the stage of monitoring and repair after deployment.

Let $A$ be the set of stages a given tool makes faster, and call the fraction of the total occupied by their combined time,

$$p = \frac{1}{T} \sum_{i \in A} T_i \in [0, 1],$$

the automation ratio of that tool.

flowchart LR
A["Framing<br/>what to solve"] --> B["Specification<br/>removing ambiguity"]
B --> C["Design<br/>decomposition and interfaces"]
C --> D["Implementation<br/>writing code"]
D --> E["Verification<br/>does it meet the spec"]
E --> F["Operation<br/>monitoring and repair"]
E -.->|if rejected, go back| D
D:::auto
classDef auto stroke-width:3px

The development stages, and where code generation acts directly

The boxed stage in the figure, implementation, is the one code generation speeds up directly. The dashed arrow represents the loop back to implementation when verification fails. That loop will matter later (Proposition 4.2).

Note

Splitting the work into stages by time is a coarse approximation. In reality framing and design go back and forth, and holes in the specification are often noticed while implementing. Still, for the purpose of counting where the speedup went, this level of coarseness already yields effective conclusions.

3. The ceiling of automation (Amdahl’s law)

We begin with the most naive question. If implementation becomes $s$ times faster, how much faster does the whole become?

Proposition 3.1（Generalised Amdahl's law）

Let the total working time be $T > 0$, and suppose a fraction $p \in [0,1]$ of it is accelerated by a speed factor $s \ge 1$ while the remaining fraction $1-p$ is unchanged. Then the total time $T'$ after acceleration and the overall speedup $S(p, s) = T / T'$ are given by

$$T' = (1-p)T + \frac{p}{s}T, \qquad S(p, s) = \frac{1}{(1-p) + \dfrac{p}{s}}.$$

Proof（Proposition 3.1）

The accelerated part takes time $pT$. To become $s$ times faster means doing the same work in $1/s$ of the time, so that part now takes $pT/s$. The unaccelerated part still takes $(1-p)T$. The two run in sequence, so the times add:

$$T' = (1-p)T + \frac{p}{s}T = T\left((1-p) + \frac{p}{s}\right).$$

Since $s \ge 1$ and $p \le 1$ we have $(1-p) + p/s > 0$ (excluding $p=1, s=\infty$), so $T' > 0$, and

$$S(p,s) = \frac{T}{T'} = \frac{1}{(1-p) + \dfrac{p}{s}}$$

follows.

∎

From this comes the most important consequence in this article.

Corollary 3.2（The ceiling of automation）

In the setting of Proposition 3.1 with $0 < p < 1$, however large the speed factor is made,

$$\lim_{s \to \infty} S(p, s) = \frac{1}{1-p},$$

and moreover $S(p,s) < 1/(1-p)$ for every finite $s \ge 1$.

Proof（Corollary 3.2）

As $s \to \infty$ we have $p/s \to 0$, so the denominator in Proposition 3.1 converges to $(1-p)$. Since $p < 1$ gives $1-p > 0$, the limit of the quotient is $1/(1-p)$.

If $s \ge 1$ is finite, then $p > 0$ gives $p/s > 0$, so the denominator is strictly greater than $1-p$, whence $S(p,s) < 1/(1-p)$. (For $p = 0$ both sides equal $1$ and the inequality becomes an equality; that case is trivial, which is why we assumed $p > 0$.)

∎

Put differently: the fraction of work that does not get faster determines, by itself, the reciprocal of the headroom for the whole. If implementation is 30% of the total, then even reducing implementation time to zero makes the whole only $1/0.7 \approx 1.43$ times faster. Without touching the other 70%, no more than that is available even in principle.

Example 3.3（Extrapolating the Copilot RCT numbers to the whole）

In the RCT of Peng et al. [6], the group using GitHub Copilot completed the task of implementing an HTTP server in JavaScript in 55.8% less time than the group that did not. Converted to a speed factor,

$$s = \frac{1}{1 - 0.558} = \frac{1}{0.442} \approx 2.26.$$

The point to notice is that this task isolates the implementation stage. Framing and specification were supplied in advance by the experimenters.

So let us provisionally set the share of implementation in real work at $p = 0.3$ and apply Proposition 3.1 (this $0.3$ is an assumption, not a measurement):

$$S(0.3,\ 2.26) = \frac{1}{0.7 + \dfrac{0.3}{2.26}} = \frac{1}{0.7 + 0.1327} = \frac{1}{0.8327} \approx 1.20.$$

Overall this is about a factor of 1.20, that is, roughly 17% less time. Even with $p = 0.5$,

$$S(0.5,\ 2.26) = \frac{1}{0.5 + 0.2212} = \frac{1}{0.7212} \approx 1.39,$$

only a factor of 1.39. That is the size of the gap between the fact that implementation became 2.26 times faster and the claim that development became 2.26 times faster.

The next figure plots the overall speedup against the automation ratio $p$, for the speed factor $s = 2.26$ and for $s = \infty$.

Automation ratio p versus overall speedup. The solid curve is s → ∞ (the ideal in which implementation is instantaneous); the dashed curve is s = 2.26.

In the left half of the graph, the region where $p$ is at most $0.5$, even the unrealistic ideal $s = \infty$ does not push the speedup past a factor of $2$. The payoff from investing in automation is governed by the automation ratio itself.

4. Generation and verification: what gets cheap and what gets expensive

Corollary 3.2 says that the other stages determine the headroom. So which stage becomes relatively heavy once generation is cheap? The answer is verification. To see it, we model the generate-verify loop. Counting the cost of generation and the cost of verification separately is the same framework as the delegation cost model（Definition 2.1）［LLMs and Programming］, restricted here to a single unit of work.

Definition 4.1（The generate-verify cycle）

For a single unit of work (one function, one change) we assume the following.

• Each act of generation costs $g \ge 0$.
• Deciding whether the generated artefact meets the specification costs $v > 0$.
• Each generation meets the specification with probability $q \in (0, 1]$, independently across attempts.
• Verification decides correctly (no false negatives and no false positives).
• If the artefact is judged not to meet the specification, generation is retried.

We write $E(g, v, q)$ for the expected total cost until the first acceptance.

Proposition 4.2（Expected cost of the generate-verify cycle）

Under the assumptions of Definition 4.1, the number of attempts $N$ until acceptance follows a geometric distribution with success probability $q$, so $\mathbb{E}[N] = 1/q$ and therefore

$$E(g, v, q) = \frac{g + v}{q}.$$

Proof（Proposition 4.2）

Each attempt succeeds independently with probability $q$, so $N = k$ occurs when the first $k-1$ attempts fail and the $k$-th succeeds:

$$\Pr[N = k] = (1-q)^{k-1} q \qquad (k = 1, 2, \ldots).$$

Setting $x = 1 - q \in [0, 1)$, the expectation is

$$\mathbb{E}[N] = \sum_{k=1}^{\infty} k\, q\, x^{k-1} = q \sum_{k=1}^{\infty} k\, x^{k-1}.$$

For $|x| < 1$ the geometric series $\sum_{k=0}^{\infty} x^k = 1/(1-x)$ may be differentiated termwise in $x$, giving

$$\sum_{k=1}^{\infty} k\, x^{k-1} = \frac{d}{dx}\left(\frac{1}{1-x}\right) = \frac{1}{(1-x)^2}$$

(this step is legitimate because a power series may be differentiated termwise inside its radius of convergence). Substituting $1 - x = q$,

$$\mathbb{E}[N] = q \cdot \frac{1}{q^2} = \frac{1}{q}.$$

Each attempt costs the constant amount $g + v$, generation plus verification, so the total cost is $(g+v)N$. By linearity of expectation,

$$E(g,v,q) = (g+v)\,\mathbb{E}[N] = \frac{g+v}{q}.$$∎

Corollary 4.3（The verification floor）

In the setting of Proposition 4.2, however small the generation cost $g$ is made,

$$E(g, v, q) \ge \frac{v}{q} \ge v,$$

and $E(g,v,q) \to v/q$ as $g \to 0$. That is, the lower bound on the expected total cost is determined not by generation but by verification and the acceptance probability alone.

Proof（Corollary 4.3）

From $g \ge 0$ and $q > 0$ we get $(g+v)/q \ge v/q$. From $q \le 1$ we get $1/q \ge 1$, hence $v/q \ge v$. The limit as $g \to 0$ is $(0+v)/q = v/q$, which follows from continuity in $g$.

∎

This corollary is the core claim of the article. In a world where generation is effectively free, the cost is determined solely by $v$, the weight of verification, and $q$, the probability of passing on the first try. And besides making the model smarter, there are two ways to raise $q$: making the specification precise (Section 6) and designing a structure that is easy to verify (Section 7). Both are human work. Note also that when $v$ is large enough, the floor itself exceeds the cost of doing the work oneself, and delegation stops paying off in principle (Corollary 3.2［LLMs and Programming］).

Definition 4.4（pass@k）

Draw $k$ solutions independently from the model for a single task; the probability that at least one of them passes verification is called pass@k [5]. Writing $q$ for the probability that one sample passes, and assuming the samples are independent,

$$\mathrm{pass}@k = 1 - (1-q)^k$$

(so $\mathrm{pass}@1 = q$).

Example 4.5（pass@k rises, but so does the number of verifications）

In the evaluation of Chen et al., Codex-12B solved HumanEval tasks 28.8% of the time with one sample and 70.2% of the time with 100 samples [5]. Feeding $q = 0.288$ into the independence model of Definition 4.4 gives

$$1 - (1 - 0.288)^{100} = 1 - 0.712^{100},$$

and since $\ln(0.712^{100}) = 100 \ln 0.712 = 100 \times (-0.3398) = -33.98$, we get $0.712^{100} \approx e^{-33.98} \approx 1.7 \times 10^{-15}$, so pass@100 ought to be essentially $1$. The measured 70.2% is far below that. This is a symptom of the independence assumption failing: the model repeats the same misunderstanding over and over.

More importantly, this 70.2% is only the probability that a correct solution is somewhere among 100 candidates. To learn which one is correct, all 100 must be verified. In the language of Proposition 4.2, the strategy of lowering $g$ and increasing the number of attempts raises the total spent on $v$ by the same amount. That the economics do not change however many regenerations are stacked up is shown in another form in regeneration does not move the break-even point（Proposition 3.3）［LLMs and Programming］. In domains where verification is not automated, this strategy does not work.

5. Verification cannot be fully automated (Rice’s theorem)

The obvious objection is: let the AI do the verification too. In part that is right — test generation and static analysis keep improving. But a machine that decides in general whether a program meets a specification does not exist, whether or not it is an AI. This is what Rice’s theorem of 1953 [4] says.

Theorem 5.1（Rice's theorem）

Let $\varphi_0, \varphi_1, \ldots$ be an (acceptable) numbering of the partial computable functions. Let $\mathcal{P}$ be a set of partial computable functions that is non-trivial, that is, $\mathcal{P} \ne \varnothing$ and $\mathcal{P}$ is not the set of all partial computable functions. Then the index set

$$I_{\mathcal{P}} = \{\, e \in \mathbb{N} : \varphi_e \in \mathcal{P} \,\}$$

is undecidable.

Proof（Theorem 5.1）

Write $\xi$ for the partial function that is undefined everywhere.

First suppose $\xi \notin \mathcal{P}$. Since $\mathcal{P} \ne \varnothing$ we may pick a partial computable function $\psi \in \mathcal{P}$. Fix an index $b$ with $\psi = \varphi_b$.

For an arbitrary instance $(e, x)$ of the halting problem, consider the following two-argument partial computable function:

$$h(\langle e, x\rangle, y) = \begin{cases} \varphi_b(y) & \text{if } \varphi_e(x) \text{ halts} \\ \text{undefined} & \text{otherwise} \end{cases}$$

The function $h$ is computable. Indeed, on input $(\langle e,x\rangle, y)$ the procedure “first run $\varphi_e(x)$ until it halts, and once it halts run $\varphi_b(y)$ and output the result” computes $h$ directly.

By the s-m-n theorem there is a total computable function $f$ such that $\varphi_{f(e,x)}(y) = h(\langle e,x\rangle, y)$ for all $e, x, y$. Then:

• If $\varphi_e(x)$ halts, then $\varphi_{f(e,x)}(y) = \varphi_b(y)$ for every $y$, so $\varphi_{f(e,x)} = \psi \in \mathcal{P}$, that is, $f(e,x) \in I_{\mathcal{P}}$.
• If $\varphi_e(x)$ does not halt, then $\varphi_{f(e,x)}$ is undefined at every $y$, so $\varphi_{f(e,x)} = \xi \notin \mathcal{P}$, that is, $f(e,x) \notin I_{\mathcal{P}}$.

Hence if $I_{\mathcal{P}}$ were decidable, then together with the total computability of $f$ it would be decidable whether $\varphi_e(x)$ halts, contradicting the undecidability of the halting problem. Therefore $I_{\mathcal{P}}$ is undecidable.

Now suppose $\xi \in \mathcal{P}$. Consider the complement $\mathcal{P}^{c}$, the set of all partial computable functions not in $\mathcal{P}$. Since $\mathcal{P}$ is non-trivial, so is $\mathcal{P}^{c}$, and $\xi \notin \mathcal{P}^{c}$. By the previous paragraph $I_{\mathcal{P}^{c}}$ is undecidable. But $I_{\mathcal{P}^{c}} = \mathbb{N} \setminus I_{\mathcal{P}}$, and the complement of a decidable set is decidable, so assuming $I_{\mathcal{P}}$ decidable yields a contradiction. Hence $I_{\mathcal{P}}$ is undecidable in this case as well.

∎

Remark 5.2

What Rice’s theorem concerns are extensional properties, those determined solely by the function a program computes. A syntactic property such as “does the source code have fewer than 100 lines?” is of course decidable, and this is no contradiction. On the other hand “does this function halt on every input?”, “does this function return a sorted list?” and “does this function satisfy specification $S$?” are all non-trivial extensional properties, hence undecidable.

The theorem also does not say that nothing can be learned about individual programs. Type systems, contracts, tests and model checking all obtain practical precision either by making the verdict conservative (answering “unknown” or “reject” when unsure) or by restricting the class of programs considered. What the theorem forbids is only the existence of a complete, sound, always-halting universal decision procedure.

The practical implication is this. Verification will forever be an assemblage of approximations designed by humans. Which tests to write, which invariants to express in types, which properties to assert: deciding these cannot in principle be moved to the generation side. The more code AI writes, the higher the price of an engineer as a designer of verification.

6. The information content of framing: specifications cannot be compressed

We have seen that verification is what bites. Next comes raising $q$, the probability of passing on the first try, which is a matter of the precision of the specification. Everyone knows intuitively that a vague instruction will not produce what one wants; this can be stated exactly by counting.

Definition 6.1（Specification code）

Let $B$ be a finite set of realisable behaviours with $|B| = N$. Regard an instruction (a prompt) as a finite bit string and fix a deterministic map $D : \{0,1\}^{*} \to B$ from instructions to behaviours ($D$ being the model together with the decoding procedure). When $D$ is surjective, that is, when every behaviour in $B$ is obtained by some instruction, we call $D$ a specification code for $B$. We write $|w|$ for the length of an instruction $w$.

Proposition 6.2（Lower bound on the description length of a specification）

Let $D$ be a specification code for $B$ with $|B| = N$ in the sense of Definition 6.1. For each $b \in B$ put $\ell(b) = \min\{|w| : D(w) = b\}$, the length of a shortest instruction realising $b$. Then

$$\max_{b \in B} \ell(b) \ \ge \ \log_2 N - 1 \qquad \text{more precisely} \quad \max_{b \in B} \ell(b) \ \ge \ \lceil \log_2 (N+1) \rceil - 1.$$

In particular, if only instructions of length at most $\ell$ are used, at most $2^{\ell + 1} - 1$ behaviours are reachable.

Proof（Proposition 6.2）

Count the bit strings of length at most $\ell$. There are exactly $2^j$ strings of length $j$, so the total over lengths $0$ through $\ell$ is

$$\sum_{j=0}^{\ell} 2^{j} = 2^{\ell+1} - 1$$

(a geometric sum). Since $D$ is a map, the behaviours obtained from instructions of length at most $\ell$ number at most this many.

Now put $L = \max_{b \in B} \ell(b)$. Every element of $B$ is then obtained by an instruction of length at most $L$. Surjectivity of $D$ gives $N \le 2^{L+1} - 1$, that is, $2^{L+1} \ge N + 1$. Taking $\log_2$ of both sides, $L + 1 \ge \log_2 (N+1)$, and since $L$ is an integer, $L \ge \lceil \log_2(N+1)\rceil - 1$. Furthermore $\log_2(N+1) > \log_2 N$, so $L > \log_2 N - 1$, whence $L \ge \log_2 N - 1$.

∎

The proposition looks like it is stating the obvious, but its implication is sharp. The length of the instruction must be at least the logarithm of the number of behaviours one wishes to distinguish. However smart the model becomes, this lower bound does not move. What intelligence can change is only which behaviours get the short codewords, that is, the quality of the defaults. A good model is one that assigns short codes to the behaviours most people want.

Example 6.3（Counting the design decisions of a single API in bits）

Consider building one paginated list endpoint. Here are the binary decisions that must be settled before implementation.

1. Offset-based or cursor-based pagination?
2. Is a request exceeding the maximum page size an error, or is it clamped to the maximum?
3. Are deleted records included or excluded?
4. When the sort key ties, is a stable secondary key added or not?
5. Is a total count returned or not?
6. Is authorisation per record or per collection?
7. Is an empty result a 200 with an empty array, or a 404?
8. Are unknown query parameters ignored, or a 400?
9. Are timestamps fixed to UTC, or in the requested time zone?
10. Is the rate limit per user or per API key?
11. Is the response cacheable or not?
12. On partial failure, is what was retrieved returned, or does the whole request fail?

That is $2^{12} = 4096$ possibilities. By Proposition 6.2, distinguishing them requires at least 12 bits of instruction. The English sentence “build me an API that returns a list of users” certainly carries far more than 12 bits of information, but almost none of it concerns these 12 axes. The model therefore has no option but to fill them with defaults, and some of those will conflict with the intended specification. This is exactly how $q$ in Proposition 4.2 gets driven down.

Conversely, the work of enumerating these 12 items and settling them is precisely where the value lies. Outwardly it looks the same as “writing a prompt”, but in substance it is framing and specification themselves.

Remark 6.4

If prompt engineering is understood as a bag of tricks for fooling the model, it becomes worthless with the next model generation. From the viewpoint of Proposition 6.2, its real content is the encoding of the behaviour space: enumerating which axes split the space into how many cases, and conveying that in the shortest description. This skill does not decay when the model changes. What decays is only the part that depended on the quirks of a particular model.

For a fuller discussion see LLMs and programming. In particular, the conditions under which delegation pays off once the labour of making instructions precise is included are collected in Proposition 3.1［LLMs and Programming］.

7. Architecture: what does decomposition save?

Finally we turn to lowering $v$, the cost of verification. The largest factor making verification heavy is the interaction between parts. The criteria for module decomposition stated by Parnas in 1972 [3] were aimed at exactly this. Here too, counting quantifies the matter.

Proposition 7.1（Reduction of the objects of inspection by module decomposition）

Suppose a system consists of $n \ge 2$ elements. Without decomposition, if any two elements may interact directly, the number of pairs to inspect is $\binom{n}{2}$.

Now let $m$ divide $n$, split the elements into $m$ modules of size $n/m$, and assume (i) within a module any two elements may interact, and (ii) interaction between distinct modules is concentrated into a single interface per pair of modules. Then the number of objects to inspect is

$$I(m) = m \binom{n/m}{2} + \binom{m}{2} = \frac{n^2}{2m} - \frac{n}{2} + \frac{m^2}{2} - \frac{m}{2}.$$

Regarding $I$ as a function of a real variable $m > 0$, it is strictly convex, its minimiser $m^{*}$ is the unique positive solution of $m^3 - m^2/2 = n^2/2$, and $m^{*} \sim (n^2/2)^{1/3}$ for large $n$. At that point

$$I(m^{*}) = \Theta\!\left(n^{4/3}\right),$$

which is strictly smaller than the $\Theta(n^2)$ of the undecomposed case.

Proof（Proposition 7.1）

First we check the formula for $I(m)$. Each module has $n/m$ elements, so it contains $\binom{n/m}{2} = \frac{(n/m)(n/m - 1)}{2}$ pairs, and there are $m$ modules, giving

$$m \cdot \frac{(n/m)(n/m-1)}{2} = \frac{n(n/m - 1)}{2} = \frac{n^2}{2m} - \frac{n}{2}.$$

The pairs of modules number $\binom{m}{2} = \frac{m(m-1)}{2} = \frac{m^2}{2} - \frac{m}{2}$, and the sum is the claimed formula.

Next, convexity. For $m > 0$,

$$I'(m) = -\frac{n^2}{2m^2} + m - \frac{1}{2}, \qquad I''(m) = \frac{n^2}{m^3} + 1 > 0,$$

so $I$ is strictly convex. Hence $I'(m) = 0$ has at most one solution, and that solution is the minimiser. Rearranging $I'(m) = 0$,

$$m - \frac{1}{2} = \frac{n^2}{2m^2} \iff m^3 - \frac{m^2}{2} = \frac{n^2}{2}.$$

The map $m \mapsto m^3 - m^2/2$ is continuous and strictly increasing for $m \ge 1/2$ and diverges to $\infty$ as $m \to \infty$, so there is exactly one positive solution $m^{*}$. For large $n$ the value $m^{*}$ is also large, so the term $m^2/2$ is negligible against $m^3$, giving $m^{*} \sim (n^2/2)^{1/3}$.

Writing $m^{*} = c\, n^{2/3}$ with $c = 2^{-1/3}$, we get

$$\frac{n^2}{2m^{*}} = \frac{n^2}{2c\,n^{2/3}} = \frac{1}{2c}\,n^{4/3}, \qquad \frac{(m^{*})^2}{2} = \frac{c^2}{2}\,n^{4/3},$$

while the remaining $-n/2 - m^{*}/2$ is $O(n)$, hence $o(n^{4/3})$. Therefore $I(m^{*}) = \left(\frac{1}{2c} + \frac{c^2}{2}\right) n^{4/3} + o(n^{4/3}) = \Theta(n^{4/3})$. Since $n^{4/3} = o(n^2)$, this is strictly smaller than the undecomposed case.

∎

Example 7.2（The concrete saving at n = 100）

Take $n = 100$. Without decomposition one inspects

$$\binom{100}{2} = \frac{100 \times 99}{2} = 4950$$

pairs.

With $m = 10$ (ten elements per module),

$$I(10) = 10\binom{10}{2} + \binom{10}{2} = 10 \times 45 + 45 = 495,$$

exactly one tenth.

The optimum solves $m^3 - m^2/2 = 100^2/2 = 5000$; at $m = 17$ we get $17^3 - 17^2/2 = 4913 - 144.5 = 4768.5$ and at $m = 18$ we get $5832 - 162 = 5670$, so $m^{*}$ lies between 17 and 18. Evaluating at the integer,

$$I(17) = \frac{10000}{34} - 50 + \frac{289}{2} - \frac{17}{2} = 294.1 - 50 + 144.5 - 8.5 = 380.1$$

(an approximation, since $17$ does not divide $100$). Roughly 380 pairs, one thirteenth of the undecomposed case.

What this calculation shows is that decomposition is not a case of “the more the better”. Making $m$ too large lets the number of inter-module interfaces $\binom{m}{2}$ take over. At $m = 50$ we get $I(50) = 100 - 50 + 1250 - 25 = 1275$, worse than $m = 17$. This is the simplest model of the pain experienced by organisations that have carved their microservices too finely.

Now the connection to LLMs. The length of context a model can see at once is finite. A design with small $I(m)$ — one in which understanding a module requires reading only that module and a few interfaces — is at the same time a design in which the model is likely to emit correct code. In the language of Proposition 4.2, a good decomposition lowers $v$ and raises $q$ simultaneously. Far from losing value in the AI era, architectural design has become a parameter that determines the AI’s performance.

8. What to learn

We now translate the four results above into a priority order for learning.

Skill	Grounds	Degree of replacement by generation
Framing (what to solve)	Corollary 3.2: the unautomated stages set the ceiling	Low. Purposes are not supplied from outside
Specification (removing ambiguity)	Proposition 6.2: the lower bound on description length is model-independent	Low. Only the quality of defaults improves
Architectural design	Proposition 7.1: lowers $v$ and raises $q$	Low. Requires grasping the constraints
Designing verification (tests, types, invariants)	Corollary 4.3, Theorem 5.1	Low. Full automation is impossible in principle
Routine implementation (CRUD, transformations, boilerplate APIs)	Example 3.3	High. This is what got faster
Memorising syntax and library APIs	—	Very high
Low-level understanding (memory, concurrency, performance)	A prerequisite for verification; needed to judge the soundness of generated code	Low
Mathematical foundations (discrete mathematics, probability, complexity, linear algebra)	The tools for articulating invariants and lower bounds	Low

A word about the last two rows. Low-level knowledge and mathematics matter not because AI cannot do them, but because they are the language of verification.

Judging whether generated code is correct requires a vocabulary for stating correctness. “This loop is $O(n^2)$, so it will not run at $n = 10^6$.” “This shared counter has a data race.” “This floating-point summation loses precision through cancellation.” “The error probability of this randomised algorithm is bounded by $2^{-k}$.” None of these judgements emerges from reading the code alone. Only someone holding a model — of complexity, of the memory model, of numerical analysis, of probability — can make them. As Corollary 4.3 says, what sets the floor on cost is verification. Someone without the vocabulary of verification cannot lower that floor.

The role of mathematics is treated in more detail in Why relearn mathematics. For instance, how to read a warning with 99% detection rate and 1% false-positive rate is handled in Example 5.4［Relearning Mathematics］, and the condition on the learning rate under which gradient descent does not diverge in Theorem 4.3［Relearning Mathematics］.

Caution

Every analysis in this article is a model calculation under explicitly stated assumptions. Neither $p = 0.3$, nor the independence of generations, nor the perfection of verification is a measurement. Rather than the numbers themselves, take away the structure of which quantity sets the ceiling. Changing the assumptions does not change the shape of Corollary 3.2 or Corollary 4.3.

9. Exercises

Exercise 9.1Easy

In a certain development team, implementation accounts for 25% of the total working time.

(1) If introducing a code generation tool makes the implementation stage 5 times faster, find the overall speedup.

(2) Find the value that the overall speedup cannot exceed even if the implementation time is reduced to zero.

(3) Discuss how much the non-implementation stages must be reduced in order to make the whole twice as fast.

Solution

(1) Substitute $p = 0.25$ and $s = 5$ into Proposition 3.1:

$$S(0.25, 5) = \frac{1}{0.75 + \dfrac{0.25}{5}} = \frac{1}{0.75 + 0.05} = \frac{1}{0.80} = 1.25.$$

The speedup is a factor of 1.25 (a 20% reduction in time).

(2) By Corollary 3.2,

$$\lim_{s \to \infty} S(0.25, s) = \frac{1}{1 - 0.25} = \frac{1}{0.75} \approx 1.333,$$

so the ceiling is about a factor of 1.33. The 1.25 of part (1) already achieves 94% of this ceiling; making code generation faster still leaves only about 7% of headroom.

(3) Doubling the whole requires $T' = T/2$. Even with implementation reduced to zero, the remaining time is $0.75\,T$, which is greater than $T/2$. Hence doubling is impossible in principle by speeding up implementation alone. On top of zeroing implementation, the remaining $0.75\,T$ must be cut further to $0.5\,T$, that is, by a third. Unless the other stages (framing, specification, design, verification, operation) are touched, a target of twice as fast cannot even be set.

Exercise 9.2Standard

In the model of Definition 4.1, take generation cost $g = 1$, verification cost $v = 4$ and success probability per attempt $q = 0.4$.

(1) Find the expected total cost until acceptance.

(2) By what percentage does the expected total cost fall when the generation cost becomes $g = 0$?

(3) By what percentage does it fall if instead the specification is made precise, raising $q$ from $0.4$ to $0.8$ (with $g = 1$ unchanged)? Compare with (2).

(4) If one adopts the strategy of generating $k = 5$ candidates independently and verifying them all, find the probability that at least one passes and the total verification cost.

Solution

(1) By Proposition 4.2,

$$E(1, 4, 0.4) = \frac{1 + 4}{0.4} = \frac{5}{0.4} = 12.5.$$

(2) With $g = 0$ we get $E(0,4,0.4) = 4/0.4 = 10$. The reduction is

$$\frac{12.5 - 10}{12.5} = \frac{2.5}{12.5} = 0.20,$$

that is 20%. As Corollary 4.3 says, $10$ is the floor, and no amount of cheapening generation gets below it.

(3) $E(1, 4, 0.8) = 5/0.8 = 6.25$. The reduction is

$$\frac{12.5 - 6.25}{12.5} = 0.50,$$

that is 50%. Doubling the success rate (50% off) beats driving the generation cost to zero (20% off). This is a consequence of $q$ sitting in the denominator of Proposition 4.2 and $g$ in the numerator. Since the success rate is a quantity raised by the precision of the specification and the quality of the design (Example 6.3, Proposition 7.1), the human-side stages are the better place to invest.

(4) By Definition 4.4,

$$\mathrm{pass}@5 = 1 - (1 - 0.4)^5 = 1 - 0.6^5 = 1 - 0.07776 = 0.92224,$$

about 92.2%. However, learning which one passes requires verifying all five, so the verification cost is $5 \times 4 = 20$. That exceeds the expected total cost 12.5 of part (1). The strategy of “generate many and pick” is advantageous only where verification is cheap, that is, where automated tests are complete.

Exercise 9.3Standard

Suppose the behaviour of a system is determined by 20 independent binary design decisions, so that $N = 2^{20}$.

(1) Using Proposition 6.2, find the lower bound on the maximum shortest-instruction length needed to express every behaviour.

(2) If one decides to use only instructions of length at most 15 bits, find how many behaviours are at least unreachable.

(3) Translate the result of (2) into what “getting by with a short prompt” means in practice.

Solution

(1) Here $N = 2^{20} = 1048576$. By Proposition 6.2,

$$L \ge \lceil \log_2(N + 1) \rceil - 1 = \lceil \log_2 1048577 \rceil - 1 = 21 - 1 = 20$$

(the ceiling is 21 because $2^{20} = 1048576 < 1048577 \le 2^{21}$). So some behaviour requires an instruction of at least 20 bits.

(2) There are $2^{16} - 1 = 65535$ bit strings of length at most 15 (the geometric sum in the proof of Proposition 6.2 with $\ell = 15$). The image of the map $D$ is no larger than this, so at most 65535 behaviours are reachable. Hence the number of unreachable behaviours is at least

$$1048576 - 65535 = 983041,$$

about 93.8% of the whole.

(3) A short prompt yields only the behaviour obtained once the model’s defaults are filled in. Every one of the 20 decisions not specified explicitly is left to the model’s prior. If the defaults happen to match one’s intent, fine; but a single mismatched decision makes the artefact fail, which lowers $q$ in Proposition 4.2.

So the strategy of “instruct briefly and iterate quickly” works only (i) in routine domains where the defaults align well with the intent, or (ii) in domains where verification is cheap and mismatches are detected at once. Elsewhere, enumerating the axes of decision and stating them explicitly turns out to be the fastest route after all.

References

1. G. M. Amdahl, “Validity of the single processor approach to achieving large scale computing capabilities”, AFIPS Spring Joint Computer Conference (1967), 483–485.
2. F. P. Brooks, Jr., “No Silver Bullet: Essence and Accidents of Software Engineering”, IEEE Computer 20(4) (1987), 10–19.
3. D. L. Parnas, “On the Criteria To Be Used in Decomposing Systems into Modules”, Communications of the ACM 15(12) (1972), 1053–1058.
4. H. G. Rice, “Classes of recursively enumerable sets and their decision problems”, Transactions of the American Mathematical Society 74 (1953), 358–366.
5. M. Chen et al., “Evaluating Large Language Models Trained on Code”, arXiv:2107.03374 (2021). arXiv
6. S. Peng, E. Kalliamvakou, P. Cihon, M. Demirer, “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot”, arXiv:2302.06590 (2023). arXiv

Appendix: when verification is imperfect

In the body we assumed that verification never errs, but real verification — tests, reviews — does miss things. Here we estimate the effect.

Drop the assumption “verification decides correctly” from Definition 4.1 and replace it as follows. Let $\alpha \in [0,1)$ be the probability that a faulty artefact is wrongly judged acceptable (the miss rate), and let the probability that a correct artefact is wrongly rejected be $0$. This $\alpha$ is the same quantity as the false positive rate of the verifier（Definition 2.2）［LLMs and Programming］. The probability that one generation genuinely meets the specification remains $q$.

The probability that one attempt is judged acceptable is then the sum of the probability $q$ that it is genuinely correct and passes, and the probability $(1-q)\alpha$ that it is faulty and slips through, namely $q + (1-q)\alpha$. By the same computation as Proposition 4.2, the expected cost until a verdict is reached is

$$\frac{g + v}{q + (1-q)\alpha},$$

so the apparent cost falls as $\alpha$ grows. But the conditional probability that an artefact is actually correct given that a verdict of acceptance has been reached is, by Bayes’ theorem,

$$\Pr[\text{correct} \mid \text{accepted}] = \frac{q}{q + (1-q)\alpha}.$$

This is nothing other than Theorem 7.2［LLMs and Programming］ rewritten in the notation of this article. For instance with $q = 0.5$ and $\alpha = 0.3$ we get $0.5 / (0.5 + 0.15) \approx 0.769$, so about 23% of what was accepted is in fact wrong.

These errors do not vanish; they move to the operation stage. That is, $T_{\text{ops}}$ in Definition 2.1 grows. And the cost of responding to an incident is generally far higher than the cost of a fix during development. Lowering $\alpha$ — through test coverage, review quality, static guarantees from types — is therefore worth doing even at the price of apparent speed. The “floor” of Corollary 4.3 looks lower than it really is when measured while ignoring $\alpha$. This is why quality metrics deserve to be re-measured in an era of fast generation.