# LLMs and Programming: The Break-Even Point of Delegation and How to Spot Plausible Errors > Reduces the decision to delegate work to an LLM to one inequality between success rate and verification cost, explains why splitting tasks helps, and shows how to catch plausible errors. > https://rikai.mugen-giken.com/en/computer-science/ai-era/llm-and-programming ## 0. Key points - Whether to hand a task to an LLM is not decided by how clever the model is. It is decided by three quantities: the **success rate $p$, the cost of writing the instructions, and the cost of verifying the result**. Delegation pays exactly when $p > g + v$, where $g$ is the instruction ratio and $v$ the verification ratio (). - From this inequality follows the conclusion that **a task whose verification costs about as much as doing the work yourself never pays to delegate, however good the model becomes** (). Designing a genuinely new architecture is unsuited to an LLM not because models are weak but because $v$ is close to $1$. - Adopting the strategy "regenerate on failure" does not move the break-even point (). Regeneration helps only when verification is trustworthy. - The probability that an adopted artefact is wrong is roughly proportional to the **false positive rate $\alpha$** of the verifier (). "Seeing through the lies of an AI" is precisely the craft of lowering $\alpha$. - Throwing a large task over the wall in one piece inflates the expected cost **exponentially** in the number of stages, whereas verifying each stage keeps it linear (). This is the quantitative case for "cut it small before handing it over". ## 1. Motivation: why opinions split on whether LLMs work People who have used an LLM for programming report startlingly polarised experiences. "My implementation speed doubled" and "I rewrote all of it in the end, so it was slower" are said by people on the same team at the same company. The split is reproduced under measurement. In a controlled experiment where subjects implemented an HTTP server in JavaScript with GitHub Copilot, the assisted group finished 55.8% faster (Peng et al., 2023). In an experiment where experienced developers worked on large open-source repositories they knew intimately, the AI-assisted group was 19% *slower*, and the subjects themselves believed they had been faster (METR, 2025). The first task had a clear specification and easy confirmation of correctness; in the second, the subjects knew the code deeply and held a high quality bar. The thesis of this article is that this difference is what separates the results. If the same technology produces opposite outcomes, then the question "are LLMs useful in development?" is malformed. The right question is **"on tasks with what properties are they useful, and on tasks with what properties are they harmful?"** And unless we identify variables that pin down those properties, the discussion degenerates into trading anecdotes. This article narrows the variables down to three. Deciding whether to hand work outside is, in economic terms, a problem of transaction costs. However capable the contractor, outsourcing loses money whenever **the effort of explaining the specification plus the effort of inspecting the delivered work** exceeds doing it oneself. LLMs are no exception. What is different is that inspection tends to cost more with an LLM than with a human contractor: a human subcontractor asks about the parts they do not understand, whereas an LLM **fills them in plausibly** and hands the result back. Below we reduce this structure to a minimal model and derive three things from it in turn: what LLMs are good and bad at, why task decomposition works, and what debugging technique this calls for. For the background picture of the engineer's role, see [Survival strategies for IT engineers in the age of AI](/en/computer-science/ai-era/engineer-survival-strategy). In particular, how far per-stage automation can push the overall reduction in time is treated in . ## 2. Preliminaries: modelling delegation by cost The object of study is "one self-contained unit of work". Write one function, write a suite of tests, produce an API specification: that is the granularity to keep in mind. For a unit of work $T$, define the following quantities. The unit may be time or money, provided it is kept fixed throughout. - $c_h > 0$: the cost of a human producing $T$ from scratch (design, writing and self-checking included). - $c_g \ge 0$: the cost of writing the instructions that ask the LLM for $T$ (the prompt and the context supplied). - $c_v \ge 0$: the cost of verifying the LLM's output. Reading it, running it, testing it and matching it against the specification, all summed. - $p \in [0, 1]$: the probability that the output passes verification (the **success rate**). Define further the ratios $$ g = \frac{c_g}{c_h}, \qquad v = \frac{c_v}{c_h} $$ and call them the **instruction ratio** and the **verification ratio**. Dividing by $c_h$ to make the quantities dimensionless is the essential move. "Verification takes 20 minutes" means $v = 1/6$, a light burden, on a task that takes 2 hours to do by hand; on a task that takes 25 minutes it means $v = 0.8$, a crushing one. What matters is the ratio, not the absolute time. Verification is treated as a binary pass/fail judgement. We call the agent that makes this judgement a verifier. A procedure that takes an output and returns "pass" or "fail" is called a **verifier**. Test suites, type checkers, static analysers, human review, and combinations of these all qualify. For a verifier we consider - the **false negative rate**: the probability that a correct output is failed. For simplicity we assume it to be $0$ below. - the **false positive rate $\alpha \in [0, 1]$**: the probability that an incorrect output is passed. Here $\alpha$ measures **the size of the holes** in the verifier. With only three test cases $\alpha$ is large; with boundary values, degenerate cases and concurrency all covered, it is small. ## 3. The break-even point of delegation ### 3.1. A single attempt Under , consider the following strategy. > Ask the LLM once and verify the output. If it passes, adopt it and stop. If it fails, discard the output and have a human produce the work from scratch. The expected cost $C_1$ of this strategy is $$ C_1 = c_g + c_v + (1 - p)\,c_h $$ and $C_1 < c_h$ holds if and only if $$ p > g + v . $$ Instructing and verifying happen regardless of the outcome, so the cost $c_g + c_v$ is incurred with probability $1$. The event in which a human additionally rewrites the work has probability $1 - p$ (by the definition of $p$ in , the probability of failing verification), and its cost is $c_h$. By linearity of expectation, $$ C_1 = (c_g + c_v) \cdot 1 + c_h \cdot (1 - p) = c_g + c_v + (1-p)c_h . $$ Now transform the inequality. $$ \begin{aligned} C_1 < c_h &\iff c_g + c_v + (1-p)c_h < c_h \\ &\iff c_g + c_v < c_h - (1-p)c_h = p\,c_h . \end{aligned} $$ Since $c_h > 0$ we may divide both sides by $c_h$, and the direction of the inequality is unchanged: $$ \frac{c_g}{c_h} + \frac{c_v}{c_h} < p , $$ that is, $g + v < p$ in the notation of . Every step is an equivalence, so the converse holds as well. Read aloud, the inequality says something obvious: **delegate when the effort of writing instructions plus the effort of inspection is less than the cost of doing it yourself discounted by the success rate.** Obvious as it is, its consequences are not. For a task with $g + v \ge 1$, we have $C_1 \ge c_h$ however high the success rate $p$ is, so delegation never pays. By definition of a probability, $p \le 1$. The hypothesis gives $g + v \ge 1 \ge p$, so $p > g+v$ fails. Since is an equivalence, $C_1 < c_h$ fails as well. is a short statement, but it is the most important conclusion in this article. Note that $p$ does not appear in it. In other words, **no amount of improvement in model performance breaks through this wall.** What can break through is lowering $v$ (making verification cheaper — though how far it can fall is limited by ) or raising $c_h$ (choosing tasks that are genuinely laborious to do by hand). ### 3.2. Repeated attempts "Throw it away if the first try fails" is not realistic. Normally one adds constraints and regenerates (). Even so, the boundary does not move. Under , suppose the generations are mutually independent and each has success rate $p \in (0,1]$. Consider the following strategy. > Repeat generation and verification up to $k$ times, stopping as soon as an output passes. If all $k$ attempts fail, have a human produce the work from scratch. The expected cost is then $$ C_k = (c_g + c_v)\,\frac{1 - (1-p)^k}{p} + (1-p)^k c_h $$ and the sequence $(C_k)_{k \ge 1}$ is - strictly decreasing when $p > g + v$, with limit $(c_g + c_v)/p < c_h$ as $k \to \infty$; - strictly increasing when $p < g + v$, with $C_k > c_h$ for every $k$; - constant when $p = g + v$, with $C_k = c_h$ for every $k$. In particular, $C_k < c_h$ for some $k$ if and only if $p > g+v$. The $i$-th generation is carried out only if attempts $1$ through $i-1$ all failed, which by independence has probability $(1-p)^{i-1}$. Whenever the $i$-th attempt is carried out, the cost $c_g + c_v$ is paid, so by linearity of expectation the expected cost of generation and verification is $$ \sum_{i=1}^{k} (1-p)^{i-1}(c_g + c_v) = (c_g+c_v)\,\frac{1-(1-p)^k}{1-(1-p)} = (c_g+c_v)\,\frac{1-(1-p)^k}{p} $$ (a geometric sum with ratio $1-p$; the denominator is nonzero because $p \ne 0$). The human rewrites only when all $k$ attempts fail, which has probability $(1-p)^k$ and cost $c_h$. Adding the two gives the stated formula. Monotonicity follows from the difference: $$ \begin{aligned} C_{k+1} - C_k &= (c_g+c_v)\,\frac{(1-p)^k - (1-p)^{k+1}}{p} + \left[(1-p)^{k+1} - (1-p)^k\right] c_h \\ &= (c_g+c_v)\,\frac{(1-p)^k \, p}{p} - (1-p)^k p \, c_h \\ &= (1-p)^k \left[(c_g + c_v) - p\,c_h\right]. \end{aligned} $$ Here $(1-p)^k > 0$ when $p < 1$. (If $p = 1$ then $C_k$ is the constant $c_g+c_v$ for $k \ge 1$ and the claim reduces to .) Hence the sign of the difference agrees with that of $(c_g+c_v) - p\,c_h$, which by the manipulation in the proof of agrees with the comparison between $g + v$ and $p$. As $k \to \infty$ we have $(1-p)^k \to 0$ because $|1-p| < 1$, so $C_k \to (c_g+c_v)/p$. When $p > g+v$, that is $c_g + c_v < p\,c_h$, this limit is smaller than $c_h$. That $C_1 = c_h$ exactly when $p = g+v$ follows from , and monotonicity then determines the whole picture. The practical implication is clear. **Whether to keep pushing after the first failure is not where the decision lies.** Tasks worth pushing on were worth delegating from the first attempt, and tasks not worth pushing on should not have been delegated at all. This is the true shape of the familiar experience of regenerating three or four times on the feeling that "it is almost working", only to realise that writing it yourself would have been faster. In the region $p < g+v$, retrying only makes things worse. ### 3.3. The decision on a single diagram With $p$ on the horizontal axis and $v$ on the vertical, the condition of becomes the half-plane below the line $v = p - g$. The figure below takes the instruction ratio to be $g = 0.05$.
boilerplate code documentation test generation local code review complex concurrency novel architecture cheaper to write it yourself cheaper to delegate boundary v = p − g 0 0.5 1 success rate p verification ratio v
The coordinates of the dots are estimates, not measurements. The claim is not about the individual numbers but about the structure: **the decision is determined by where the task sits in the plane.** To learn where your own team actually sits, record for a while the time spent writing instructions, the time spent reviewing, and the number of rejections. Those three give estimates of $g$, $v$ and $p$. ## 4. A map of strengths and weaknesses Using and , we classify common activities. What the classification needs is not "how clever is the model" but **how cheaply the correctness of the task can be established**. | Task | Success rate $p$ | Verification ratio $v$ | Verdict | Why verification is cheap or expensive | |---|---|---|---|---| | Boilerplate code (DTO conversion, CRUD, configuration files) | high | low | delegate | Running it settles the matter; the specification is closed and unexpected inputs are rare | | Documentation, comments, commit messages | high | low | delegate | The standard of correctness lives in the code itself, so cross-checking is fast | | Implementing a known algorithm | medium to high | low | delegate | Can be matched against a reference implementation or known invariants | | Generating test code | medium | low to medium | delegate with conditions | It runs, but one still has to check *what it fails to test* | | Local code review (style violations, unhandled exceptions, known vulnerability patterns) | medium | low | delegate | Each remark can be adjudicated by looking at one place; a wrong remark costs little | | Mechanical refactoring of existing code | medium | medium | it depends | Entirely dependent on test coverage | | Implementing complex concurrent or distributed logic | low | high | do not delegate | Passing the tests does not establish correctness; non-reproducible faults survive | | Design decisions with performance requirements | low | high | do not delegate | Adjudication requires load testing or analysis, which is the work itself | | Deciding a genuinely novel architecture | low | $\approx 1$ | do not delegate | Judging the answer requires the same thinking as producing it | Look at the last three rows. What they share is not "low $p$" but "high $v$", and the distinction matters. If only $p$ were low, progress in models would fix it. When $v$ is high, says it will not be fixed. Consider writing frontend type definitions and conversion functions from the response types of an internal API. By hand this takes 90 minutes ($c_h = 90$); pasting in the API specification and writing the instructions takes 8 minutes ($c_g = 8$); reading the output, getting it through the type checker and running the existing tests takes 12 minutes ($c_v = 12$). $$ g + v = \frac{8 + 12}{90} = \frac{20}{90} \approx 0.222 . $$ Hence by , delegation pays as soon as the success rate exceeds $22.2\%$. In practice one can expect $p \approx 0.8$, so the condition holds with room to spare. With unlimited retries, gives an expected cost of $$ \frac{c_g + c_v}{p} = \frac{20}{0.8} = 25 \ \text{minutes}, $$ about $3.6$ times more efficient than the $90$ minutes of doing it by hand. Tasks of this kind are where the experience of "dramatically faster" comes from. Consider deciding whether to migrate a service that has run on a single database to a configuration partitioned by region and operated under eventual consistency. Reaching a conclusion yourself requires enumerating failure modes, taking stock of consistency requirements and producing a rough migration estimate; put $c_h = 40$ hours. Asking an LLM to "propose the best architecture for these requirements" takes 0.5 hours ($g = 0.0125$). The problem is verification. Judging whether the returned architecture meets the requirements means enumerating the failure modes and taking stock of the consistency requirements yourself after all. The only stage that becomes easier is *thinking of the options*, so we estimate $c_v = 35$ hours, that is $v = 0.875$. $$ g + v = 0.0125 + 0.875 = 0.8875 . $$ Unless $p > 0.8875$ — that is, unless **nine times out of ten the returned design is adoptable as it stands** — delegation loses. Real success rates are far below this, so the task is unsuited to delegation. Moreover, if the estimate of $c_v$ was optimistic and the true value is $c_v = 40$, then $g + v > 1$ and by no model whatsoever can rescue the situation. The conclusion of is not "do not use an LLM for design". It is **do not use it in the mode of delegation**. In the very same situation, asking "list twenty failure modes that could arise in this architecture" changes the picture entirely. The artefact is now a list of candidates, and each item can be adjudicated independently and quickly. That is, $v$ drops dramatically while $c_h$ — the effort of coming up with twenty items unaided — remains substantial. The point is to **translate a high-$v$ task into low-$v$ subtasks**. Instead of asking for the answer, ask it to close gaps or to widen the space of options. That, I think, is the right way to use an LLM in the design phase. ## 5. Cutting the task into pieces "Do not throw a large job over the wall at once; cut it small and hand over the pieces" is standard advice drawn from experience. It can be proved as a theorem. Suppose a task $T$ decomposes into $n$ stages $T_1, \dots, T_n$, and that for each stage the combined cost of generation and verification is $1$ (one unit of cost). Let $r \in (0, 1)$ be the probability that the output of a stage is correct, and assume that correctness across distinct stages and across retries of the same stage are mutually independent. We call $r$ the **per-stage accuracy**. Under , assume verification is perfect (false positive rate $\alpha = 0$ and false negative rate $0$), and compare the following two schemes. - **Batch scheme**: generate all $n$ stages at once and verify the whole thing at once. On failure, discard everything and regenerate from the start. Repeat until it passes. - **Sequential scheme**: generate $T_1, \dots, T_n$ in this order, verifying each stage immediately after generating it, and regenerating only that stage on failure. On success, proceed to the next stage. The expected total costs until completion are then $$ E_{\text{batch}} = \frac{n}{r^{\,n}}, \qquad E_{\text{seq}} = \frac{n}{r} $$ and their ratio is $$ \frac{E_{\text{batch}}}{E_{\text{seq}}} = \left(\frac{1}{r}\right)^{n-1} . $$ That is, the cost of the batch scheme grows **exponentially** in the number of stages $n$. We first record an auxiliary fact. If independent trials with success probability $s \in (0,1]$ are repeated until the first success, the number of trials $N$ is geometrically distributed and $\mathbb{E}[N] = 1/s$. Indeed, $\Pr[N = i] = (1-s)^{i-1}s$ gives $$ \mathbb{E}[N] = \sum_{i=1}^{\infty} i (1-s)^{i-1} s = s \cdot \frac{1}{(1-(1-s))^2} = \frac{1}{s} $$ (applying $\sum_{i \ge 1} i x^{i-1} = (1-x)^{-2}$, valid for $|x| < 1$, at $x = 1-s$). **Batch scheme.** The probability that all $n$ stages are correct is $r^n$ by the independence assumption in . Verification is perfect, so passing is equivalent to being correct overall, and one trial succeeds with probability $s = r^n$. A single trial generates and verifies all $n$ stages, at cost $n$. Hence the expected total cost is $n \cdot \mathbb{E}[N] = n / r^n$. **Sequential scheme.** Fix a stage $T_i$. One trial succeeds with probability $s = r$ and costs $1$, so the expected cost of completing $T_i$ is $1/r$. The stages are independent and each begins only after the previous one is settled, so the total cost is the sum of the per-stage costs. By linearity of expectation, $$ E_{\text{seq}} = \sum_{i=1}^{n} \frac{1}{r} = \frac{n}{r} . $$ Taking the ratio gives $\dfrac{n/r^n}{n/r} = r^{1-n} = (1/r)^{n-1}$. Take the per-stage accuracy to be $r = 0.98$ (a stage completed correctly with probability 98%, i.e. a rather good model) and compute the ratio $(1/0.98)^{n-1}$, using $\ln(1/0.98) = 0.020203$. | Stages $n$ | Exponent $(n-1)\ln(1/r)$ | Cost multiplier of the batch scheme | |---|---|---| | $10$ | $0.1818$ | $1.20\times$ | | $50$ | $0.9899$ | $2.69\times$ | | $200$ | $4.0204$ | $55.7\times$ | With ten stages, batching costs only 20% more, which is imperceptible in practice. With two hundred stages it costs 56 times as much. The phenomenon of "delightful on a small prototype, catastrophic on real feature work" is explained by this exponential. Even raising the model's $r$ from $0.98$ to $0.99$ leaves a multiplier of $(1/0.99)^{199} = e^{2.0} \approx 7.4$ at $n = 200$, still large. **Decomposition helps more than improving the model.** For the sequential scheme of to work, an error in stage $T_i$ must be detectable from $T_i$ alone. An error in which each function is individually correct but the pieces break when combined, because the interface was interpreted differently on the two sides, lies outside this assumption. In practice, therefore, stage decomposition must be accompanied by **verification at the seams**: integration tests, contract tests, boundaries pinned down by types. The gains from decomposition are bought at the price of defining the boundaries explicitly. How much the amount that must be inspected at once shrinks when boundaries are fixed is treated in . Conversely, in a codebase without clear boundaries, decomposition has little effect and the benefit of introducing an LLM is correspondingly small. ## 6. Fitting this into the development process With the above in hand, we set out concrete uses stage by stage. The policy is uniform: **translate the question into a low-$v$ form, cut $n$ small, and prepare verification in advance.**
B["the LLM generates a candidate"] B --> C{"run the verifier"} C -->|"fail"| B C -->|"pass"| D["adopt"] D --> E["genuinely correct"] D --> F["false positive: a plausible error survives"]`} />
**Design.** Do not ask for answers; ask for materials. "List three architectures that satisfy these requirements, each with its failure modes." "Where will this design break first?" "Enumerate the assumptions this specification leaves unstated." In every case the output is a list whose items can be adjudicated one at a time and independently. This is exactly the translation of in action. The decision itself is made by a human, because the moment the decision is delegated, $v$ jumps. **Implementation.** Cut the work into stages and prepare verification **first**. Writing the tests first drops $c_v$ to "the time it takes to run the tests". The reason $v$ was small in is precisely that existing tests and a type checker were in place. Conversely, adding a feature with an LLM to a codebase without tests makes $c_v$ equal to "the time a human takes to read every generated line", which often exceeds the time to write it oneself. Test-driven development is being reappraised in the age of LLMs not as a matter of philosophy but as a matter of . **Testing.** Having an LLM write tests is effective, with one pitfall. **Never have the same model write both the implementation and the tests in the same context.** A specification the model misread while implementing will be misread the same way while testing. A wrong implementation then passes a wrong test, and the false positive rate $\alpha$ jumps. A verifier is worthless unless it is independent of what it verifies. In practice, useful separations include generating tests from the specification (without showing the implementation code), filling in expected values by hand, and enumerating boundary cases yourself. **Review.** Code review by an LLM is in fact the highest-return use of all, and the reason is the smallness of $v$. A remark such as "this line may receive `null`" can be adjudicated in seconds by looking at that line. When a false remark slips through, what is lost is tens of seconds, not a production incident. The asymmetry **a remark may be wrong, but the code must not be** is what makes review such a favourable use case. Design-level questions such as "will this abstraction survive the next two years of change" have the same structure as , and verification there is expensive. ## 7. Seeing through an AI's "lies" So far the verifier has been taken as given. We now turn to the quality of the verifier itself. Deciding semantic properties of programs mechanically and completely is impossible by , so the false positive rate $\alpha$ of any real verifier cannot be made $0$. This is the heart of debugging in the age of LLMs. An output of an LLM that satisfies **formal coherence** (the syntax parses, the types check, the naming follows convention, the explanation reads as natural English or Japanese) while being **semantically false** is called, in this article, a **plausible error**. Calls to nonexistent APIs, implementations that drop a boundary condition, and citations of sources that say something else all qualify. What makes plausible errors dangerous is that they pass straight through the verifier called "a human's first impression". Broken syntax draws attention; a plausible error does not. If anything, the tidy naming and careful comments attached to it make it more trusted than code one wrote oneself. The effect has been observed experimentally: subjects with AI assistance wrote less secure code than unassisted subjects, and yet **believed more strongly that they had written secure code** (Perry et al., 2023). That is an increase in $\alpha$, plain and simple. How much does $\alpha$ actually matter? The question can be answered quantitatively. The structure of the computation is the same as finding the predictive value of a positive test result (). Let the false positive rate of a be $\alpha \in [0,1]$ and its false negative rate be $0$. Assume the generations are independent and each output is correct with probability $p \in (0,1)$. If generation is repeated until a pass occurs and **the first passing output is adopted**, then, conditionally on at least one pass occurring, the probability that the adopted output is wrong equals $$ P_{\text{err}} = \frac{(1-p)\,\alpha}{p + (1-p)\,\alpha} . $$ In particular, for small $\alpha$, $$ P_{\text{err}} \approx \frac{1-p}{p}\,\alpha , $$ so **the error rate is essentially proportional to the false positive rate**. The value does not depend on any bound on the number of attempts. For a single generation, the following three events are mutually exclusive and exhaust the sample space. - Correct and passing: probability $p$ (by the assumption of zero false negative rate, a correct output always passes). - Wrong and passing: probability $(1-p)\alpha$ (by the definition of $\alpha$ in ). - Wrong and failing: probability $(1-p)(1-\alpha)$. Hence a single generation passes with probability $q = p + (1-p)\alpha$, and $q > 0$ since $p > 0$. The event "the first pass occurs on the $i$-th attempt" has probability $(1-q)^{i-1}q$ by independence. Given that, the probability that this passing output is wrong is the ratio of the probability of "wrong and passing" to that of "passing" at the $i$-th generation, namely $(1-p)\alpha / q$, which does not depend on $i$. By the law of total probability, $$ P_{\text{err}} = \frac{\sum_{i \ge 1} (1-q)^{i-1}(1-p)\alpha}{\sum_{i \ge 1} (1-q)^{i-1} q} = \frac{(1-p)\alpha \cdot \frac{1}{q}}{q \cdot \frac{1}{q}} = \frac{(1-p)\alpha}{q} = \frac{(1-p)\alpha}{p + (1-p)\alpha}. $$ Both series converge because $0 < q \le 1$. The approximation follows because the denominator is $p + O(\alpha) \to p$ as $\alpha \to 0$. We substitute numbers into the formula of , evaluating $P_{\text{err}} = (1-p)\alpha / (p + (1-p)\alpha)$ directly. | $p$ | $\alpha$ | Computation | $P_{\text{err}}$ | |---|---|---|---| | $0.4$ | $0.40$ | $0.24 / (0.40 + 0.24)$ | $37.5\%$ | | $0.7$ | $0.40$ | $0.12 / (0.70 + 0.12)$ | $14.6\%$ | | $0.4$ | $0.05$ | $0.03 / (0.40 + 0.03)$ | $7.0\%$ | | $0.7$ | $0.05$ | $0.015 / (0.70 + 0.015)$ | $2.1\%$ | Taking the first row as the baseline: making the model cleverer so that $p$ rises from $0.4$ to $0.7$ improves the figure from $37.5\%$ to $14.6\%$, while tightening verification so that $\alpha$ falls from $0.40$ to $0.05$ improves it from $37.5\%$ to $7.0\%$. The latter is the larger gain. Furthermore, raising $p$ requires swapping the model, whereas lowering $\alpha$ is within one's own control. The way to read this table is: **adding one test case beats switching models more often than not.** assumes that the generations are independent. In a workflow where the output of a failing test is handed back to the LLM for repair, that assumption breaks — and it breaks in the wrong direction. The model tunes its output towards **passing that test**, so correctness in the parts the test does not inspect is no longer guaranteed. In the extreme, it hard-codes the expected value to make the test pass. The effective $\alpha$ goes up. The phenomenon has been measured. EvalPlus, which greatly expanded the test cases for HumanEval, the standard benchmark for code generation, saw the pass rates of many models drop by more than ten percentage points (Liu et al., 2023). That is direct evidence that the original tests constituted a verifier with $\alpha > 0$. A benchmark pass rate must not be read as the model's $p$. ### 7.1. Concrete procedures for catching them Five practices actually lower $\alpha$. 1. **Reduce claims to something executable.** Do not be satisfied by reading "this function also works on the empty list". Write the one line of code that passes an empty list. A natural-language explanation from an LLM is itself an unverified output. 2. **Check existence against primary sources.** Library function names, argument lists and return types are confirmed in the official documentation or the type definition files. The most frequent kind of plausible error is a nonexistent API built to follow a naming convention that does exist. If editor completion does not offer it, it very likely does not exist. 3. **Enumerate the degenerate cases yourself.** Empty, one element, duplicates, maximum value, negative, division by zero, `null`, extremely large inputs, concurrent access. Do not delegate this enumeration to the LLM: as says, the verifier must be independent of what it verifies. 4. **Read the diff line by line.** The lines one did not write are exactly the ones that need careful reading, though in practice the opposite tends to happen. To reduce the amount that must be read, cut the work into small stages, as prescribes. 5. **Suspect any mismatch between the explanation and the code.** A comment stating the correct specification above an implementation that does something else is a common pattern. A comment is evidence of what the model *intended* to do, not of what it *did*. The next example shows in a single piece of code why these procedures are necessary. Asked to "write a function returning the variance (the population variance) of a list of numbers", a model often returns something like this. ```python def variance(xs): n = len(xs) s1 = sum(xs) s2 = sum(x * x for x in xs) return s2 / n - (s1 / n) ** 2 ``` This is the formula $\operatorname{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$ transcribed directly, and mathematically it is correct. It also passes naive tests: `variance([1, 2, 3, 4])` returns $1.25$, since $\mathbb{E}[X^2] = 30/4 = 7.5$ and $(\mathbb{E}[X])^2 = 2.5^2 = 6.25$, which is the right answer. Pass it `xs = [1e8, 1e8 + 1, 1e8 + 2]`, however, and although the true population variance is $$ \frac{(-1)^2 + 0^2 + 1^2}{3} = \frac{2}{3} \approx 0.6667 , $$ evaluating this code in IEEE 754 double precision returns **$2.0$**, three times the true value. The detailed trace is deferred to the Appendix, but the cause is that near $10^{16}$ the spacing between floating-point numbers is already $2$, so at the step where **the difference of two nearly equal huge numbers** $s_2/n$ and $(s_1/n)^2$ is taken, the significant digits vanish wholesale (catastrophic cancellation). This code parses, type-checks, uses the right formula, explains itself naturally, and passes the representative tests. It is a plausible error in exactly the sense of . Catching it requires executing procedure 3, "enumerate the degenerate cases yourself", and adding to the tests **data with a large mean and a small variance**. And that enumeration cannot be thought of without **prior knowledge**: the numerical stability of statistical computations. The correct implementations are the two-pass method (compute the mean first, then the sum of squared deviations) or Welford's online method. The two-pass version reads ```python def variance(xs): n = len(xs) m = sum(xs) / n return sum((x - m) ** 2 for x in xs) / n ``` and returns $2/3$ correctly on the same input. What shows is that the ability to see through an LLM's errors **is the domain knowledge itself**. To someone who has never met catastrophic cancellation, those four lines read as correct however long one stares at them. This is where the prediction "AI writes the code, so knowledge becomes unnecessary" fails. The cheaper generation becomes, the more relative value the capacity to verify acquires. This consequence is developed from the side of mathematical learning in [Why relearn mathematics](/en/computer-science/ai-era/relearning-mathematics). ## 8. Exercises For a certain feature, you estimate that writing it yourself takes 120 minutes ($c_h = 120$), writing the instructions for the LLM takes 10 minutes ($c_g = 10$), and verifying the output takes 20 minutes ($c_v = 20$). 1. Find the condition on the success rate $p$ under which the strategy "try once, and write it yourself if it fails" pays. 2. Compute the expected cost of that strategy when $p = 0.5$. 3. Compute the expected cost, when $p = 0.5$, of the strategy that keeps regenerating until a pass occurs (with no bound on the number of attempts). **1.** By , $$ g + v = \frac{c_g + c_v}{c_h} = \frac{10 + 20}{120} = \frac{30}{120} = 0.25 . $$ By , the condition is $p > 0.25$: delegation pays once the success rate beats one in four. **2.** Substituting into the formula of , $$ C_1 = c_g + c_v + (1-p)c_h = 10 + 20 + 0.5 \times 120 = 30 + 60 = 90 \ \text{minutes}, $$ which is 30 minutes cheaper than the 120 minutes of doing it by hand. **3.** From the limit $k \to \infty$ in , $$ \frac{c_g + c_v}{p} = \frac{30}{0.5} = 60 \ \text{minutes}. $$ One may also argue that the expected number of attempts is $1/p = 2$, so the cost is $30 \times 2 = 60$ minutes. Since $p$ exceeds the threshold $0.25$, the expected cost decreases as attempts are added, exactly as predicts. Generation has success rate $p = 0.6$, and the verifier — a test suite with holes — has false positive rate $\alpha = 0.25$. Generation is repeated until a pass, and the first passing output is adopted. 1. Find the probability that the adopted artefact is wrong. 2. To hold that probability at $5\%$ or below, how small must $\alpha$ be? **1.** Substituting $p = 0.6$ and $\alpha = 0.25$ into the formula of : the numerator is $(1-p)\alpha = 0.4 \times 0.25 = 0.1$ and the denominator is $p + (1-p)\alpha = 0.6 + 0.1 = 0.7$, so $$ P_{\text{err}} = \frac{0.1}{0.7} = \frac{1}{7} \approx 0.143 . $$ About $14.3\%$: one time in seven, something wrong reaches production. **2.** Solve $P_{\text{err}} \le 0.05$ for $\alpha$. The denominator is positive, so it may be cleared without reversing the inequality. $$ \begin{aligned} \frac{0.4\alpha}{0.6 + 0.4\alpha} \le 0.05 &\iff 0.4\alpha \le 0.05(0.6 + 0.4\alpha) \\ &\iff 0.4\alpha \le 0.03 + 0.02\alpha \\ &\iff 0.38\alpha \le 0.03 \\ &\iff \alpha \le \frac{0.03}{0.38} \approx 0.0789 . \end{aligned} $$ The false positive rate must be brought to about $7.9\%$ or below — that is, the tests must be strengthened until they reject twelve out of every thirteen wrong implementations. Using the approximation $P_{\text{err}} \approx \frac{1-p}{p}\alpha = \frac{2}{3}\alpha$ from gives $\alpha \le 0.075$, close to the exact answer. In the sequential scheme of , suppose the verifier of each stage is imperfect, with false positive rate $\beta \in [0,1)$ and false negative rate $0$. Let the per-stage accuracy be $r$, the number of stages $n$, and assume generation and verification at each stage are independent. At each stage, generation is repeated until a pass and the first passing output is adopted before moving on. 1. Express the probability that the final artefact (all $n$ stages) is correct in terms of $r$, $\beta$ and $n$. 2. Compute the value for $r = 0.98$, $\beta = 0.2$, $n = 50$. 3. What practical conclusion about stage decomposition follows? **1.** Apply to a single stage. Replacing $p$ by $r$ and $\alpha$ by $\beta$ there, the probability that the output adopted at that stage is wrong equals $$ \frac{(1-r)\beta}{r + (1-r)\beta} , $$ so the probability that it is correct equals $$ \rho = 1 - \frac{(1-r)\beta}{r + (1-r)\beta} = \frac{r}{r + (1-r)\beta} . $$ The stages are independent, so the probability that all $n$ stages are correct is $$ \rho^{\,n} = \left(\frac{r}{r + (1-r)\beta}\right)^{n} . $$ **2.** Substituting the numbers: $(1-r)\beta = 0.02 \times 0.2 = 0.004$ and the denominator is $0.98 + 0.004 = 0.984$, so $$ \rho = \frac{0.98}{0.984} = 0.995935 . $$ From $\ln \rho = -0.0040733$, $$ \rho^{50} = e^{50 \times (-0.0040733)} = e^{-0.20366} \approx 0.816 . $$ About $81.6\%$. Turned around: with probability about $18.4\%$, the work is completed with an error left in some stage. **3.** The decomposition of improves the **expected cost** exponentially, but **correctness** degrades exponentially in the number of stages (since $\rho < 1$, the quantity $\rho^n$ decreases in $n$). Decomposition does not close the holes in verification; it increases to $n$ the number of opportunities to slip through them. Hence, if the work is cut finely, the per-stage $\beta$ must be lowered at the same time. To reach a final accuracy of at least $95\%$ with $n = 50$ requires $\rho \ge 0.95^{1/50} = e^{-0.001026} = 0.998975$, and $0.98 / (0.98 + 0.02\beta) \ge 0.998975$ demands roughly $\beta \le 0.05$. The advice "cut it small and hand it over" holds only together with the condition "put a verifier at every cut". ## References - Frederick P. Brooks, Jr., "No Silver Bullet: Essence and Accidents of Software Engineering", *IEEE Computer* 20(4) (1987), 10–19. The distinction between essential and accidental complexity. The high-$v$ tasks of this article correspond broadly to the region dominated by essential complexity. - Brian W. Kernighan and P. J. Plauger, *The Elements of Programming Style*, 2nd ed., McGraw-Hill, 1978. The source of the famous observation that debugging is twice as hard as writing the code. A classical observation about the size of the verification ratio $v$. - Mark Chen et al., "Evaluating Large Language Models Trained on Code", 2021. [arXiv:2107.03374](https://arxiv.org/abs/2107.03374) — the original paper on HumanEval and pass@k. - Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming Zhang, "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Synthesis", *NeurIPS* 2023. [arXiv:2305.01210](https://arxiv.org/abs/2305.01210) — EvalPlus, showing that pass rates fall when the tests are strengthened. - Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh, "Do Users Write More Insecure Code with AI Assistants?", *ACM CCS* 2023. [arXiv:2211.03622](https://arxiv.org/abs/2211.03622) - Sida Peng, Eirini Kalliamvakou, Peter Cihon, Mert Demirer, "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot", 2023. [arXiv:2302.06590](https://arxiv.org/abs/2302.06590) — the object of study is the single, clearly specified task of implementing an HTTP server in JavaScript. - METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", 2025 — the subjects worked on large repositories they maintain themselves, a situation with small $c_h$ (they are fast because they know the code) and large $c_v$ (their quality bar is high). - Nicholas J. Higham, *Accuracy and Stability of Numerical Algorithms*, 2nd ed., SIAM, 2002 — the numerical stability of variance computations is treated in Chapter 1. The background to . ## Appendix: A numerical trace of catastrophic cancellation **We trace, in IEEE 754 double precision, why the return value in is $2.0$.** Double precision has a 53-bit significand, so beyond $2^{53} = 9007199254740992 \approx 9.007 \times 10^{15}$ the integers are no longer representable one by one. The spacing (ulp) on the interval $[2^{53}, 2^{54})$ is $2$, and on $[2^{54}, 2^{55})$ it is $4$. The input is $x_1 = 10^8$, $x_2 = 10^8+1$, $x_3 = 10^8+2$, with $n = 3$. These are small enough to be represented exactly. - $s_1 = 300000003$ and $s_1/n = 100000001$, both exact. - The true value of $(s_1/n)^2$ is $10000000200000001$. This lies in $[2^{53}, 2^{54})$ where the spacing is $2$, and the value is odd, so it sits exactly midway between $10000000200000000$ and $10000000200000002$. Round-to-nearest-even selects $10000000200000000$, whose significand ends in a $0$ bit. - The terms of $s_2$ are $10000000000000000$, $10000000200000001 \to 10000000200000000$ (for the same reason), and $10000000400000004$ (exact, being even). Adding from the left gives $20000000200000000$ and then $30000000600000004$, both multiples of the spacing, so no rounding error enters. - $s_2 / n = 30000000600000004 / 3 = 10000000200000001.333\ldots$. Of the two candidates at spacing $2$, namely $10000000200000000$ (distance $1.333$) and $10000000200000002$ (distance $0.667$), the nearer is chosen, giving $10000000200000002$. The final subtraction is $10000000200000002 - 10000000200000000 = 2.0$, and that subtraction itself is error-free. The return value is therefore exactly $2.0$, three times the true value $2/3$. **The error entered not in the subtraction but in the two roundings that preceded it.** The subtraction merely exposed that error, and this is the essence of catastrophic cancellation. **The error worsens as the inputs grow.** Writing $\mu$ for the mean and $\sigma$ for the standard deviation, the computation takes the difference $\sigma^2$ of $\mathbb{E}[X^2] \approx \mu^2$ and $(\mathbb{E}[X])^2 \approx \mu^2$, so the number of digits lost is roughly $\log_{10}(\mu^2/\sigma^2) = 2\log_{10}(\mu/\sigma)$. If $\mu/\sigma$ is about $10^8$, that is 16 digits, consuming the entire significance of double precision. is precisely this situation. ## Appendix: The relation between pass@k and the model of this article **The pass@k familiar from papers on code generation models can be reread in the framework of this article.** pass@k is "the probability that at least one of $k$ samples passes the tests" (), which equals $1 - (1-p)^k$ when each sample passes independently with probability $p$. This is the same quantity as "the probability of passing within $k$ attempts" in . Two cautions apply. First, pass@k **does not count cost**. Raising $k$ raises the probability of passing, but it also raises the number of verifications (). What matters in practice is not the probability of passing but the expected cost, and that is governed by the threshold $g+v$ (). Second, the test suite behind pass@k is a verifier with $\alpha > 0$ in the sense of (Liu et al., 2023). A benchmark figure is an upper bound on $p$, not $p$ itself; the effective $p$ on your own project can only be measured with your own verifier.