Build and verify: verification, not trust.

6.1 Verification, not trust

A language model produces plausible text. Plausible means: the code looks like code that could stand in this place. Whether it is correct is another question. In Chapter 1 you learned about hallucinations: invented methods, wrong assumptions about a library, a test that supposedly passes but was never run. An agent can find some of these errors itself, because it has tools and can call the compiler and tests. But it can also overlook them, explain them away or “fix” them by adapting the test to the wrong code.

This leads to the most important principle of this chapter:

You don't check whether you trust the agent. You check whether the result meets the criteria.

That sounds obvious, but it shifts the work. In classic development, review is often the smaller part: you trust the colleague who wrote the code and look at the tricky spots. With agents the ratio flips. Writing has become cheap; verifying remains expensive. If you don't automate and structure verification, you drown in diffs or eventually wave them through unread.

Three levels of verification

It helps to split verification into three levels that answer different questions:

Open full-size graphicOpen full-size graphic

No level replaces the others. Deterministic checks are reliable but only see what someone formulated as a rule or test beforehand. Judgement-based checks also find the unplanned, but are occasionally wrong. The human carries the responsibility but has limited attention. Good pipelines put the cheap, reliable checks first, so that expensive human attention is only needed for what machines cannot decide.

Good is not the same as right

One distinction will accompany you throughout the course: code can be good (clean, secure, well structured) and still do the wrong thing. Most automated checks answer the first question. The second requires a yardstick outside the code: acceptance criteria. How to write them is covered in detail in Chapter 4. Here the topic is how to feed them into verification.

6.2 Tests as oracles

In testing theory, the authority that decides whether a result is correct is called the oracle. For coding agents, automated tests are the most important oracle because they combine three properties: they are executable, repeatable and independent of the agent's self-report.

What makes a test a good oracle

• It checks behaviour, not implementation. A test that only checks that a particular private method was called breaks with every refactoring and proves little about behaviour.
• It can fail. A test that would also be green with wrong code is no oracle. The check: would it turn red if you commented out the core logic?
• It covers negative and edge cases. Agents prefer to write the “happy path”. The interesting bugs lie in empty input, maximum values, missing permissions and concurrent access.
• It can be traced back to a requirement. If a test carries an acceptance criterion ID, you know which requirement it proves and can see which requirement has no test yet.

Typical deceptions

Agents optimise for what you visibly check. If the only check is “tests are green”, there are several ways to get there that you don't want:

• A failing test gets deleted, commented out or given an ignore annotation.
• The expected values in the test are adapted to the wrong result.
• The agent claims in its final report that “all tests pass” without anything having changed in the tests or without them having run at all.
• The new code isn't reached by any test at all, yet the suite is green.

Two things help against this: first, you let the pipeline run the tests instead of having the agent report on them. Second, in reviews you look specifically at changes to test files. A diff that changes production code and expected test values at the same time deserves a second look.

Test first, for agents too

A proven way of working: first write the test that describes the desired behaviour and fails today, then implement until it turns green. For agents this is particularly effective because the red test sets an objective goal. For bug fixes, correspondingly: first a test that reproduces the bug. The factory's engineering guardrails require exactly this (guardrail “Make it verifiable”): translate the task into verifiable success criteria and ship the reproducing test so that the gate can enforce it.

// Bug-fix assignment: "Discount becomes negative at order value 0"
@Test
void rabattBeiWarenwertNullIstNull() {
    var kasse = new Kasse(Rabattstaffel.STANDARD);
    assertThat(kasse.rabattFuer(Betrag.euro(0))).isEqualTo(Betrag.euro(0));
}
// First red (reproduces the bug), then green (proves the fix).

6.3 Build gates: the first, cheapest hurdle

A gate is a checkpoint that a result has to pass before it moves on. The simplest and most important gate is the full build with all tests. It answers the question “Is this state internally consistent?” deterministically and without room for interpretation.

What belongs in a build depends on the stack. Typical are:

• compilation or type checking,
• unit and integration tests,
• architecture tests (for example layering rules with ArchUnit),
• linters and format checks,
• coverage thresholds,
• dependency and licence checks.

The more of this the build contains, the more errors the cheapest verification stage catches. This has a pleasant side effect: on a red build, the agent gets back a concrete, machine-readable error message that it can use to make targeted fixes. “The reviewer didn't like it” is harder to act on than “Test rabattBeiWarenwertNullIstNull expected 0, got -5”.

The build must run from the outside

An agent can and should build and test on its own while working. But the result of that self-test is a self-report. The gate therefore runs once more, outside the agent session, in an environment the agent has not modified. Only this run counts.

6.4 Code review by human and machine

A green build tells you that nothing known is broken. It says nothing about things for which there is no test: an SQL query built by string concatenation, an API key in the source code, a class in the wrong layer, an invented method that happens to sit in a branch that never executes. That is what reviews are for.

Machine reviewers

Machine reviewers come in two kinds:

• Static, rule-based reviewers search the diff for known patterns: hard-coded keys, path traversal, layering violations, modified database migrations. They are fast, cheap and reproducible, but only see what there is a rule for.
• LLM-based reviewers read the diff like a human and comment on logic errors, unclear names or missing edge cases. They find the unforeseen, but are themselves a language model with all its weaknesses. They are therefore used with a confidence score and are not allowed to decide on their own.

Independence: whoever writes does not approve

A central principle is independent verification: the agent that wrote a change must not approve it itself. It is the same logic as the four-eyes principle among humans. An agent assessing its own work tends to confirm its own assumptions. A separate reviewer with a different assignment (and ideally a different context or model) sees different things.

Reviewers don't write

This leads to a second rule: reviewers are read-only. A reviewer who “quickly” fixes a bug is no longer a reviewer but a second author whose changes nobody has checked. Moreover, a manipulated or faulty reviewer could otherwise alter the object under review while reviewing it. Findings therefore go back to the author as structured feedback, never as changes in the workspace.

The human in the review

Machine reviews relieve the human but do not replace them. Good human reviews of agent code focus on:

• Does the diff match the assignment, or did the agent do more (or something else)?
• Were tests changed, and if so, why?
• Are there changes outside the expected area (build files, configuration, migrations)?
• Did the agent disclose assumptions or deviations (in the decision log, in the code or under plans/)? The PR text itself comes from the factory and only refers to them.

The last point is no accident: the guardrails require the agent to make assumptions and deliberate deviations visible instead of asking questions that nobody answers during an autonomous run. A review should read these spots first.

6.5 Fail closed: when the check itself fails

Any verification infrastructure can fail: a scanner is missing from the path, an LLM reviewer gets no response, a configuration file is unreadable. The decisive question is what happens then. There are two options:

• Fail open: The check is skipped and the result counts as passed. That is convenient and dangerous, because an attacker or a configuration error only has to make the check crash.
• Fail closed: A check that could not take place does not count as passed. The result is held until someone clarifies the cause.

For anything concerning security or evidence, fail closed is the only defensible stance. The factory's parallel roadmap formulates this as architecture principle AP-7: a failed mandatory reviewer, a policy that cannot be checked or a failed attestation must never count as success.

Fail closed comes at a cost: occasionally a run stops because a tool is missing, even though the code would be fine. That is the price for a green result meaning something. And there is an honest alternative to silent circumvention: if you deliberately don't want a reviewer, you switch it off, visibly and traceably, instead of letting it fail.

6.6 Correction loops: new information instead of blind repetition

When a gate fails, it is tempting to simply run the agent again. Language models are not deterministic; maybe it works the second time. That is exactly a bad idea, for three reasons:

1. It costs tokens and time without specifically increasing the probability of success. The agent has the same information as before and will very likely make a similar mistake.
2. It obscures causes. A run that turned green “on the third try” may be green by chance, for example because a flaky test happened to pass this time.
3. It invites optimising for the gate. The more often you repeat, the greater the chance that some run passes the gate in an undesired way.

A sensible correction loop therefore only repeats when new information is available: compiler output, a failed test with its message, a reviewer finding, a merge conflict with the affected files, a red CI result. This information goes into the next run as a work instruction. The parallel roadmap calls this AP-6: Rule Loop instead of Blind Retry.

Set limits

Even an informed loop needs an upper limit. If two targeted correction attempts are not enough, the cause is usually not a trifle but a misunderstanding in the assignment, missing context or a contradiction in the requirements. Then a human needs to take a look, not a third run.

Good correction input

• Concrete: error message, file, line, affected test.
• Complete but filtered: the relevant excerpt of a log, not ten thousand lines.
• With a goal: “Fix the bug without changing the test expectation” is better than “make it green”.

6.7 Placing human approvals correctly

A human who has to confirm every intermediate step becomes a bottleneck and at some point clicks reflexively. A human who is never asked carries responsibility for something they haven't seen. The art lies in placing approvals where a decision actually changes something.

Two points have proven themselves:

• Before execution: May this assignment run at all, with this agent, this model and this budget? Here you prevent expensive or risky runs before they produce costs or changes.
• Before adoption: Is the validated result good enough to go into the main line? Here you decide with full knowledge: diff, build result, gate findings.

The second approval is the more important one when it comes to responsibility. An approval that only happens after the code is already on the main branch is a façade.

Segregation of duties

In regulated environments, segregation of duties applies in addition: whoever triggered a change must not approve it themselves. That is the human counterpart to “whoever writes does not approve”.

6.8 Pull requests and CI as a second verification authority

In many teams, the pull request is where code is reviewed and adopted: CI builds and tests, colleagues comment, branch rules enforce green checks before merging. Agentic development should fit into this flow instead of working around it. That has several advantages:

• The target repo's CI is a second, independent check. It runs in a different environment, often with additional tests, and knows nothing about the agent.
• Existing rules (protected branches, mandatory reviews) automatically apply to agent code too.
• The history stays in one place: anyone who later asks why a change was made finds it in the PR with description and discussion.

Feedback is what matters: if CI turns red after the agent has finished, that information must go back into a correction loop. Otherwise a red PR is left lying around and someone has to fix it manually.

6.9 From “good” to “right”: acceptance criteria in the gate

All the checks mentioned so far mostly answer the question of whether code is good. To check whether it does the right thing, the gate needs a domain yardstick. Two building blocks make this possible:

• Acceptance criteria with stable IDs, supplied with the assignment. A reviewer can then ask specifically: is AC-007-03 fulfilled by the diff?
• Tests that reference these IDs (in the name, as an annotation or in a comment). This makes coverage measurable: for which criteria is there at least one test, and for which not?

Conversely, you can also ask: which changes in the diff belong to no criterion? Such uncovered behaviour is not automatically wrong, but it hints at scope creep or a forgotten criterion.

6.10 Summary: the verification chain of a build run

Assignment (objective, acceptance criteria if any)
   │
   ├─ Approval before EXECUTION (default without policy: run.approval-vor-merge)
   ▼
EXECUTION on branch sdlc/run-<id8>
   ▼
VALIDATION: mvn verify ──red──► correction loop (build feedback, ≤ 2)
   ▼ green
Quality gate: 6 read-only reviewers
   │   advisory: display   blocking: FAILED/ERROR ──► correction loop
   ▼
Approval before merge (approval policy / pflichtFreigabePhasen=VALIDATION)
   ▼
Merge into base   or   push + pull request
                          ▼ (git.pr-feedback.enabled)
                       WAITING_FOR_PR: red CI ──► correction loop
                          ▼ green + merged
                       COMPLETED

• Verify results, not intentions. An agent's self-report is not verification.
• Put cheap, deterministic checks first and make them as strict as possible.
• Reviewers are independent and read-only; whoever writes does not approve.
• Fail closed: a check that failed to run is not a success.
• Repeat only with new information, and limit the attempts.
• Place human approvals where they decide something, above all before adoption.

Chapter questions

1. Why isn't it enough for an agent to write in its final report that all tests are green?

   Model answer

   The report is a self-report. The agent may be wrong, or may not have run, may have changed or deleted tests. Only a build and test run executed by the pipeline outside the agent session may count. The factory's hallucination-reviewer looks for exactly the pattern “tests pass” without test changes.

2. What distinguishes a good test as an oracle from a bad one? Name three characteristics.

   Model answer

   It checks behaviour rather than implementation, it can actually fail (would be red with wrong code), it covers negative and edge cases, and it can be traced back to a requirement, ideally via an AC ID.

3. Explain the difference between fail open and fail closed using the example of a reviewer whose CLI is missing.

   Model answer

   Fail open would skip the missing reviewer and count the gate as passed. Fail closed counts the failed check as not passed. The factory distinguishes: a reviewer that fails technically yields ERROR, never a silent pass, and in mode blocking this starts the correction loop. A missing CLI yields SKIPPED; that is explicitly not PASSED, and the strict policy requires the mandatory categories, so that the absence doesn't slip through silently.

4. Why are the factory's reviewers read-only, and how is this enforced technically for the LLM reviewers?

   Model answer

   A writing reviewer would be a second, unchecked author and could alter the object under review during the review. Findings therefore go to the author as feedback. Technically, the Claude Code review runs with --permission-mode dontAsk and --disallowedTools for Edit, Write and Bash.

5. A build run has failed twice in the correction loop and is in NEEDS_CORRECTION. What do you do, and what don't you do?

   Model answer

   Don't simply restart it hoping for luck. Instead, look at the run branch and the error messages, classify the cause (unclear objective, missing context, contradictory requirement, oversized assignment) and act specifically: sharpen the objective, add context or memory, decompose the assignment, and then start a new run. The base branch is untouched, so there is no rush.

6. Which two approval points does the factory have, and why is the second one decisive for regulated environments?

   Model answer

   Before EXECUTION (the agent only runs after approval; without an approval policy, the default via run.approval-vor-merge) and before the merge in VALIDATION (via an approval policy or pflichtFreigabePhasen). Without the second point, the code would already be on the base branch before anyone approved; the approval would then be a façade.

7. Your team wants agent PRs to go through the existing CI as well and to be fixed automatically when CI is red. What do you set up in the factory?

   Model answer

   The Git remote URL on the project, a GitHub token under /integrations, the host in the allowlist (SOFTWAREFABRIK_GIT_ALLOWED_HOSTS), and enable git.pr-feedback.enabled. A successful build run then opens a PR, waits in WAITING_FOR_PR for green CI and a merge, and red CI starts a correction. Today this only works with GitHub.

8. When is advisory the better gate mode than blocking, and when is it the other way round?

   Model answer

   advisory fits during the introduction phase or for reviewers whose noise you don't know yet: findings become visible but don't block. blocking fits once the findings are reliable, and is mandated for regulated profiles so that a FAILED or ERROR actually forces a correction.

9. Transfer: A gate checks whether code is good. What is missing to check whether it does the right thing, and how does the factory close this gap?

   Model answer

   A domain yardstick: acceptance criteria with IDs and tests that reference these IDs. Since v0.35 the factory passes the criteria of the approved spec to all reviewers, and since v0.36 to the agent in the objective header. Since v0.38.0, the spec-conformance reviewer checks, rule-based, whether every bound criterion is backed by at least one test and which changes are not assigned to any criterion.