# Proving a change did not break anything

Pick the cheapest check that would fail if the change were wrong, run it, and stop there. Escalate a rung only when the change's reach demands it, and run the full set once, near the end. A new assertion is trusted only after it has been watched to fail, and any change to a screen or a route is checked as both audiences.

- **Status:** Available
- **Audience:** developer, owner
- **Last verified:** 2026-09-10
- **Canonical:** https://connectbyjbrh.com/docs/workflows/regression-check/

## The ladder

| Rung | Proves | Does not prove |
|---|---|---|
| The documentation check | That what is written still matches the code it describes | Anything about behaviour |
| The fast gate | That the change is well-formed: it imports, it lints, the module graph resolves | That it does what it was meant to |
| One suite, for the area touched | That the behaviour the change was about still holds | That nothing else moved |
| The suites selected by the files touched | That the neighbourhood of the change is intact, escalating by itself when a wide-reaching file is involved | That an unrelated area is unaffected |
| The full set | That the whole system still behaves — the corpus runs to 171 suites | That the screen looks right to a person |
| A browser pass as each audience | That the change is visible, reachable and correct on screen | Nothing beyond what was looked at |

The ladder is not a sequence to climb every time. It is a menu ordered by cost, and the discipline is to take the lowest rung that would actually fail if the change were wrong, then stop. Running the full set three times in one task is not thoroughness — it is the same evidence bought three times.

## Proving one change, stage by stage

1. Trigger — a change is written and appears to work.
2. User event — somebody decides what would have to be true for the change to be wrong. That question, not habit, chooses the rung.
3. Authentication and workspace resolution — the suites run against a database of their own, never a live one.
4. Ingest — the changed files are the input to selection: which suites cover this area, and does any file here reach further than it looks.
5. Canonical record — the test database. Suites write, so it is a real database and not a mock, which is what makes the results mean something.
6. Classification — narrow change or wide one. A change to a shared module escalates the selection by itself rather than relying on somebody to notice.
7. Rules — a database whose name does not end in the test suffix is refused by the runner. That guard exists because a suite that writes is only safe where writing is expected.
8. Approval — a new assertion is disabled once and watched to go red before it is believed. An assertion that has never failed has proved nothing.
9. Action — the suites run, in parallel by default; a serial run is available when interleaved output would hide which suite failed.
10. Result — pass or fail per suite. A pre-existing failure is established as pre-existing once, and then left alone unless fixing it is the task.
11. Timeline — the change is committed with the evidence, so the next person can see what was actually run rather than that it was claimed.
12. Audit — the full set runs once near completion, and again only if production code changed after it passed. A change to a test, a comment or a document does not re-trigger it.

## The two-audience rule

One codebase serves the operator and every customer workspace over the same implementation, and the paths differ: a customer's requests are rewritten to a customer surface, and anything the rewriter does not recognise becomes a refusal. A separate allowlist refuses any call a customer session is not permitted to make. So a feature that works for one audience and 403s for the other is almost always one of those two and almost never the feature.

- Every change to a screen or a route is looked at in a browser as the operator and as a customer, not reasoned about from the source.
- New behaviour goes in a module both routers call. A second implementation for the other audience is how the two drift apart.
- Parity is enumerated rather than asserted: a suite reads the route table out of the running application and compares every capability on one side against the other.
- The list of known gaps carries a reason for each, may shrink, and must never grow — and the suite fails on a new one-audience capability as well as on a gap that has quietly been closed.

> **Note** Two of the three operator-only defects found on one day in September 2026 were invisible in the source and obvious within thirty seconds in a browser: a stack of notifications covering a control, and a remembered panel size hiding another.

## What a green run does not mean

1. It does not mean the assertion was ever capable of failing. An assertion written against a comment beside the code, rather than the code, has passed while proving nothing.
2. It does not mean the screen is right. Layout, focus, contrast and whether a control is reachable at all are not visible to a suite.
3. It does not mean the other audience is fine, unless the suites or the browser pass covered it.
4. It does not mean a probe is trustworthy. A probe that fires on a correct release trains everybody to ignore it, which is worse than not having one.
5. It does not mean the run used the interpreter you think. A wildly different pass count between runs usually means a different environment, not a broken system.

## Questions

### Why not run everything every time?

Because the cost is not zero and the information is. A full run after a one-line documentation fix tells you what the previous full run already told you. The rule that keeps this honest is the other half: the full set does run once, near the end, whenever production code changed.

### How do you know a new test is actually testing something?

By turning the fix off once and watching the suite go red, then restoring it. That is per suite rather than per assertion — one demonstration covers everything in it. A suite that has only ever been green is an assumption with a green tick beside it.

### What about a failure that was already there?

Establish once that it is not yours — usually by seeing that nothing you touched appears in the change — say so, and leave it. Investigating it repeatedly in the same task is a cost with no result, and fixing it unasked mixes two changes into one.

## Related

- [Five ways a test suite has passed while proving nothing](https://connectbyjbrh.com/research/false-passing-tests/)
- [A probe that fires on a correct release is worse than no probe](https://connectbyjbrh.com/research/probes-that-cry-wolf/)
- [One implementation, two audiences](https://connectbyjbrh.com/research/two-audiences-one-implementation/)
- [Defects that are invisible in the source and obvious in a browser](https://connectbyjbrh.com/research/invisible-ui-defects/)
- [Verifying published documentation](https://connectbyjbrh.com/docs/workflows/docs-verification/)
- [Security and isolation](https://connectbyjbrh.com/docs/security/)

## What this page is based on

- Connect source pack §1 — one codebase, two audiences, and parity enumerated rather than asserted (`docs-source/sources/GENERAL.md`)
- Connect measured facts (`docs-source/facts.py`) — 171 test suites
- Connect source pack §11 — measurements are counted as statements rather than wall-clock (`docs-source/sources/GENERAL.md`)