A probe that fires on a correct release is worse than no probe
On 8 September 2026 a release script asserted that a whole subtree was byte-for-byte what it replaced, and reported 235 backend files different for a release that changed no Python at all. The comparison had walked __pycache__: the build that had been serving for two hours had written bytecode the freshly unpacked one had not. The assertion was sound; the set of files was not.
What the probe was trying to do, and why it was right to try#
The strongest thing a release that should change nothing can assert is that a whole subtree is identical to the one it replaced. It catches a truncated upload, a stale artefact, a half-applied deploy and a file that was edited on the server — a class of problem that no health check notices, because a wrong build serves 200s perfectly well.
ops/deploy_template.sh is what every release script here is cut from, so the mistake propagated by copying rather than by being written twice. That is the second lesson: a defect in a template is a defect in every release made from it until somebody pins the template.
Why the false alarm is the expensive failure#
A probe that fires on a correct release trains its readers. The alarm gets reclassified as noise within two or three releases, and then the one release where the difference is real is waved through with the same shrug. There are four releases' worth of notes in this repository about probes that cried wolf, and the pattern is the same every time.
It is worse than having no probe, because no probe leaves the risk visible and known. A noisy probe replaces the risk with a false sense that something is watching, and it costs a few minutes of attention per release for the privilege.
What a release probe should compare#
| Rule | Why |
|---|---|
| Hash the artefact, not the running directory | A live process writes into its own tree — caches, compiled bytecode, logs — and none of it came from the release |
| Prune generated content explicitly | tree_sum prunes __pycache__, so the hash covers what shipped rather than what the interpreter produced afterwards |
| Make the file order deterministic | It sorts under -print0 / sort -z, so a filename containing a space cannot reorder the list and turn an identical tree into a different hash |
| Make the explanation match the comparison | tree_diff excludes the same directory, so when a tree really does differ the printed reason names real files instead of bytecode |
| Compare intent, not just bytes | The release's own identifier against the one now serving answers "is this the build we meant", which a hash alone does not |
| Use the health endpoint that exists | /api/health. There is no /healthz, /health or /readyz, and a probe pointed at one of those fails on a perfect release |
The mirror case: a real failure that read as a bad deploy#
The opposite mistake is just as costly. The database password is not in .env: the managed database rotates the master credential into a secrets store and boot reads it from the secret named by MAYA_MAIL_DB_SECRET_ARN. A stale copy left in .env once made the *next restart* authenticate as nobody — and because the symptom appeared at a restart, it read as a bad deploy.
Nothing about that release was wrong. A probe comparing the artefact would have said so immediately and pointed the investigation at the environment instead, which is the whole value of separating "the code that shipped is not the code that was built" from "the environment this code runs in has changed".
Pinning the helper, and what that check cannot prove#
The fixed comparison lives in the template rather than in whichever release script was last edited, and a suite asserts against the template's source: that it is still a template, that it still offers a whole-tree comparison, and that both the hash and the diff exclude the same generated directory.
- It is a source check, not an execution. The template contains a placeholder for the release identifier, and running it would deploy something — so what is proved is that the shape survived the next copy, not that the shipped script behaved.
- It cannot prove the artefact was built from the intended source; that is what comparing the release identifier is for.
- It says nothing about behaviour after the release. A byte-identical tree with a changed environment is exactly the stale-credential case above.
- UNKNOWN: no figure is published for how often the probe fired before the fix, beyond the one release that produced the 235-file report.
Questions#
Why not just compare the two directories with a standard diff?
Because one of them is a directory a live process has been writing into. Compiled bytecode, caches and transient files appear there without ever having been part of a release, and a comparison that includes them reports a difference on every correct deploy.
Is byte-for-byte comparison worth keeping at all?
Yes, on the artefact. It catches truncated uploads and half-applied releases that health checks cannot see. The fix was to compare the right set of files deterministically, not to abandon the assertion.
What should a probe do on a correct release?
Nothing at all. Silence on a good release is the property that makes noise on a bad one worth reading, and any probe that cannot manage silence should be removed rather than tolerated.