# Five ways a test suite has passed while proving nothing Each of these ran green for months. The common shape is that the assertion and the risk were about different things — a rule restated instead of enumerated, a database policy the ORM path never touched, latency read from the wrong timestamps, a metric taken from a field the SDK does not report, and a fixture smaller than the limit the code was capping at. - **Status:** Available - **Audience:** both, developer - **Last verified:** 2026-09-10 - **Canonical:** https://connectbyjbrh.com/research/false-passing-tests/ ## One: a rule asserted rather than enumerated "Every capability is available to both audiences" is a sentence. Written as a test that checks a handful of known routes, it passes forever and says nothing about the route added last Tuesday. What made it real: `tools/test_audience_parity_v1.py` reads the flattened route table out of the running application and compares every operator capability against every customer one. `ALIASES` records where the customer surface renames something; `KNOWN_GAPS` lists what a tenant still cannot reach, each with a reason. The list may shrink and must never grow, and the suite fails both on a new one-audience capability and on a gap that has been closed without being removed. > **Note** The general move is from *sample* to *enumeration*. A suite that walks the real inventory fails on the thing nobody thought to add to the list, which is the only failure worth catching. ## Two: exercising the ORM, never the database's own rules Three `prospect_*` tables shipped with row-level security switched off. Every suite over them passed, because `create_all` makes a table at boot while the policy comes from `tools/pg_harden.py`, an operational script — and a test that reaches rows through the scoped ORM session never asks the database to enforce anything. What made it real: a check that enumerates scoped tables and fails on any one whose policy is absent or not forced. `test_global_prospect_intelligence_v1` now fails on a new table that repeats the omission, which converts a rule people were expected to remember into one the build applies. ## Three: a clock on the wrong source Voice latency was read from transcript rows. The first real call showed "8 ms" beside replies the caller had waited seconds for, because a transcript row arrives when the sentence is complete, not when the audio began. The number was precise, reproducible and about something else. What made it real: `TurnClock` times the wire — the caller's last word (user state leaving `speaking`) to the first audio back (agent state entering `speaking`) — and posts a `turn_timing` event per reply, plus a `barge_in` event carrying how long the voice kept talking after being spoken over. Past two seconds, an interruption is counted as ignored. The lesson generalises past voice: a measurement is only as good as the event it is anchored to, and a plausible number from a convenient source is more dangerous than no number, because it ends the investigation. ## Four: a field that does not exist, reading as zero Call metering looked for `input_audio_duration` in the model usage the SDK reports. There is no such field. Every metered call therefore recorded 0.0 seconds of audio, and every assertion of the form "duration is not negative" or "usage was recorded" passed. What made it real: reading the breakdown the SDK actually reports — `input_audio_tokens`, `input_text_tokens`, `output_reasoning_tokens`, `session_duration` — and charging a call that reports no tokens from its own duration at the published per-minute rate rather than at zero. `merge_usage` adds every session a call held, because a restart used to bill only the survivor. > **Careful** A metric that is always zero is indistinguishable from a metric that is correctly zero. Any measurement worth trusting needs at least one assertion that fails when the source is absent — a floor, a cross-check, or a required non-zero on a known-busy fixture. ## Five: a fixture smaller than the limit Two summary surfaces derived totals from bounded slices — the newest 500 rows, the newest 400 threads. Every suite over them passed, because no fixture ever exceeded the slice. Driven with 540 contacts whose 40 already-contacted rows were the oldest, the old code reported 500 organisations for 530 and 0 already contacted for 40; a workspace with 430 conversations showed a Closed tab of 0 and an empty page five. What made it real: fixtures larger than every limit in the path, with the interesting rows placed outside the newest window, and counts moved into the database. The full account is in [headline numbers that stop at the page size](/research/counting-past-the-limit/). ## What makes an assertion real - **Show it fail once.** Disable the fix, watch the suite go red, restore. An assertion never seen red is a hypothesis. - **Assert against the code, not against a comment beside it.** A check that reads documentation passes while the behaviour drifts underneath it. - **Enumerate the inventory** — routes, scoped tables, capabilities — rather than listing the ones you remember. - **Anchor a measurement to an event on the wire**, and say which event. - **Make the fixture bigger than every limit** the code path contains. - **Point a writing suite only at a database whose name ends `_test`**, and keep the enforcement in the runner rather than in a convention. ## Questions ### How do you notice a suite that is passing for the wrong reason? Break the thing it claims to protect and see whether it complains. That single exercise separates a real assertion from a restatement, and it is cheap enough to do once per new suite rather than per assertion. ### Is a green run evidence of anything? It is evidence that the assertions written are satisfied. Whether those assertions cover the risk is a separate question, and the five cases here are all failures of coverage rather than of execution. ### Do more tests help? Not by themselves. Each of these five would have survived any number of additional tests of the same shape. What changed the outcome was moving the assertion — to the running route table, to the database's own policy, to the wire, to the reported usage fields, and to a fixture past the limit. ## Related - [A probe that fires on a correct release is worse than no probe](https://connectbyjbrh.com/research/probes-that-cry-wolf/) - [Headline numbers that stop at the page size](https://connectbyjbrh.com/research/counting-past-the-limit/) - [Three independent layers of tenant isolation](https://connectbyjbrh.com/research/three-layers-of-isolation/) - [Measuring conversational latency correctly](https://connectbyjbrh.com/research/measuring-voice-latency/) - [One implementation, two audiences](https://connectbyjbrh.com/research/two-audiences-one-implementation/) - [The query cost that grows with the business](https://connectbyjbrh.com/research/n-plus-one-that-grows/) ## What this page is based on - `docs-source/sources/GENERAL.md` §1 and §3 — the parity suite, `SCOPED_TABLES` and `pg_harden.py` - `docs-source/sources/CHANNELS.md` §4 and §5 — the `prospect_*` policy gap and the capped-count measurements - `docs-source/sources/PHONE.md` §4 and §6 — the usage fields and the on-the-wire latency clock - Connect capability registry (docs-source/facts.py) — `MEASURED`, 171 test suites