# A background thread that outlives its task

`asyncio.to_thread` cannot be cancelled. Cancelling the task that awaits it ends the coroutine and leaves the worker running, detached and unreachable. At shutdown that orphan asked the connection pool for a connection while everything else was tearing down, waited out the pool timeout and raised — so the fix is to ask the thread to stop and then wait for it, with a bound.

- **Status:** Available
- **Audience:** both, developer
- **Last verified:** 2026-09-10
- **Canonical:** https://connectbyjbrh.com/research/background-threads/

## Cancelling the awaiter is not cancelling the work

The mental model most code is written with is that cancelling a task cancels what the task is doing. For a coroutine that is true. For `asyncio.to_thread` it is not: the awaited work is running on a real thread, cancellation raises inside the *coroutine*, and the thread carries on to the end of whatever function it was given. Nothing observes it any more, and nothing can interrupt it.

That is survivable while the application is running — the orphan finishes and disappears. It stops being survivable at shutdown, because the orphan and the teardown are now competing for the same finite resources on the way out.

## What the orphan did on the way out

1. The lifespan handler cancelled the monitor task. The coroutine ended; the tick thread did not.
2. The tick reached its next database call and asked the pool for a connection, which the rest of the teardown was busy returning and closing.
3. It waited the full pool timeout, then raised `QueuePool limit of size 5 overflow 10 reached`.
4. That surfaced as **Application shutdown failed. Exiting.** — and made every restart slower than the server's twenty-second graceful window.

Read from the outside, that is a deployment problem: restarts are slow and the last line of the log is a pool error. Read from the inside it is one detached thread doing exactly what it was told, several seconds after anybody wanted it to.

## The cooperative stop

The thread is asked to stop rather than cancelled, and the asking is two flags rather than one. One says *do not start, and stop between units of work*; the other says *a tick is in flight right now*, so shutdown can wait for it to finish instead of racing it.

| Flag | Read where | Effect |
|---|---|---|
| `_stop_requested` | Before the first query of a tick, and again between workspaces | An orphaned tick that has not touched the pool yet does not touch it now — the cheapest possible exit |
| `_tick_idle` | Cleared as a tick begins, set again in a `finally` | `stop()` can wait for a tick that is genuinely in progress rather than assuming |

1. Set the stop flag first, before cancelling anything.
   - Result: A tick that has not begun its work exits immediately and costs nothing, which is the common case.
2. Cancel the task and await it, so the coroutine side is genuinely finished rather than merely told.
   - Result: The coroutine stops. The thread may still be inside a unit of work; that is what the second flag is for.
3. Wait on the in-flight flag with an explicit bound — eight seconds here.
   - Result: Well inside the server's twenty-second graceful window, and short enough that a wedged tick delays a restart rather than failing it.

> **Note** The bound is the design decision, not the waiting. An unbounded wait turns one stuck tick into a deployment that cannot complete; no wait at all is the orphan again. A bound that is comfortably inside the supervising timeout is the only version that degrades well.

## The same rule one level up: drain, then kill

The voice worker faces the identical question with a much longer unit of work, because its in-flight item is a live telephone call. Its `drain_timeout` is 180 seconds — long enough to finish conversations in progress during a deploy — and `ops/connect-voice.service` allows 210 seconds before systemd kills the process. The outer number must exceed the inner one, or the supervisor kills the worker in the middle of the drain it was asked to perform and the whole arrangement is theatre.

- A **stop** says *start nothing new*. A **drain** additionally says *finish what is running*. They need different bounds because their units of work have different lengths: a mail tick is seconds, a call is minutes.
- Anything that reaps abandoned state must be reachable from more than one clock. `sweep_stale` runs from the engine tick as well as from the worker heartbeat, because a browser call has no worker to heartbeat at all — one sat marked active for 24.78 hours before that was true.
- A heartbeat file forgets a process not seen for an hour, so a machine that vanished does not hold capacity for ever.

## What this does not fix

- A tick that blocks inside a single query is not stopped by either flag; it is only bounded by the wait, after which the process exits with the thread still running. That is a deliberate trade in favour of restarting.
- The flags are per process. A second process has its own, and coordination between them is a different problem with a different answer.
- The interval between ticks is configurable and bounded (30 to 900 seconds, 60 by default). None of the above changes if it is tuned.
- There is no measurement here for how often the orphan actually raised in production before the change: UNKNOWN. The symptom was the slow restart.

## Questions

### Why not run the work in a task rather than a thread?

Because the work is synchronous database access, and putting blocking calls on the event loop stalls every request the application is serving. The thread is the right tool; what was wrong was treating it as if cancellation reached it.

### Why eight seconds and not thirty?

Because it has to be comfortably inside the graceful window the server itself allows, and the point of the wait is to catch the ordinary case — a tick a second or two from finishing — not to outlast a hung query. Past the bound, restarting is better than waiting.

### Does this apply to a worker holding a live call?

The same shape, different numbers. There the in-flight unit is a conversation somebody is having, so the drain is 180 seconds and the supervising timeout is 210 — see [Phone and voice in Connect](/docs/phone/) for what a deploy does to calls already in progress.

## Related

- [Why only one loop may think](https://connectbyjbrh.com/research/single-thinker/)
- [A copy of a password somebody else rotates](https://connectbyjbrh.com/research/stale-credentials/)
- [Phone and voice in Connect](https://connectbyjbrh.com/docs/phone/)
- [A probe that fires on a correct release is worse than no probe](https://connectbyjbrh.com/research/probes-that-cry-wolf/)
- [Idempotency for retried telephony webhooks](https://connectbyjbrh.com/research/idempotent-telephony-webhooks/)

## What this page is based on

- `backend/app/tenant_inbox.py` — the shutdown note, `_stop_requested`, `_tick_idle`, the bounded wait in `stop()` and the tick interval bounds
- `docs-source/sources/PHONE.md` §5 — `drain_timeout` 180 s, `ops/connect-voice.service` 210 s, and the heartbeat file
- `docs-source/sources/PHONE.md` §4 — `sweep_stale` running from the engine tick as well as the worker heartbeat