A background thread that outlives its task

asyncio.to_thread cannot be cancelled. Cancelling the task that awaits it ends the coroutine and leaves the worker running, detached and unreachable. At shutdown that orphan asked the connection pool for a connection while everything else was tearing down, waited out the pool timeout and raised — so the fix is to ask the thread to stop and then wait for it, with a bound.

Status: Available What this means
Audience: both, developer
Last verified: 2026-09-10
Product version: 6.3.2

Cancelling the awaiter is not cancelling the work#

The mental model most code is written with is that cancelling a task cancels what the task is doing. For a coroutine that is true. For asyncio.to_thread it is not: the awaited work is running on a real thread, cancellation raises inside the *coroutine*, and the thread carries on to the end of whatever function it was given. Nothing observes it any more, and nothing can interrupt it.

That is survivable while the application is running — the orphan finishes and disappears. It stops being survivable at shutdown, because the orphan and the teardown are now competing for the same finite resources on the way out.

What the orphan did on the way out#

The lifespan handler cancelled the monitor task. The coroutine ended; the tick thread did not.
The tick reached its next database call and asked the pool for a connection, which the rest of the teardown was busy returning and closing.
It waited the full pool timeout, then raised QueuePool limit of size 5 overflow 10 reached.
That surfaced as Application shutdown failed. Exiting. — and made every restart slower than the server's twenty-second graceful window.

Read from the outside, that is a deployment problem: restarts are slow and the last line of the log is a pool error. Read from the inside it is one detached thread doing exactly what it was told, several seconds after anybody wanted it to.

The cooperative stop#

The thread is asked to stop rather than cancelled, and the asking is two flags rather than one. One says *do not start, and stop between units of work*; the other says *a tick is in flight right now*, so shutdown can wait for it to finish instead of racing it.

Flag	Read where	Effect
`_stop_requested`	Before the first query of a tick, and again between workspaces	An orphaned tick that has not touched the pool yet does not touch it now — the cheapest possible exit
`_tick_idle`	Cleared as a tick begins, set again in a `finally`	`stop()` can wait for a tick that is genuinely in progress rather than assuming

Set the stop flag first, before cancelling anything.
Result A tick that has not begun its work exits immediately and costs nothing, which is the common case.
Cancel the task and await it, so the coroutine side is genuinely finished rather than merely told.
Result The coroutine stops. The thread may still be inside a unit of work; that is what the second flag is for.
Wait on the in-flight flag with an explicit bound — eight seconds here.
Result Well inside the server's twenty-second graceful window, and short enough that a wedged tick delays a restart rather than failing it.

The same rule one level up: drain, then kill#

The voice worker faces the identical question with a much longer unit of work, because its in-flight item is a live telephone call. Its drain_timeout is 180 seconds — long enough to finish conversations in progress during a deploy — and ops/connect-voice.service allows 210 seconds before systemd kills the process. The outer number must exceed the inner one, or the supervisor kills the worker in the middle of the drain it was asked to perform and the whole arrangement is theatre.

A stop says *start nothing new*. A drain additionally says *finish what is running*. They need different bounds because their units of work have different lengths: a mail tick is seconds, a call is minutes.
Anything that reaps abandoned state must be reachable from more than one clock. sweep_stale runs from the engine tick as well as from the worker heartbeat, because a browser call has no worker to heartbeat at all — one sat marked active for 24.78 hours before that was true.
A heartbeat file forgets a process not seen for an hour, so a machine that vanished does not hold capacity for ever.

What this does not fix#

A tick that blocks inside a single query is not stopped by either flag; it is only bounded by the wait, after which the process exits with the thread still running. That is a deliberate trade in favour of restarting.
The flags are per process. A second process has its own, and coordination between them is a different problem with a different answer.
The interval between ticks is configurable and bounded (30 to 900 seconds, 60 by default). None of the above changes if it is tuned.
There is no measurement here for how often the orphan actually raised in production before the change: UNKNOWN. The symptom was the slow restart.

Questions#

Why not run the work in a task rather than a thread?

Because the work is synchronous database access, and putting blocking calls on the event loop stalls every request the application is serving. The thread is the right tool; what was wrong was treating it as if cancellation reached it.

Why eight seconds and not thirty?

Because it has to be comfortably inside the graceful window the server itself allows, and the point of the wait is to catch the ordinary case — a tick a second or two from finishing — not to outlast a hung query. Past the bound, restarting is better than waiting.

Does this apply to a worker holding a live call?

The same shape, different numbers. There the in-flight unit is a conversation somebody is having, so the drain is 180 seconds and the supervising timeout is 210 — see Phone and voice in Connect for what a deploy does to calls already in progress.