asyncio.wait_for does not bound an asyncpg query — and neither does command_timeout

Adding a timeout to a readiness probe that runs SELECT 1 over an asyncpg pool. Measured against a Postgres container that was docker paused (so it completes the TCP handshake and then never speaks — the exact case a timeout exists for):

await asyncio.wait_for(pool.fetchval("SELECT 1"), 2.5)

hung for the FULL 20s of curl --max-time 20. The pool also had command_timeout=15 set, and that did not fire either.

Why: both bounds are enforced by CANCELLING the query, and asyncpg cancellation opens a SECOND connection to the server to send a Postgres CancelRequest. That second connect blocks on the same unresponsive server. wait_for awaits the cancellation it just requested, so the timeout inherits the hang it was added to prevent. command_timeout has the identical hole.

Fix — separate the wait from the cleanup, so nothing awaits the cancellation:

probe = asyncio.ensure_future(pool.fetchval("SELECT 1")) done, _ = await asyncio.wait({probe}, timeout=2.5) if not done: _probes.add(probe) # strong ref: asyncio keeps only a weak one probe.add_done_callback(_probes.discard) probe.add_done_callback(lambda t: t.cancelled() or t.exception()) probe.cancel() # fire and forget, do NOT await raise HTTPException(503, "db did not answer within 2.5s") await probe

After: 503 in 2.50s, three requests running. On unpause, 200 in 5ms, no leaked tasks.

Two details that matter: - Keep a strong reference to the abandoned task. asyncio holds only a weak ref, so it can be garbage-collected mid-cancellation. - Add a done-callback that RETRIEVES the exception. A detached task whose exception nobody reads prints a full traceback when collected — and a healthcheck polling every 10s during an outage turns that into thousands of tracebacks that bury the real log line.

A sleeping-stub unit test cannot reproduce this: the stub cancels instantly, so wait_for looks correct and the test passes. It only shows up against a real server that is reachable-but-silent. docker pause is the cheapest way to produce that.

Related, same session: a FastAPI /health that returns 200 with no DB check will report healthy through a dead database if the pool is created with min_size=0 — because asyncpg's create_pool opens NO connection at that size, so it succeeds against a DSN naming a host that does not even resolve. Any "did the pool come up?" flag is therefore True in exactly the cases you wanted it to catch. Probe a readiness endpoint that does a real round trip, and note that Docker Compose does not act on a healthcheck at all (restart policies fire on process exit), so pointing a Compose healthcheck at readiness cannot cause a restart loop — Swarm and Kubernetes do act on it, and there the split matters.