Nessie v2 webhooks: GET /webhooks/{id}/deliveries stuck at status 'pending', attempt 1, http_status null; old trycloudflare hooks stay active and collect 530s
Testing Nessie v2 notify webhooks (POST /webhooks {"mode":"notify","url":"https://<random>.trycloudflare.com/webhook","events":["transaction.completed"]}) from a container running cloudflared quick tunnels. Registration works and returns the hook with a whsec_... secret shown once. POST /webhooks/{id}/ping returns 200 with a delivery record {"event_type":"ping","attempt":1,"status":"pending"}.
What we then saw in GET /webhooks/{id}/deliveries: 1. Deliveries can sit at status "pending", attempt 1, http_status null, latency_ms null for a long time. In one session the ping and 35 transaction.completed deliveries were still unattempted after about 35 minutes, while the tunnel URL answered an external curl with 200 and the receiver logged nothing. Hooks registered a few minutes earlier had been attempted. We could not see the cause from outside. A backlog or stalled delivery worker on the hosted service is likely (GET /health reported a fresh restart shortly afterwards). Do not build a demo that depends only on webhook timing. 2. Each run of a quick tunnel gets a new random trycloudflare.com hostname, and each test run registered a new hook without deleting the old one. Old hooks stay active: true, every event is fanned out to all of them, and their rows fill up with attempt 2/3, http_status 530, error "bad status". 530 is Cloudflare's response for a tunnel hostname whose connector is gone. Attempts that reached our FastAPI receiver but were rejected show http_status 400.
What to do: - On shutdown, DELETE /webhooks/{id}. On startup, GET /webhooks and delete hooks whose url points at a tunnel you no longer run. - Treat the event feed (GET /events?after=<seq>) as the source of truth and webhooks as a latency optimisation. The same events arrive there with the same evt_ ids, and the feed needs no public URL. De-duplicate on Nessie-Event-Id / event id when you consume both. - Use the delivery log to debug: each row has event_type, status, attempt, http_status, latency_ms, error and (for decision hooks) outcome. A null http_status means Nessie has not reached your URL yet. A number means it did, and that is what your endpoint or the tunnel answered. - Hosted delivery only calls public https URLs. localhost and private ranges are refused, so a tunnel (cloudflared, ngrok) is needed for local receivers. - Return 2xx quickly and verify Nessie-Signature (t=<unix>,v1=<hex HMAC-SHA256 of "t." + raw body>) on the raw bytes, not on re-serialized JSON.