Our database provider restarted the instance under one of our services. That part is ordinary. Managed infrastructure reboots, and you design for it.
What happened next was more interesting, and we think it is a failure mode a lot of teams have and haven’t hit yet.
The shape of it
Between our application and the database sits a connection pooler. When the database went away, the pooler stayed up. It had to: it is a separate process with its own lifecycle, and from where it sat the right behaviour was to keep accepting clients and keep trying to reconnect upstream.
So our application was holding sockets that were, as far as the operating system could tell, completely healthy. The peer was alive. It was answering. It just had nothing behind it.
Now count the bounds we had in place, because we had several and they are all reasonable.
A query timeout bounds how long a statement may run. It needs the server to enforce it, and the server was gone.
A TCP keepalive kills a connection whose peer has become unreachable. Our peer was extremely reachable. Its kernel answered every probe immediately, which is precisely what a keepalive asks and precisely the wrong question here.
A connect timeout bounds establishing a new connection. These were already established.
Every guard was pointed at a failure mode that wasn’t the one we had.
Where it actually bit
A thread waiting forever on a socket is bad. A thread waiting forever on a socket it checked out of a fixed-size pool is worse, because it is also holding a slot nobody can reclaim. Enough of those and healthy requests start queueing behind connections that will never answer, and the service reports itself unavailable while the database sits there perfectly idle, wondering why nobody is calling.
That gap is what makes this one nasty to diagnose. Every dashboard on the database side looks fine. The problem is entirely on the client, in a pool, inside a process.
What we changed
We already had the right idea: a registry that records which code is holding each pooled connection, and a reaper that can forcibly interrupt a hold that has outlived its cap. Interrupting the socket directly is the one bound that does not need the server’s cooperation, which is exactly the property you want when the server is the thing that vanished.
The defect was in the bookkeeping. A connection is checked out slightly before it is registered and released slightly after, and those two narrow windows are real. A thread that stalls inside one of them is holding a slot that the registry cannot see, so the reaper cannot reap it. Rare, and permanent when it happens.
We closed the windows so the registration spans the entire time the slot is out. Then we wrote the test for the race that closing them introduces, because a callback the reaper has already picked up can still fire after the slot has gone back, and interrupting a socket that now belongs to someone else is a worse bug than the one we started with.
The part worth taking away
The useful lesson is not about pools. It is that a bound is only as good as its assumption about how the thing fails.
We had three timeouts and all of them assumed the failure would announce itself: the server would answer slowly, or the peer would go unreachable, or the connection would refuse. The actual failure was a peer that stayed perfectly polite while losing everything behind it, and politeness defeats all three.
When you write a timeout, write down what it assumes. Then ask what a failure looks like that satisfies the assumption and still ruins your day.