fix: admin server dies silently by upgrading warp

[steve-chavez]

Fixes #5012. Uses yesodweb/wai#1084

[steve-chavez]

Tried this with https://github.com/PostgREST/postgrest-benchmark and export PGRSTBENCH_MAX_FDS="false", it gives the same result as #5064.

curl localhost:3001/ready -i

## hangs for a while then fails

HTTP/1.1 500 Internal Server Error
Transfer-Encoding: chunked
Date: Mon, 06 Jul 2026 22:24:58 GMT
Server: postgrest/15 (pre-release)
Content-Type: text/plain; charset=utf-8

Something went wrong

# it recovers
curl localhost:3001/ready -i

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Mon, 06 Jul 2026 22:38:51 GMT
Server: postgrest/15 (pre-release)

[steve-chavez]

Going to merge 9cce1d6 on main, seems generally useful.

[mkleczek]

Uses yesodweb/wai#1084

Thinking about it, I suppose it is inevitable we implement our own settingsAccept as in #5064 anyway - even if this one is fixed. We have a very specific set of requirements, eg:

1. warp solution keeps a separate connection counter per instance. In our case it would be much more useful to base the logic on a global connection count (taking into account both main and admin instance).
2. It would be very useful to reserve a small number of file descriptors so that admin server is always available (ie. always can accept at least one connection)

[Vlix]

@mkleczek It wasn't intended for this situation, but if you use the Settings from makeSettingsAndServerState in both warp instances, they will both use the same counter and shutdown flag. (you can adjust the individual settings, but using the original means the counter and flag at least are shared)

Now mind you, this hasn't been tested at all, so please take care if you do end up relying on this. Though I've checked the code and using the same counter/flag in two runSettings*'s in the same executable should just make both behave as if they were one warp server as far as connections and graceful shutdown are concerned.
(i.e. the waitForDecreased counter will see total connections and try again if in a request is done in one of the two instances, AND the moment one starts the graceful shutdown, they will both stop accepting requests and finish the already started request handlers. THOUGH the instance that did not initiate the shutdown will NOT stop/return: it will keep running the acceptLoop, but just not handle any requests)

[steve-chavez]

It would be very useful to reserve a small number of file descriptors so that admin server is always available (ie. always can accept at least one connection)

@Vlix Do you think Warp could include such an strategy?

[Vlix]

@Vlix Do you think Warp could include such an strategy?

I've looked a bit into "reserving file descriptors", but haven't really found anything on this.

Technically, this is already how accept works, though... in a way. It will grab a file descriptor before actually starting a connection; so when warp starts, one file descriptor is used to reserve the next incoming connection, and then when data comes in, it can handle and fork it.
And then after forking a thread to handle the request, a new file descriptor is requested to be ready for the next connection.

[steve-chavez]

It would be very useful to reserve a small number of file descriptors so that admin server is always available (ie. always can accept at least one connection)

On second thought, what would that buy us? Currently when we ran out of fds, a request to the Admin server will hang for a while then fail with 5xx, which is acceptable behavior. The only gain would be a faster 5xx response, but I'm not sure if that's worth it. On k8s side you can configure a short timeout for the probe anyway.

[wolfgangwalther]

Currently when we ran out of fds, a request to the Admin server will hang for a while then fail with 5xx, which is acceptable behavior. The only gain would be a faster 5xx response, but I'm not sure if that's worth it.

No, now that we don't do a TCP probe in the live check, we would report 200 OK, even when out of fds, wouldn't we? Which is the whole point, we don't want the pod to be suddenly killed under load - that was Michal's argument. So we need to make sure the admin server responds properly, which in turn means it needs to have a reserved file descriptor, otherwise it can't.

[steve-chavez]

No, now that we don't do a TCP probe in the live check, we would report 200 OK, even when out of fds, wouldn't we?

Not exactly, now a response to Admin /{live,ready} hangs until fds are available, then replies with 200 OK.

So we need to make sure the admin server responds properly, which in turn means it needs to have a reserved file descriptor, otherwise it can't.

So you're saying that Admin needs to respond quickly with a reserved fd when API server is under load otherwise the pod will be killed because of the probe timeout.

Let's assume that we had /$/ready and /$/live on the API server instead of a second Admin server. Would we need to reserve some fds for those specific endpoints too? Are there any other servers that do that?

I dunno but this concern of reserved fds seems like a last-ditch effort to me, we're much more constrained by db connections than by fds.

Perhaps we should expose a metric like process_open_fds? That way kubernetes could scale out before we ran out of fds?

[mkleczek]

I dunno but this concern of reserved fds seems like a last-ditch effort to me, we're much more constrained by db connections than by fds.

Nginx does this for the very same reasons.

[wolfgangwalther]

We can not represent these git-fetched dependencies on hackage. That's why we usually say we don't depend on unreleased versions of packages.

That has been (my) feedback consistently for quite some time now, whenever somebody wanted to introduce something like this.

[steve-chavez]

@Vlix Do you think a new release will come soon in Warp? 🙏

[Vlix]

yes 👉 warp-3.4.14

[mkleczek]

Let's assume that we had /$/ready and /$/live on the API server instead of a second Admin server. Would we need to reserve some fds for those specific endpoints too?

It won't work because we need an fd before we know what URI is requested.

I dunno but this concern of reserved fds seems like a last-ditch effort to me, we're much more constrained by db connections than by fds.

That's a very nuanced discussion.

PostgREST is a kind of a connection pooler. It multiplexes requests coming from a lot of HTTP connections using a limited number of PostgreSQL connections. HTTP connections can be short-lived but also can be very long-lived due to HTTP 1.1 keep-alive - a client may open a HTTP connection and keep it idle. The connection will eat and fd on the server but will not put any strain on db connection pool.

In k8s open file limit is by default only 1024. DB pool eats some portion of that. So I'd say it is not uncommon to run out of fds.

Perhaps we should expose a metric like process_open_fds? That way kubernetes could scale out before we ran out of fds?

I thought about it but this metric should already be available in k8s OOTB.

[Vlix]

Reserving file descriptors with the current warp code would be kind of difficult, I think.
The network library uses C functions to actually get the FDs, so first getting FDs and using them to make Sockets is very non-trivial.

[mkleczek]

Reserving file descriptors with the current warp code would be kind of difficult, I think. The network library uses C functions to actually get the FDs, so first getting FDs and using them to make Sockets is very non-trivial.

From what I read, Nginx opens a couple of /dev/null fds. Once it catches EMFILE it closes one of them and retries. This is, of course, racy and one needs to be careful not to consume freed fd somewhere else by accident.

Nevertheless, I think for PostgREST it would be enough to have a configurable connection count limit. We can implement it with a custom settingsAccept that would atomically $ check . (< limit) =<< currentConnectionCount before blocking in accept.

@steve-chavez @wolfgangwalther WDYT?

[Vlix]

Please create an issue in the wai repo describing the use case and please include the above explanation of how Nginx does it. That is potentially a way to implement it, but we'll need to be careful. (acquiring and closing correctly, not causing exhaustion ourselves, etc.)

I can't guarantee this will be implemented, nor if it is, how quickly, but it's an interesting feature.

[wolfgangwalther]

I think for PostgREST it would be enough to have a configurable connection count limit.

Not sure whether it needs to be configurable or whether it could be automatically deduced from the (soft) limit on file descriptors?

From what I read, Nginx opens a couple of /dev/null fds. Once it catches EMFILE it closes one of them and retries. This is, of course, racy and one needs to be careful not to consume freed fd somewhere else by accident.

Assuming that soft and hard limit on file descriptors are different, it could also be possible to just increase the soft limit in that case. I'm not sure whether that would work for nginx, because IIRC it automatically increases its soft limit to the max available hard limit anyway (?).

[mkleczek]

I think for PostgREST it would be enough to have a configurable connection count limit.

Not sure whether it needs to be configurable or whether it could be automatically deduced from the (soft) limit on file descriptors?

I tried to find a portable way to find out what the fd limit is but I don't think there is any (and I am not talking Windows, just MacOS, FreeBSD, Linux. Looks like configuration variable is simpler.

From what I read, Nginx opens a couple of /dev/null fds. Once it catches EMFILE it closes one of them and retries. This is, of course, racy and one needs to be careful not to consume freed fd somewhere else by accident.

Assuming that soft and hard limit on file descriptors are different, it could also be possible to just increase the soft limit in that case.

Right, but it doesn't solve the problem, just moves it.

IIRC it automatically increases its soft limit to the max available hard limit anyway (?).

It doesn't. Nginx has worker_rlimit_nofile configuration stanza.