Graceful shutdown

[atitan]

Hello,

We made the transition from puma to falcon recently, and it works really well.

We're seeing some random Interrupt exception in the log, found out to be caused by auto scaling's scale in, which send signal to containers to stop them.

I'd like to know if falcon was capable of handling graceful shutdown like puma do?

[ioquatix]

Not yet but it’s on track for the 1.0 release. If you have some time to help out, maybe you can explain your use case in a bit more detail.

[atitan]

We're using AWS EC2+ECS for production.
Every EC2 instance has 4 cores, where ECS places one task(containers) to each of them.
In the container, Falcon is forked to 5 processes to keep maximum usage of CPUs.

For deployment, we use full rolling update, which starts new container on every instances while old one still running.
ELB+ECS will eventually drain old containers and switch to new ones, so the signal exception will not cause error here.

When it comes to instance auto scaling, it operates directly on instances because it is a feature in EC2.
At the time instance getting terminated, it may have requests still being processed.

Hope the described scenario would help.

[ioquatix]

Thanks, that is helpful.

Are you using falcon serve and sigint?

[ioquatix]

Also it would be great to add a list of websites using falcon, or have used falcon. If you want to make a PR, including a link to your website, we can add it to the README.md.

[ioquatix]

I've got a cold so I may be unattentive on this issue for the next week, but hopefully I can pick it up soon. I've just spent some time playing around with async-io to flesh out what is required support graceful shutdowns in general, then that can be used by falcon.

[atitan]

Take care.

We use dumb-init as container entrypoint to rewrite and forward SIGTERM with SIGINT.
It's because Puma once had issue handling SIGTERM, which makes GKE unhappy about container status.
Using SIGINT makes Puma to return 0 on exit.
The outcome seemed great so we adopted this practice on every project we had since then.

[veesahni]

I have similar need for a rolling restart. Slightly different than what @atitan described, so I'd like to describe the use case:

• We have nginx proxying to multiple ruby servers
• We do a rolling restart of the ruby servers
• falcon would be managed by systemd
• systemd would be configured to send a signal defined by falcon (SIGQUIT, SIGTERM, etc) to the controller process to trigger a graceful shutdown
• Upon receiving the signal, it is expected that falcon immediately stop listening for and accepting new connections.. and thus nginx will take the node out of rotation automatically
• Upon receiving the signal, it is also expected that any idle child processes processes shut themselves down
• Once all idle processes are gone, it's expected that the main controller will shut down
• If the main controller is still up after 90s (this is systemd default), systemd will send a SIGKILL to forcefully bring it down
• Once the falcon is shutdown (either via graceful shutdown or via SIGKILL), systemd will start the process

In practice, we setup systemd with a KillSignal parameter configured for a graceful shutdown.
Thus systemctl stop falcon would gracefully shutdown..
.. and systemctl restart falcon would be nothing special - just a stop followed by a start.

More details about systemd's standard restart procedure is described in man systemd.kill or see here, online

[yard]

Hey guys! I have played a bit with graceful shutdown for falcon host trying to make it work locally but unfortunately didn't manage to make it work reliably. Will dump my approach and findings here, maybe someone (well, @ioquatix most likely :) will be able to suggest how to proceed.

Essentially, I decided to approach the graceful shutdown in the dumbest way possible: upon receiving a signal (I was handling INT, but it doesn't really matter and can be made configurable anyway) I invoke a (newly added) drain! method on the SharedEndpoint instance. This method, in its turn, is expected to stop accepting new connections immediately, wait for currently running requests to complete and shut down any active connections hanging around (there could be some even after completing all the request due to Keep-Alive).

I managed to get SharedEndpoint to stop accepting new connections by running calls to accept method in a loop with a small timeout until a flag is set (which gets enabled in drain! method). Once we complete handling the request(s), the connection(s) get closed, the loop exits, the application gets killed and everything is nice.

What I didn't manage to handle properly is exactly the Keep-Alive case. Whenever a connection is not closed by the client, it just stays in #each loop waiting for new requests and does not die – and since I was thinking about adding another signalling like drain! to connection object inside Async::HTTP::Server accept loop, but figured it might be better to ask first.

To sum up, I think the main challenge with my implementation is about implementing a proper signalling mechanism that can stop all these nested loops properly – almost like we use Thread.current.raise(Interrupt) to stop whatever code is running on INT signal normally (but we actually need something more fine-grained, since we don't want to stop the client logic). Adding drain! method throughout the whole stack seems weird and I believe there might be a better pattern?

[ioquatix]

I agree with improving this. The changes will need to take place in falcon and async-container. Using signals for everything might be tricky - there aren't really enough, so it might be the case we need to use some kind of RPC for detailed rolling restarts + status information.

I believe for the current implementation, sending SIGINT should start the shutdown process, followed by SIGTERM (or SIGKILL) to stop everything.