feat(restart): graceful gunicorn reload via supervisorctl signal HUP

[mrrobot47]

Summary

fm restart (and fmx restart inside the container) currently issue
supervisorctl stop && start for the web service. That stop briefly
closes the gunicorn listening socket; any client that connects in the
window between stop and start sees a TCP RST or
connect: connection refused, which reverse proxies surface as a 502.
For production deployments this is the dominant cause of a small but
consistent 502 burst on every restart — clients reading at the moment
of the stop also see in-flight requests RST'd.

This adds an opt-in --graceful flag that recycles workers via
supervisorctl signal HUP all instead. The gunicorn master keeps its
listening socket open across the reload, so upstream proxies observe no
connection-refused window and no in-flight RSTs from the listener.

Behavior

• fm restart <bench> --graceful — signals the frappe (gunicorn)
  container with SIGHUP via supervisorctl. The socketio container is
  restarted normally (node has no graceful-reload signal). Worker
  containers are not affected by this flag.
• fm restart <bench> (default) — unchanged, still runs
  supervisorctl restart all.
• fm restart <bench> --force — unchanged, still does the explicit
  stop && start cycle for callers that need a hard restart.
• fmx restart <service> --graceful — same semantics inside the
  container, dispatching SIGHUP to each process via the supervisor
  XML-RPC API. Composes with the existing --maintenance-mode and
  --drain-workers orchestration; mutually exclusive with --migrate.

--graceful is mutually exclusive with --force and ignored when
combined with --container (a full container restart can't be made
graceful at this layer).

Mechanism

The web service runs gunicorn with --preload. On SIGHUP, the
gunicorn master:

1. Re-reads its configuration.
2. Forks fresh workers against the existing listening socket.
3. Sends each old worker a graceful-shutdown signal once a replacement
   is ready.

At no point does the master close the listening socket, so a
connecting client never observes connect-refused / RST during the
swap. The existing --preload setup in _write_gunicorn_wrapper is
already compatible with this — no template changes required.

Backward compatibility

Default behavior is unchanged. The flag is opt-in and the existing
hard-restart path remains the default and is still reachable via
--force. No existing callers need to be updated. The pre-existing
force and default branches of BenchSupervisor.restart_supervisor_service
are untouched; the new graceful branch is added alongside them.

Test plan

• Unit tests added at tests/unit/site_manager/test_bench_supervisor_restart.py:
  • graceful sends supervisorctl signal HUP all and nothing else
  • graceful never issues stop / start / restart
  • graceful + force raises a BenchOperationException before any
    docker command is executed
  • docker failures during the SIGHUP exec surface as
    BenchOperationException
  • regression: force still does stop + start; default still does
    restart all
  • Bench.restart_web_containers_services only applies graceful to
    the frappe container, never to socketio
  • use_container_restart=True bypasses the supervisor path entirely
• Manual: fm restart <bench> --graceful against a running bench
  while a load generator hits /api/method/ping — confirm zero
  connection refused in the access log across the reload window
  (covered manually before opening this PR; reproduction script is
  out of scope here).

Caveats / follow-up

• With gunicorn --preload, SIGHUP recycles workers but does not
  re-import application code in the master. This flag is suitable for
  restart-without-RST (release leaked memory, apply runtime config
  changes that workers re-read on fork) but is not a substitute for
  a full restart when picking up new application code — for that, a
  follow-up could wire a --graceful=usr2 path that re-execs the
  gunicorn master. The docs and help text call this out so callers pick
  the right flag.
• The socketio container intentionally falls back to a normal restart
  under --graceful because node terminates on SIGHUP. A follow-up
  could investigate a custom signal handler in the socketio entrypoint
  to make socketio reloads non-disruptive too.
• Container-level restart (--container) cannot be made graceful at
  this layer — the right place for that is upstream connection-draining
  at the proxy, which is out of scope here.

[Copilot]

Pull request overview

Adds an opt-in “graceful restart” path for the web (gunicorn) service to avoid the brief connection-refused window caused by stop/start-style restarts, by using SIGHUP-based reloads via supervisord/supervisorctl.

Changes:

• Introduces --graceful to fm restart, wiring it through Bench.restart_web_containers_services and BenchSupervisor.restart_supervisor_service.
• Adds an analogous --graceful path to fmx restart, implemented as a supervisor XML-RPC signal dispatch.
• Adds/updates docs and unit tests covering graceful vs force vs default behavior and wiring (frappe-only graceful).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
frappe_manager/site_manager/modules/bench_supervisor.py	Adds graceful branch to restart supervisor services via supervisorctl signal HUP all and enforces force/graceful mutual exclusion.
frappe_manager/site_manager/site.py	Threads graceful through bench restart APIs; scopes graceful reload to frappe container only.
frappe_manager/commands/restart.py	Adds --graceful CLI option and validation/wiring into bench web restart.
Docker/frappe/fmx/fmx/commands/restart.py	Adds --graceful to container-side fmx restart and implements per-process SIGHUP signaling.
docs/commands/restart.md	Documents the new --graceful option and caveats (gunicorn --preload).
tests/unit/site_manager/test_bench_supervisor_restart.py	New unit tests covering graceful path, exclusivity, error propagation, and bench wiring behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.