fix(hono-server): drain in-flight requests on shutdown (P1-3) (#1587)

Verification reframed the launch-readiness P1-3 finding: the kernel already
handles SIGINT/SIGTERM/SIGQUIT with an ordered, 60s-bounded graceful shutdown
(registerShutdownSignals + performShutdown + onShutdown hooks), and serve.ts
boots through it — so "no signal handling" was a false positive.

The real, narrower bug: HonoHttpServer.close() called closeAllConnections(),
force-killing in-flight requests. Now it drains: server.close() + drain active,
closeIdleConnections() to release idle keep-alive, and a bounded drain window
(default 10s, < kernel 60s) force-closes only stragglers so shutdown can't hang.

+2 integration tests (drain completes a slow request; force-close bounds a hung
one). docs/launch-readiness.md P1-3 updated with the verification finding.

Co-authored-by: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: 3 people

.changeset/graceful-http-drain.md

@@ -0,0 +1,16 @@
+---
+'@objectstack/plugin-hono-server': patch
+---
+
+fix(hono-server): drain in-flight requests on shutdown instead of force-closing (P1-3)
+
+`HonoHttpServer.close()` called `closeAllConnections()`, which terminated active
+connections mid-response — so a SIGTERM during a rolling deploy dropped in-flight
+requests. It now drains gracefully: `server.close()` stops accepting new
+connections and lets active requests finish, `closeIdleConnections()` releases
+idle keep-alive sockets so the process exits promptly, and a bounded drain window
+(default 10s, configurable, well under the kernel's 60s `shutdownTimeout`)
+force-closes only the stragglers so shutdown can't hang.
+
+Note: the kernel already handles SIGINT/SIGTERM/SIGQUIT with an ordered,
+timeout-bounded shutdown — this fixes the one place that wasn't draining.

docs/launch-readiness.md

@@ -121,14 +121,23 @@ fix or acceptance.**
   sweepers at startup; persist automation logs to a table rather than memory.
 - **Owner:** _______  ·  Verify ☐  ·  Sign-off ☐  ·  Notes: _______
 
-### P1-3 — No process-level graceful shutdown
-- **Area:** `runtime` (`http-server.ts`) + framework adapters
-- **Risk:** No SIGTERM/SIGINT handling → in-flight requests dropped on
-  rolling deploys (K8s SIGKILL after grace period); cluster (Redis) + kernel +
-  HTTP not drained in a coordinated order.
-- **Action:** Ship a graceful-shutdown utility (drain HTTP → stop kernel services
-  → close cluster) and wire it into app scaffolding; document a ≥60s grace floor.
-- **Owner:** _______  ·  Verify ☐  ·  Sign-off ☐  ·  Notes: _______
+### P1-3 — Graceful shutdown (mostly a false positive; one real drain bug fixed)
+- **Area:** `core` (`kernel.ts`), `cli` (`serve.ts`), `plugin-hono-server` (`adapter.ts`)
+- **Verification finding:** The sweep's "no SIGTERM/SIGINT handling" is **wrong**.
+  `Kernel.registerShutdownSignals()` (called at start) already handles
+  SIGINT/SIGTERM/SIGQUIT → `shutdown()` → `performShutdown()` (ordered plugin
+  destroy in reverse + `kernel:shutdown` hook + `onShutdown` handlers), bounded by
+  a **default 60s** `shutdownTimeout`. `serve.ts` boots through the kernel, so the
+  production path inherits all of this. The ≥60s grace floor already exists.
+- **Real (narrower) gap — FIXED:** the standalone Hono server's `close()` called
+  `closeAllConnections()`, which **force-killed in-flight requests** instead of
+  draining them. Replaced with: `server.close()` (stop new + drain active) +
+  `closeIdleConnections()` (release idle keep-alive), and force-close only after a
+  bounded **drain window** (default 10s, < the kernel's 60s). +2 integration tests.
+- **Residual (not blocking):** embedding framework adapters (express/fastify/…)
+  intentionally leave signal handling to the host app; cluster/Redis close should
+  be registered via `kernel.onShutdown(...)` by the cluster plugin — confirm it is.
+- **Owner:** _______  ·  Verify ✅ (mostly false positive; drain bug fixed)  ·  Sign-off ☐  ·  Notes: Kernel shutdown already correct; hono drain fixed + tested. Awaiting human sign-off.
 
 ### P1-4 — Per-request hostname → environment resolution (no cache)
 - **Area:** `rest` — `src/rest-server.ts:~504–530`

packages/plugins/plugin-hono-server/src/adapter-drain.test.ts

@@ -0,0 +1,52 @@
+// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
+
+import { describe, it, expect } from 'vitest';
+import { HonoHttpServer } from './adapter';
+
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
+
+/**
+ * P1-3 regression: `close()` must DRAIN in-flight requests (let them finish)
+ * rather than force-killing them, and must force-close only after the drain
+ * window elapses so shutdown can't hang.
+ */
+describe('HonoHttpServer — graceful close drains in-flight requests (P1-3)', () => {
+  it('lets an in-flight request complete instead of aborting it', async () => {
+    const server = new HonoHttpServer(0, undefined, 5000); // generous drain window
+    server.getRawApp().get('/slow', async (c) => {
+      await sleep(200); // still running when close() is called
+      return c.text('drained-ok');
+    });
+    await server.listen(0);
+    const port = server.getPort();
+
+    const reqP = fetch(`http://127.0.0.1:${port}/slow`);
+    await sleep(50); // ensure the request is being handled
+    const closeP = server.close();
+
+    const res = await reqP;
+    expect(res.status).toBe(200);
+    expect(await res.text()).toBe('drained-ok'); // completed, not reset
+    await closeP;
+  });
+
+  it('force-closes after the drain window so shutdown cannot hang', async () => {
+    const server = new HonoHttpServer(0, undefined, 100); // tiny drain window
+    server.getRawApp().get('/hang', async (c) => {
+      await sleep(5000); // far longer than the drain window
+      return c.text('late');
+    });
+    await server.listen(0);
+    const port = server.getPort();
+
+    const reqP = fetch(`http://127.0.0.1:${port}/hang`).catch((e) => e); // will be aborted
+    await sleep(50);
+
+    const t0 = Date.now();
+    await server.close();
+    const elapsed = Date.now() - t0;
+
+    expect(elapsed).toBeLessThan(2000); // didn't wait out the 5s request
+    await reqP; // the aborted request settled — fine
+  });
+});

packages/plugins/plugin-hono-server/src/adapter.ts

@@ -46,7 +46,13 @@ export class HonoHttpServer implements IHttpServer {
 
     constructor(
         private port: number = 3000,
-        private staticRoot?: string
+        private staticRoot?: string,
+        /**
+         * Max time (ms) to let in-flight requests drain on `close()` before
+         * force-closing the remainder. Kept well under the kernel's 60s
+         * `shutdownTimeout` so a slow request can't hang the whole shutdown.
+         */
+        private drainTimeoutMs: number = 10_000,
     ) {
         this.app = new Hono();
     }
@@ -285,12 +291,32 @@ export class HonoHttpServer implements IHttpServer {
 
     async close() {
         if (!this.server) return;
-        // Destroy all keep-alive sockets so the server stops immediately
-        if (typeof this.server.closeAllConnections === 'function') {
-            this.server.closeAllConnections();
-        }
-        await new Promise<void>((resolve, reject) => {
-            this.server.close((err: any) => (err ? reject(err) : resolve()));
+        const server = this.server;
+        // Graceful drain (P1-3): stop accepting new connections and let in-flight
+        // requests finish rather than force-killing them mid-response.
+        // `closeIdleConnections()` releases idle keep-alive sockets so the process
+        // can exit promptly; active requests keep running until they complete or
+        // the drain window elapses.
+        await new Promise<void>((resolve) => {
+            let settled = false;
+            const finish = () => { if (!settled) { settled = true; resolve(); } };
+
+            // Fires once every connection has ended (drained).
+            server.close(() => finish());
+            if (typeof server.closeIdleConnections === 'function') {
+                server.closeIdleConnections();
+            }
+
+            // Safety net: if requests outlast the drain window, force-close the
+            // remainder so shutdown can't hang past the kernel's shutdownTimeout.
+            const timer = setTimeout(() => {
+                if (typeof server.closeAllConnections === 'function') {
+                    server.closeAllConnections();
+                }
+                finish();
+            }, this.drainTimeoutMs);
+            if (typeof timer.unref === 'function') timer.unref();
         });
+        this.server = undefined;
     }
 }