[cueweb/docs] Add per-facility connection health and runtime facility config (#2439)

## Related Issues

Main issue:
- #2016

Issues related to this PR:
- #2440

## Summarize your change.

[cueweb] Add per-facility connection health and runtime facility config

- [x] Build on the Cuebot Facility switcher with live per-facility
health and a runtime config admin screen, so each facility's gateway can
be monitored and re-pointed without a redeploy.
- [x] Per-facility health: app/api/facility/health probes every
configured facility's REST gateway in parallel and returns { name, ok,
latencyMs }. The header Cuebot Facility menu polls it every 30s
(use_facility_health),
shows a green/red dot per facility, and disables selecting a facility
whose gateway is down.
- [x] Runtime override store (lib/facility-store.ts): file-backed
per-facility gateway URL / JWT secret overrides plus an append-only
audit log. Layered over the env defaults by lib/facility-server.ts
(override > env > legacy), so an empty store reproduces the env-only
behavior exactly. Cached briefly to keep the per-request resolver off
the hot filesystem path; the JWT secret is written 0600 and never logged
or returned to clients.
- [x] Admin screen at /settings/facilities: edit each facility's gateway
URL and JWT secret at runtime (applied without redeploy) via a server
action, with a change-history table. Reachable from the "Manage
facilities…" item in the header facility menu.
- [x] Server/client split: move the gateway helpers (createJwtToken,
fetchObjectFromRestGateway, handleRoute) out of api_utils.ts into a
server-only app/utils/gateway_server.ts, and the override-aware
resolvers into lib/facility-server.ts. This keeps the filesystem-backed
store out of the client bundle (Next.js forbids node:fs in the client
graph, which api_utils is part of). All route handlers now import
handleRoute from gateway_server; lib/facility.ts stays pure/client-safe.
- [x] Document CUEWEB_FACILITY_STORE in .env.example (point it at a
mounted volume to persist overrides across container restarts).

[cueweb/docs] Document per-facility health and runtime facility config

Document the per-facility connection health + runtime config feature
across the CueWeb docs, and add the supporting screenshots.

- [x] quick-starts: the green/red health dot (30s poll, down = disabled)
and the Manage facilities... runtime admin screen +
CUEWEB_FACILITY_STORE persistence.
- [x] concepts: a "Per-facility health and runtime configuration"
subsection - the live health probe (reachability + latency only, down
facilities un-selectable) and the override model (resolution order
override > env > legacy default, file-backed store + audit log,
server-only credentials).
- [x] deploying-cueweb: document CUEWEB_FACILITY_STORE (mount a volume
to persist) and the runtime Manage-facilities screen + admin-gating
note.
- [x] other-guides + user-guide (user-facing): the Cuebot Facility menu
now shows a green/red health dot per facility (a down facility can't be
selected) and a "Manage facilities..." item; add a short how-to for the
admin screen (edit each facility's gateway URL / JWT secret at runtime,
change history, blank URL falls back to the default).
- [x] reference: add the CUEWEB_FACILITY_STORE env var; extend the
Cuebot Facility dropdown entry; add a GET /api/facility/health section
and a "Per-facility health and runtime config" section (resolution
order, override store, audit log, write serialization, fail-closed
authz).
- [x] tutorials: health-dot behavior in the switch step plus a
"Re-pointing a facility at runtime (admin)" walkthrough with the
manage-facilities screenshots and the admin-gating /
CUEWEB_FACILITY_STORE note.
- [x] developer-guide: add lib/facility-server.ts,
lib/facility-store.ts, app/utils/use_facility_health.ts,
app/utils/gateway_server.ts, app/settings/facilities/ and
api/facility/health/ to the file trees, plus an implementation note
(server/client split, store + audit, authz).
- [x] README: extend the Cuebot Facility entry with the per-facility
gateway pair, /api/facility/health polling + dot, and the Manage
facilities... screen.

## LLM usage disclosure
Parts of this solution's implementation were developed with assistance
from Claude Opus.

Author: ramonfigueiredo

cueweb/.env.example

@@ -25,6 +25,12 @@ NEXT_PUBLIC_OPENCUE_ENDPOINT=http://your-rest-gateway-url.com
 # CUEBOT_DEV_JWT_SECRET=dev-gateway-jwt-secret
 # CUEBOT_CLOUD_REST_GATEWAY_URL=https://cloud-rest-gateway.example.com
 # CUEBOT_CLOUD_JWT_SECRET=cloud-gateway-jwt-secret
+#
+# Per-facility URLs/secrets can also be edited at runtime (no redeploy) from the
+# /settings/facilities admin screen; those overrides are persisted to this file
+# and layered on top of the env vars above. Point it at a mounted volume to keep
+# overrides across container restarts (defaults to a file in the OS temp dir).
+# CUEWEB_FACILITY_STORE=/data/cueweb/facilities.json
 
 # Optional allow-list for the Stuck Frames "Last Line" reader
 # (/api/stuck-frames/lastline), as a colon-separated list of absolute path

cueweb/README.md

@@ -40,7 +40,7 @@ CueWeb replicates the core functionality of CueGUI (Cuetopia and Cuecommander) i
    - OpenCue logo (theme-aware: `opencue-icon-black.png` in light mode, `opencue-icon-white.png` in dark mode) followed by the **CueWeb** wordmark.
    - Six dropdown menus that mirror the CueGUI menu bar:
      - **File** → Disable Job Interaction (read-only safety toggle, see below).
-     - **Cuebot Facility** → `local` · `dev` · `cloud` · `external` (overridable via `NEXT_PUBLIC_CUEBOT_FACILITIES`). The active facility is shown as a small chip on the menu trigger.
+     - **Cuebot Facility** → `local` · `dev` · `cloud` · `external` (overridable via `NEXT_PUBLIC_CUEBOT_FACILITIES`). The active facility is shown as a small chip on the menu trigger. Each facility may target its own gateway via the server-only pair `CUEBOT_<NAME>_REST_GATEWAY_URL` / `CUEBOT_<NAME>_JWT_SECRET` (falling back to `NEXT_PUBLIC_OPENCUE_ENDPOINT` / `NEXT_JWT_SECRET`). The menu polls `/api/facility/health` every 30s to show a green/red dot per facility and disables selecting one whose gateway is down. **Manage facilities…** opens an admin screen (`/settings/facilities`) to edit each facility's gateway URL / JWT secret at runtime — applied without a redeploy, layered over the env defaults, with a change-history log; persist these overrides by pointing `CUEWEB_FACILITY_STORE` at a mounted volume.
      - **Cuetopia** → Monitor Jobs.
      - **CueCommander** → Allocations, Limits, Monitor Cue, Monitor Hosts, Redirect, Services, Shows, Stuck Frame, Subscription Graphs, Subscriptions. Unimplemented routes 404 gracefully until the corresponding pages land.
      - **Other** → **Attributes** (toggles the docked Attributes panel, see below), **Show Shortcuts** (opens the same overlay as pressing `?`), **Notify on Shortcut** (toggles the per-shortcut toast).

cueweb/app/__tests__/api/utils/api_utils.test_fetchObjectFromRestGateway.test.ts

@@ -15,7 +15,7 @@
  */
 
 import { loadClientEnvVars, loadServerEnvVars } from '@/app/utils/config';
-import { createJwtToken, fetchObjectFromRestGateway } from '@/app/utils/api_utils';
+import { createJwtToken, fetchObjectFromRestGateway } from '@/app/utils/gateway_server';
 import jwt from "jsonwebtoken";
 
 // Mock loadClientEnvVars and createJwtToken to return two predefined environment variables
@@ -25,8 +25,8 @@ jest.mock('@/app/utils/config', () => ({
   loadClientEnvVars: jest.fn(),
   loadServerEnvVars: jest.fn(),
 }));
-jest.mock('@/app/utils/api_utils', () => ({
-  ...jest.requireActual('@/app/utils/api_utils'), // Keep the original implementation for other functions
+jest.mock('@/app/utils/gateway_server', () => ({
+  ...jest.requireActual('@/app/utils/gateway_server'), // Keep the original implementation for other functions
   createJwtToken: jest.fn(), // Mock the createJwtToken function
 }));
 jest.mock('jsonwebtoken', () => ({

cueweb/app/__tests__/api/utils/api_utils_createJwtToken.test.ts

@@ -15,7 +15,7 @@
  */
 
 import { loadServerEnvVars } from '@/app/utils/config';
-import { createJwtToken } from '@/app/utils/api_utils';
+import { createJwtToken } from '@/app/utils/gateway_server';
 
 interface JwtParams {
   sub: string;

cueweb/app/api/allocation/getall/route.ts

@@ -14,7 +14,7 @@
  * limitations under the License.
  */
 
-import { handleRoute } from '@/app/utils/api_utils';
+import { handleRoute } from '@/app/utils/gateway_server';
 import { NextRequest, NextResponse } from "next/server";
 
 // Lists every allocation (for the subscription allocation dropdowns). The

cueweb/app/api/comment/action/delete/route.ts

@@ -14,7 +14,7 @@
  * limitations under the License.
  */
 
-import { handleRoute } from '@/app/utils/api_utils';
+import { handleRoute } from '@/app/utils/gateway_server';
 import { NextRequest, NextResponse } from "next/server";
 
 export async function POST(request: NextRequest) {

cueweb/app/api/comment/action/save/route.ts

@@ -14,7 +14,7 @@
  * limitations under the License.
  */
 
-import { handleRoute } from '@/app/utils/api_utils';
+import { handleRoute } from '@/app/utils/gateway_server';
 import { NextRequest, NextResponse } from "next/server";
 
 export async function POST(request: NextRequest) {

cueweb/app/api/facility/health/route.ts

@@ -0,0 +1,88 @@
+/*
+ * Copyright Contributors to the OpenCue Project
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { NextResponse } from "next/server";
+
+import { createJwtToken } from "@/app/utils/gateway_server";
+import { getAllFacilityTargets } from "@/lib/facility-server";
+
+/**
+ * Per-facility connection health (J2). Probes every configured facility's REST
+ * gateway in parallel and returns `{ name, ok, latencyMs }[]`, so the header
+ * facility menu can show a green/red dot per facility and disable selecting a
+ * facility whose gateway is down. Probes use a short timeout and never surface
+ * gateway payloads — only reachability + round-trip time.
+ */
+
+const PROBE_TIMEOUT_MS = 5000;
+
+interface FacilityHealth {
+  name: string;
+  ok: boolean;
+  latencyMs: number;
+  error?: string;
+}
+
+async function probe(name: string, gatewayUrl: string, jwtSecret: string): Promise<FacilityHealth> {
+  if (!gatewayUrl) {
+    return { name, ok: false, latencyMs: 0, error: "No gateway configured" };
+  }
+
+  let token: string;
+  try {
+    token = createJwtToken(
+      {
+        sub: "cueweb-facility-health",
+        role: "user-role",
+        iat: Math.floor(Date.now() / 1000),
+        exp: Math.floor(Date.now() / 1000) + 60,
+      },
+      jwtSecret,
+    );
+  } catch {
+    return { name, ok: false, latencyMs: 0, error: "JWT signing failed" };
+  }
+
+  const start = Date.now();
+  const controller = new AbortController();
+  const timer = setTimeout(() => controller.abort(), PROBE_TIMEOUT_MS);
+  try {
+    const response = await fetch(`${gatewayUrl}/show.ShowInterface/GetActiveShows`, {
+      method: "POST",
+      headers: { "Content-Type": "application/json", Authorization: `Bearer ${token}` },
+      body: "{}",
+      cache: "no-store",
+      signal: controller.signal,
+    });
+    await response.text().catch(() => undefined);
+    return { name, ok: response.ok, latencyMs: Date.now() - start };
+  } catch {
+    return { name, ok: false, latencyMs: Date.now() - start, error: "Gateway probe failed" };
+  } finally {
+    clearTimeout(timer);
+  }
+}
+
+export async function GET(): Promise<NextResponse> {
+  const targets = await getAllFacilityTargets();
+  const facilities = await Promise.all(
+    targets.map((t) => probe(t.name, t.gatewayUrl, t.jwtSecret)),
+  );
+  return NextResponse.json(
+    { facilities, checkedAt: new Date().toISOString() },
+    { status: 200 },
+  );
+}

cueweb/app/api/frame/action/createdependonframe/route.ts

@@ -14,7 +14,7 @@
  * limitations under the License.
  */
 
-import { handleRoute } from '@/app/utils/api_utils';
+import { handleRoute } from '@/app/utils/gateway_server';
 import { NextRequest, NextResponse } from "next/server";
 
 // Mirrors Frame.createDependencyOnFrame (pycue) ->

cueweb/app/api/frame/action/createdependonjob/route.ts

@@ -14,7 +14,7 @@
  * limitations under the License.
  */
 
-import { handleRoute } from '@/app/utils/api_utils';
+import { handleRoute } from '@/app/utils/gateway_server';
 import { NextRequest, NextResponse } from "next/server";
 
 // Mirrors Frame.createDependencyOnJob (pycue) ->