fix: reactivate disabled backend servers from the UI + v1.6.3 (Issue #24)

A backend server toggled OFF (is_active=false) vanished from the UI with no way
to reactivate it: GET /api/backends honored include_inactive for backends but the
server sub-queries hardcoded 'AND is_active = TRUE'.

- get_backends: server sub-queries now honor include_inactive (default callers
  unchanged); added last_config_status to the server payload so the UI can tell a
  DISABLED server (re-enableable) from a DELETION (pending delete).
- toggle_server: persists an entity snapshot so an Apply-Management Reject rolls
  back is_active (previously left the server stuck disabled).
- BackendServers.js: requests include_inactive, shows disabled servers with the
  ON/OFF switch + an 'Inactive' tag, hides only DELETION-pending servers, and
  keeps soft-deleted BACKENDS hidden (so include_inactive doesn't resurface them).
- Config generation unchanged: disabled servers stay '# DISABLED:' comments and
  convert back to live lines when re-enabled.

Startup migration hardening (multi-replica / rolling-deploy safety): create_essential_tables
fails fast on lock contention and retries; run_all_migrations is serialized by a
session advisory lock and gated by a schema_migrations version marker, so an
already-current schema is skipped instead of issuing lock-heavy DDL that a serving
replica's traffic could block at startup. Idempotent and fail-open.

Version reported consistently across all layers (version.json, backend fallback,
frontend package) -> 1.6.3.

Author: taylanbakircioglu

README.md

@@ -2367,6 +2367,7 @@ Developed with ❤️ for the HAProxy community
 
 ## Release Notes
 
+- **v1.6.3** (2026-06-01) — Bugfix: a backend **server** toggled OFF (`is_active=false`) disappeared from the UI with no way to reactivate it. `GET /api/backends` now honors `include_inactive` for servers (previously only backends), so disabled servers stay visible with an OFF switch + "Inactive" tag and can be re-enabled; soft-deleted (pending-delete) servers stay hidden. A Reject of a server toggle now correctly rolls back `is_active` (entity snapshot). Startup migrations are hardened for multiple replicas / rolling deploys (advisory lock + schema-version gate, so an already-current schema isn't re-migrated under a serving peer's load). Version is reported consistently across all layers. No schema changes; config generation unchanged (disabled servers stay commented out).
 - **v1.6.2** (2026-05-30) — Bugfixes: (1) agents (which authenticate with their `X-API-Key` token) could not reach `GET /api/clusters` / `/api/clusters/{id}` after the v1.5.x cluster-read hardening, breaking agent assignment ("401: Authorization header missing"); these endpoints now accept either a user JWT or an agent token (anonymous access is still rejected). (2) The uninstall-script generator returned 400 for macOS agents (which report platform `darwin`); it now normalizes the platform the same way the install generator does. No UI or schema changes.
 - **v1.6.1** (2026-05-21) — Security patch: bump `axios` to 1.16.x (prototype-pollution hardening, header-injection fix, keep-alive memory leak fix) and `fast-uri` to 3.1.2 (GHSA-v39h-62p7-jpjc). No functional changes.
 

backend/database/migrations.py

@@ -50,9 +50,62 @@ async def ensure_agents_table():
         await conn.execute("ALTER TYPE config_status ADD VALUE IF NOT EXISTS 'DELETION';")
         logger.info("Ensured REJECTED and DELETION values exist in config_status enum.")
 
-        # First, create essential tables if they don't exist
-        await create_essential_tables(conn)
-        
+        # First, create essential tables if they don't exist.
+        #
+        # Rolling-restart resilience: create_essential_tables() runs idempotent
+        # CREATE ... IF NOT EXISTS statements on every startup. Its CREATE INDEX
+        # block needs a SHARE lock that conflicts with concurrent writes (e.g.
+        # agent heartbeats updating backend_servers/agents). During a redeploy a
+        # writer can hold that lock, so the DDL blocked for the full 60s
+        # command_timeout -> TimeoutError -> startup crash -> crash-loop.
+        #
+        # Fix: fail fast on locks (lock_timeout), retry briefly, and on
+        # persistent contention SKIP the idempotent bootstrap and continue — on
+        # an established DB the objects already exist; a fresh DB has no writers
+        # so the first attempt always succeeds. lock_timeout is scoped to this
+        # call and RESET afterwards, so every other migration below keeps its
+        # original (wait-indefinitely) behavior. Non-lock errors still propagate
+        # (genuine schema problems must NOT be masked).
+        import asyncio as _asyncio
+        _lock_excs = (_asyncio.TimeoutError,)
+        try:
+            import asyncpg as _asyncpg
+            _lock_excs = _lock_excs + (
+                _asyncpg.exceptions.LockNotAvailableError,
+                _asyncpg.exceptions.QueryCanceledError,
+            )
+        except Exception:
+            pass
+        try:
+            await conn.execute("SET lock_timeout = '10s'")
+        except Exception:
+            pass
+        try:
+            for _attempt in range(1, 4):
+                try:
+                    await create_essential_tables(conn)
+                    break
+                except _lock_excs as _lock_err:
+                    if _attempt < 3:
+                        logger.warning(
+                            f"create_essential_tables: lock contention "
+                            f"(attempt {_attempt}/3), retrying in 3s "
+                            f"({type(_lock_err).__name__})"
+                        )
+                        await _asyncio.sleep(3)
+                    else:
+                        logger.warning(
+                            "create_essential_tables: persistent lock contention; "
+                            "skipping idempotent schema bootstrap and continuing "
+                            "startup (objects already exist on an established DB) "
+                            f"({type(_lock_err).__name__})"
+                        )
+        finally:
+            try:
+                await conn.execute("RESET lock_timeout")
+            except Exception:
+                pass
+
         # Ensure status column exists in config_versions table
         status_column_exists = await conn.fetchval("""
             SELECT 1 FROM information_schema.columns 
@@ -1639,13 +1692,105 @@ async def ensure_agent_activity_logs_table():
             await close_database_connection(conn)
         # Don't raise - this is not critical for system operation
 
+# Schema-version gate for the migration runner.
+#
+# >>> BUMP THIS whenever you add/modify ANY step in _run_all_migrations_inner()
+# >>> that changes the schema (table/column/index/constraint) OR seeded/role data
+# >>> (e.g. update_system_roles_to_enterprise_rbac). Otherwise the new step will
+# >>> NOT run on databases already marked at the current version.
+#
+# When the DB already records >= this version, run_all_migrations() skips the
+# whole (lock-heavy) idempotent sequence, so redeploys/scale-ups issue NO DDL and
+# a concurrently-serving replica's traffic cannot block ALTER / CREATE INDEX (the
+# rolling-deploy startup crash that motivated this gate).
+#
+# Backward compatibility (the product runs at many versions across companies):
+#  - First start on this code: no marker -> applied_version is NULL -> the FULL
+#    sequence runs (upgrades any prior version), THEN the marker is written. So
+#    upgrading from any older version is unaffected.
+#  - The marker is written ONLY after _run_all_migrations_inner() completes with
+#    no exception, so an interrupted/failed migration never marks an incomplete
+#    schema as done — the next start retries.
+#  - Behavior change vs the historical "re-run every idempotent ensure_* on every
+#    start": once marked, same-version restarts no longer re-run (and therefore no
+#    longer auto-repair manual drift). To force a re-run, bump SCHEMA_VERSION or
+#    delete the schema_migrations row.
+SCHEMA_VERSION = 1
+
+
 async def run_all_migrations():
-    """Run all database migrations"""
+    """Run all database migrations.
+
+    Hardened for multiple backend replicas / rolling deploys:
+    - A session-level advisory lock serializes the run so only one pod migrates
+      at a time (others wait, then hit the version gate and skip). It is
+      session-scoped, so it auto-releases if a pod dies mid-migration.
+    - A schema-version marker (schema_migrations) gates the run: when the DB is
+      already at SCHEMA_VERSION the whole idempotent sequence is skipped, so no
+      DDL is issued and a serving replica's traffic can't block it.
+    If the advisory lock or marker can't be used, we fall back to running the
+    (idempotent) migrations rather than crashing startup.
+    """
     logger.info("Starting database migrations...")
-    
+    MIGRATION_ADVISORY_LOCK_KEY = 1836016242  # single-key advisory space ("migr"); distinct from the (ns,id) locks used elsewhere
+    lock_conn = None
+    lock_acquired = False
+    try:
+        lock_conn = await get_database_connection()
+        try:
+            await lock_conn.execute("SELECT pg_advisory_lock($1)", MIGRATION_ADVISORY_LOCK_KEY)
+            lock_acquired = True
+            logger.info("Acquired migration advisory lock (migrations serialized across pods)")
+        except Exception as _lock_e:
+            logger.warning(f"Could not acquire migration advisory lock; proceeding (migrations are idempotent): {_lock_e}")
+
+        # Schema-version gate: skip the lock-heavy sequence if the DB is current.
+        applied_version = None
+        try:
+            await lock_conn.execute("""
+                CREATE TABLE IF NOT EXISTS schema_migrations (
+                    id INTEGER PRIMARY KEY DEFAULT 1,
+                    version INTEGER NOT NULL,
+                    applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                    CONSTRAINT schema_migrations_singleton CHECK (id = 1)
+                )
+            """)
+            applied_version = await lock_conn.fetchval("SELECT version FROM schema_migrations WHERE id = 1")
+        except Exception as _mk_e:
+            logger.warning(f"schema_migrations marker unavailable; running full migrations: {_mk_e}")
+            applied_version = None
+
+        if applied_version is not None and applied_version >= SCHEMA_VERSION:
+            logger.info(f"Schema already at version {applied_version} (>= {SCHEMA_VERSION}); skipping migration run.")
+            return
+
+        await _run_all_migrations_inner()
+
+        try:
+            await lock_conn.execute("""
+                INSERT INTO schema_migrations (id, version, applied_at)
+                VALUES (1, $1, CURRENT_TIMESTAMP)
+                ON CONFLICT (id) DO UPDATE SET version = EXCLUDED.version, applied_at = EXCLUDED.applied_at
+            """, SCHEMA_VERSION)
+            logger.info(f"Recorded schema version {SCHEMA_VERSION} in schema_migrations.")
+        except Exception as _wr_e:
+            logger.warning(f"Could not record schema version marker (migrations still applied): {_wr_e}")
+    finally:
+        if lock_acquired and lock_conn is not None:
+            try:
+                await lock_conn.execute("SELECT pg_advisory_unlock($1)", MIGRATION_ADVISORY_LOCK_KEY)
+            except Exception:
+                pass
+        if lock_conn is not None:
+            await close_database_connection(lock_conn)
+
+
+async def _run_all_migrations_inner():
+    """The full idempotent migration sequence. Runs under the migration advisory
+    lock and is gated by the schema-version marker in run_all_migrations()."""
     # First, ensure basic database schema exists
     await run_init_sql()
-    
+
     # Then run additional migrations
     await ensure_agents_table()
     await ensure_config_versions_metadata_column()

backend/main.py

@@ -8,7 +8,7 @@
 import asyncio
 from datetime import datetime, timedelta
 
-_version_info = {"version": "1.6.0", "releaseName": "Multi-Factor Authentication (MFA)", "releaseDate": "2026-05-18"}
+_version_info = {"version": "1.6.3", "releaseName": "Inactive server reactivation fix (issue #24)", "releaseDate": "2026-06-01"}
 for _vpath in ["/app/version.json", os.path.join(os.path.dirname(__file__), "..", "version.json")]:
     try:
         with open(_vpath) as _vf:

backend/routers/backend.py

@@ -333,31 +333,38 @@ async def get_backends(
                 """)
 
         result = []
+        # Issue #24: servers must honor include_inactive exactly like the backend
+        # queries above. Pre-fix these sub-queries hardcoded `is_active = TRUE`
+        # (added in f34a6ee to hide soft-deleted entities), so a server toggled
+        # OFF (is_active=false) vanished from the UI with no way to reactivate it.
+        # Default callers (include_inactive=false) keep the is_active filter →
+        # byte-identical behavior; include_inactive=true now also returns inactive
+        # (disabled / soft-deleted) servers. last_config_status is selected so the
+        # frontend can tell DISABLED (re-enableable) from DELETION (pending delete).
+        server_active_filter = "" if include_inactive else "AND is_active = TRUE"
         for backend in backends:
-            # Get servers for this backend with cluster_id (ONLY show active servers)
-            # CRITICAL FIX: Add is_active = TRUE filter to prevent soft-deleted servers from appearing
             if cluster_id:
-                servers = await conn.fetch("""
+                servers = await conn.fetch(f"""
                     SELECT id, server_name, server_address, server_port, weight, maxconn,
                            check_enabled, check_port, backup_server, ssl_enabled, ssl_verify, ssl_certificate_id,
                            ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers,
                            cookie_value, inter, fall, rise,
-                           is_active, cluster_id,
+                           is_active, cluster_id, last_config_status,
                            haproxy_status, haproxy_status_updated_at, backend_name
-                    FROM backend_servers 
-                    WHERE backend_name = $1 AND cluster_id = $2 AND is_active = TRUE ORDER BY server_name
+                    FROM backend_servers
+                    WHERE backend_name = $1 AND cluster_id = $2 {server_active_filter} ORDER BY server_name
                 """, backend["name"], cluster_id)
             else:
-                servers = await conn.fetch("""
+                servers = await conn.fetch(f"""
                     SELECT id, server_name, server_address, server_port, weight, maxconn,
                            check_enabled, check_port, backup_server, ssl_enabled, ssl_verify, ssl_certificate_id,
                            ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers,
                            cookie_value, inter, fall, rise,
-                           is_active, cluster_id,
+                           is_active, cluster_id, last_config_status,
                            haproxy_status, haproxy_status_updated_at, backend_name
-                    FROM backend_servers 
-                    WHERE backend_name = $1 AND is_active = TRUE ORDER BY server_name
-                """, backend["name"]) 
+                    FROM backend_servers
+                    WHERE backend_name = $1 {server_active_filter} ORDER BY server_name
+                """, backend["name"])
 
             # Prepare server list with real-time HAProxy status from agents
             server_list = []
@@ -413,6 +420,7 @@ async def get_backends(
                     "fall": s.get("fall"),
                     "rise": s.get("rise"),
                     "is_active": s["is_active"],
+                    "last_config_status": s.get("last_config_status") or "APPLIED",  # Issue #24: lets UI distinguish DISABLED (re-enableable) from DELETION
                     "status": server_status,
                     "status_age_minutes": status_age_minutes,
                     "last_status_update": s.get("haproxy_status_updated_at").isoformat().replace('+00:00', 'Z') if s.get("haproxy_status_updated_at") else None,
@@ -1909,12 +1917,12 @@ async def toggle_server(server_id: int, request: Request, authorization: str = H
 
         conn = await get_database_connection()
 
-        # Get server info
+        # Get server info. Issue #24: fetch the FULL row (not just 5 columns) so we
+        # can snapshot the pre-toggle state for reject-rollback (see config version below).
         server = await conn.fetchrow("""
-            SELECT id, server_name, backend_name, is_active, cluster_id
-            FROM backend_servers WHERE id = $1
+            SELECT * FROM backend_servers WHERE id = $1
         """, server_id)
-        
+
         if not server:
             await close_database_connection(conn)
             raise HTTPException(status_code=404, detail="Server not found")
@@ -1975,22 +1983,45 @@ async def toggle_server(server_id: int, request: Request, authorization: str = H
 
                 # Generate new HAProxy config (after database commit)
                 config_content = await generate_haproxy_config_for_cluster(cluster_id)
-                
+
                 # Create new config version
                 config_hash = hashlib.sha256(config_content.encode()).hexdigest()
                 version_name = f"server-{server_id}-toggle-{int(time.time())}"
-                
+
                 # Get system admin user ID for created_by
                 conn2 = await get_database_connection()
                 admin_user_id = await conn2.fetchval("SELECT id FROM users WHERE username = 'admin' LIMIT 1") or 1
-                
+
+                # Issue #24: persist an entity snapshot so a Reject of this toggle
+                # rolls back is_active to its pre-toggle value. Pre-fix the toggle's
+                # config version carried NO metadata, so reject only reset
+                # last_config_status and the server stayed disabled (out of sync with
+                # the still-active live config). Mirrors the server-edit snapshot path;
+                # reject's rollback_entity_from_snapshot('server') restores is_active.
+                import json
+                from utils.entity_snapshot import save_entity_snapshot
+                entity_snapshot_metadata = await save_entity_snapshot(
+                    conn=conn2,
+                    entity_type="server",
+                    entity_id=server_id,
+                    old_values=dict(server),            # full pre-toggle row
+                    new_values={"is_active": new_status},
+                    operation="UPDATE",
+                )
+                old_config = await conn2.fetchval("""
+                    SELECT config_content FROM config_versions
+                    WHERE cluster_id = $1 AND status = 'APPLIED' AND is_active = TRUE
+                    ORDER BY created_at DESC LIMIT 1
+                """, cluster_id)
+                metadata = {"pre_apply_snapshot": old_config or "", **entity_snapshot_metadata}
+
                 # Create PENDING config version
                 config_version_id = await conn2.fetchval("""
-                    INSERT INTO config_versions 
-                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status)
-                    VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING')
+                    INSERT INTO config_versions
+                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status, metadata)
+                    VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING', $6)
                     RETURNING id
-                """, cluster_id, version_name, config_content, config_hash, admin_user_id)
+                """, cluster_id, version_name, config_content, config_hash, admin_user_id, json.dumps(metadata))
 
                 logger.error(f"SERVER TOGGLE DEBUG: Created PENDING config version {version_name} for cluster {cluster_id}")
 

frontend/package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "haproxy-openmanager-frontend",
-  "version": "1.6.2",
+  "version": "1.6.3",
   "description": "HAProxy Load Balancer Management UI",
   "license": "AGPL-3.0-or-later",
   "dependencies": {