Skip to content

adr-0117-dependency-graph-runtime.md

Latest commit

 

History

History
86 lines (67 loc) · 7.46 KB

adr-0117-dependency-graph-runtime.md

File metadata and controls

86 lines (67 loc) · 7.46 KB
Raw

Workstream ADR 0117: Service Dependency Graph As First-Class Runtime

  • ADR: ADR 0117
  • Title: Persist a machine-usable DAG of service/host/network/cert dependencies in Postgres; expose traversal API for blast-radius, health propagation, and incident correlation used by triage and risk scoring
  • Status: live_applied
  • Branch: codex/ws-0117-live-apply
  • Worktree: ../proxmox-host_server_ws-0117-live-apply
  • Owner: codex
  • Depends On: adr-0048-command-catalog, adr-0054-netbox-topology, adr-0058-nats-event-bus, adr-0075-service-capability-catalog, adr-0092-platform-api-gateway, adr-0098-postgres-ha, adr-0113-world-state-materializer
  • Conflicts With: adr-0113-world-state-materializer (both read from NetBox; coordinate on refresh timing to avoid stampede)
  • Shared Surfaces: platform/graph/, graph.nodes and graph.edges Postgres schema, config/dependency-graph.yaml, config/windmill/scripts/graph/, /v1/graph/* API gateway endpoints

Scope

  • create Postgres migration migrations/0012_graph_schema.sql — graph.nodes and graph.edges tables with indexes from ADR 0117
  • create config/dependency-graph.yaml — initial manual edge declarations for all platform services (netbox→postgres, keycloak→postgres, windmill→postgres, mattermost→postgres, grafana→loki, grafana→prometheus, all services→step-ca, all services→docker-runtime)
  • create platform/graph/__init__.py
  • create platform/graph/client.py — DependencyGraphClient with descendants(), ancestors(), path(), neighbourhood(), health_propagation() methods using recursive CTEs
  • create config/windmill/scripts/graph/import-from-netbox.py — Windmill workflow: imports device→rack, IP, VLAN edges from NetBox world-state snapshot
  • create config/windmill/scripts/graph/import-from-catalog.py — Windmill workflow: imports depends_on edges from config/workflow-catalog.json
  • create config/windmill/scripts/graph/propagate-health.py — Windmill workflow subscribed to platform.world_state.refreshed on service_health surface; runs health propagation and emits derived_health_degraded NATS events
  • register /v1/graph/* routes on the platform API gateway (ADR 0092): GET /v1/graph/nodes, GET /v1/graph/nodes/{id}/descendants, GET /v1/graph/nodes/{id}/ancestors, GET /v1/graph/nodes/{id}/health, GET /v1/graph/path
  • write tests/unit/test_graph_client.py — test recursive traversal, cycle detection, empty graph, single-node graph

Non-Goals

  • Code-level or build-time dependency analysis
  • Replacing NetBox as the topology source of truth
  • Real-time edge updates (graph is rebuilt from sources on each refresh cycle, not updated incrementally)

Expected Repo Surfaces

  • migrations/0012_graph_schema.sql
  • config/dependency-graph.yaml
  • platform/graph/__init__.py
  • platform/graph/client.py
  • config/windmill/scripts/graph/import-from-netbox.py
  • config/windmill/scripts/graph/import-from-catalog.py
  • config/windmill/scripts/graph/propagate-health.py
  • docs/adr/0117-service-dependency-graph-runtime.md
  • docs/workstreams/adr-0117-dependency-graph-runtime.md

Expected Live Surfaces

  • graph.nodes contains rows for all platform services, VMs, and hosts
  • graph.edges contains at minimum all depends_on edges declared in config/dependency-graph.yaml
  • DependencyGraphClient().descendants("service:postgres") returns the correct set of dependent services
  • GET /v1/graph/nodes/service:postgres/descendants via the API gateway returns the same set
  • Stopping the health probe for a service causes a derived_health_degraded NATS event for its dependents within 60 seconds

Verification

  • Run pytest tests/unit/test_graph_client.py -v → all tests pass
  • Confirm SELECT count(*) FROM graph.nodes; returns at least 15 rows after the import workflows run
  • Call DependencyGraphClient().descendants("service:postgres") → verify keycloak, netbox, windmill, mattermost all appear
  • Simulate a health probe failure for postgres (set available: false in a test snapshot) → confirm derived_health_degraded NATS events arrive for downstream services within the propagation window

Merge Criteria

  • Unit tests pass including cycle-detection test
  • Graph populated for all current platform services
  • Health propagation verified end-to-end
  • API gateway endpoints responding with correct data
  • Risk scorer (ADR 0116) downstream_count stub replaced with real DependencyGraphClient().descendants() call

Outcome

  • Repository implementation remains tracked from repo release 0.117.0.
  • Live apply completed from the separate codex/ws-0117-live-apply branch on 2026-03-26 and is verified against platform version 0.130.20.
  • The replay repaired the Windmill graph runtime's dependency bootstrap path, exported the graph DSNs into the managed runtime environment, restored graph-schema grants for windmill_admin, and refreshes the ADR 0113 world_state.current_view materialized view during converge so the graph import workflows can run against populated world-state data.

Live Apply Note

  • Focused repository validation passed with the ADR 0117 pytest suite (29 passed), make syntax-check-windmill, make syntax-check-api-gateway, and scripts/validate_repository_data_models.py --validate.
  • make converge-windmill was replayed successfully from this worktree branch after the fixes landed, with the final run completing on docker-runtime, postgres, and proxmox-host without failed hosts.
  • Production verification confirmed world_state.current_view is populated, PostgreSQL holds the graph runtime rows, Windmill schedule arguments are stored in schedule.args, and authenticated gateway traversal returns the expected descendants and shortest path for the current service graph.
  • A repo-managed graph rebuild also completed successfully from the live guest checkout, proving the deployed worker wrapper can execute the managed scripts directly on docker-runtime.
  • Protected integration files still deferred to merge-to-main are VERSION, release sections in changelog.md, the top-level README.md integrated status summary, and versions/stack.yaml.

Notes For The Next Assistant

  • Recursive CTEs for descendants() must have a cycle guard: include a visited set in the CTE to avoid infinite loops if a graph cycle is ever introduced by a bad import. Log a warning and return the partial result rather than crashing.
  • The health_propagation() method should only propagate degraded status, not down. A down service means the health probe is failing; a derived_health_degraded on a downstream service means "your upstream is sick, you might be at risk". Conflating these two statuses in triage signals would cause confusion.
  • The import-from-catalog.py workflow reads config/workflow-catalog.json which is on disk in the repo. In Windmill, this means the workflow must clone or read the repo via the git resource. Coordinate with the platform API gateway workstream to ensure the catalog is also accessible via /v1/platform/catalog.
  • Edge de-duplication: the import workflows run on a schedule and re-insert edges. Use INSERT INTO graph.edges ... ON CONFLICT DO NOTHING to avoid duplicate edges accumulating over time.
  • During ad hoc Windmill API execution, pass dsn and world_state_dsn explicitly to the graph scripts even though the scheduled jobs persist those values in schedule.args; the schedule list endpoint may render args: null even when the database state is correct.