Release: MCP server, LangGraph incident agent, inference/RAG service, Azure infra#42
Merged
Conversation
- Introduced `local-https-setup.md` for configuring HTTPS in local development using mkcert. - Created `local-url-matrix.md` to standardize local API and service URLs. - Added `references.md` to compile official documentation and resources for core backend, database, ops, observability, security, AI integration, and data processing. - Updated `sandbox-aws-profile.md` to reflect changes in command usage for Floci. - Added `system-requirements.md` detailing system packages and installation steps for Ubuntu/Debian. - Removed outdated `tech-map.md` as its content is no longer relevant.
…security checklist to remove unnecessary S3 encryption command; enhance environment setup documentation with detailed variable guidance and remove outdated file
…te import streamlit from panels
…y reading, navigation and maintenance
… optional groups, remove unused - Create services/dashboard/pyproject.toml - Update dashboard Dockerfile to use own pyproject.toml
…om MVP scope module-level imports
…alongside DashboardApiError in every panel's data-fetching hook.
…ng hooks/workflows
… Azure provisioning Fix Python 2 except syntax in drift_events panel and new contract snapshot job. Add scheduled contract snapshot ingestion that fetches source responses and runs drift detection automatically. Add Azure Free Tier provisioning script (B1s VM + PostgreSQL Flexible Server). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add services/dashboard as a uv workspace member alongside ingestor. Switch Justfile ops recipe to use the ingress compose profile. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add register() to AuthResource API client, do_register() to AuthManager, and a Sign Up tab in the auth sidebar alongside Login. Includes client-side validation for required fields, password match, and minimum length. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
15 tests covering both job functions: inactive source skip, circuit breaker open (pre-check and mid-call), successful probe with sample persistence, upstream failure (503), network errors, URL construction, snapshot with/without drift detection, HTTP errors, non-dict responses, and JSON decode failures. Also fixes except syntax bug in run_source_contract_snapshot. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
512Mi was below what the chart's own entry-point template requires (redpanda 409Mi + reserve 201Mi = 610Mi), so the release failed at template execution before creating any resources. Bump to 2Gi, close to the chart's own upstream default (2.5Gi), for comfortable headroom on a local single-node dev cluster.
…rams docs/02-architecture/architecture.md (last updated April, ~2.5 months stale) described an aspirational multi-service design — processor/ inference/analytics services, MongoDB, Qdrant, AWS ECS Fargate — none of which exist; only ingestor+dashboard are implemented, and the real cloud target is an Azure VM with Terraform now owned entirely by the sibling infra repo. Archive it alongside the other two already-archived architecture docs. Add two lean, current-state replacements instead of one large doc, so each can be updated independently as services/infra actually change: - application-architecture.md: system context + container diagram, plus a router/feature map distinguishing MVP-active from present-but-deferred code (grounded in the actual routers/ directory and audit-gaps.md, not the old doc's claims). - infrastructure-architecture.md: one diagram, two zones — local playground (this repo) vs real cloud (api-observatory-infra) — matching the boundary rule fixed earlier this session. Both include a "How to Update" section, and CLAUDE.md's Plan Maintenance triggers now point to them so they don't go stale the same way.
k3s-deploy-infra installs the Bitnami postgresql chart under release
name "postgresql" (for chart in postgresql redis), which Bitnami's
naming convention collapses to service name "postgresql" (release
name == chart name, no "<release>-<chart>" suffix) — confirmed by the
chart's own NOTES output ("postgresql.data-zoo.svc.cluster.local").
secret.example.yaml pointed DATABASE_URL at "postgres-postgresql"
instead, a service that never existed, so /readyz's `SELECT 1` check
never succeeded and the ingestor rollout timed out at 0/2 ready.
namespace.yaml enforces pod-security.kubernetes.io/enforce: restricted, which requires seccompProfile.type: RuntimeDefault (or Localhost) on every pod. ingestor/dashboard never set it, so every pod-creation attempt was rejected by admission control after the namespace label landed — 0/2 and 0/1 ready with no pods ever appearing, not even crash-looping ones. postgresql-0/redis-master-0/redpanda-0 were unaffected only because they were created before this apply added the restricted label to the (previously unlabeled) namespace; PSA doesn't evict already-running pods retroactively.
Three separate bugs surfaced during local k3s validation: 1. ingestor probes hit /health and /readyz via the pod IP directly; kubelet's default Host header doesn't match TrustedHostMiddleware's allowlist (main.py), so every probe got 400 and the startupProbe killed the container after 6 failures. Set an explicit Host header on all three probes instead of loosening the middleware. 2. dashboard-deployment.yaml declared containerPort 8003 and probed /health + /readyz — but the container actually runs Streamlit on 8501 (per its Dockerfile) with only /_stcore/health. This manifest had never been exercised end-to-end before. Fixed port and probe paths; the Service's external port 8003 is unchanged since it's a separate mapping via the named "http" port. 3. k3s-up never ran migrations, unlike the docker-compose flow's separate `just migrate` step — ingestor logs showed UndefinedTableError for source_profiles on every startup. Add a one-off Job (migrate-job.yaml) running `alembic upgrade head` against the same secret, wired into k3s-up between k3s-secret and k3s-deploy so the schema exists before the app pods start.
docker-compose.yml sets SERVICE_VERSION explicitly (default 0.1.0); the k8s overlay never did, and the repo-root VERSION file fallback isn't copied into the Docker image, so get_service_version() raised an unhandled RuntimeError on every /health call — the probe fix alone wasn't enough, health itself was throwing 500s. Add service_version to the shared ConfigMap and wire it in via configMapKeyRef, mirroring the existing ingestor_url pattern.
…tion libs/version.py is deliberately strict/fail-fast — it exists for cross-service version-consensus checks (do all services agree on the same contracts version before a coordinated rollout, or has one drifted). But /health and /readyz, which Kubernetes uses to decide whether to kill or de-register a pod, called it directly in the request path. Discovered when /health crashed with an unhandled RuntimeError during local k3d validation (no SERVICE_VERSION, no VERSION file in the image) — a missing version string shouldn't be able to take down a healthy pod's liveness probe. - /health: drop version reporting entirely, matching its own docstring's promise to be lightweight and dependency-free. - /readyz: keep version as best-effort context, but resolve contracts version defensively (falls back to "unknown" + a log warning instead of raising); service version was already safely resolved via settings.app_version's default. - New GET /version: the actual deliberate consensus check. Calls the strict resolver unwrapped and fails loudly (500) on misconfiguration — that failure is the point, for CI/release tooling/operators to query before declaring a rollout or to detect drift.
… action
TREE_SHA=$(git rev-parse HEAD^{tree} | cut -c1-7) was independently
duplicated in 4 places across ci.yml (two build jobs), cd-dev.yml, and
release.yml. Extract it once into .github/actions/resolve-tree-sha so
there's a single source of truth for how image tags are computed —
pure DRY cleanup, no behavior change. cd-dev.yml's adoption (plus its
new rollback-input override) is a separate commit since it also wires
SERVICE_VERSION through to the deployed container.
CD pulled and tagged the correct tree-<SHA> image but never set
SERVICE_VERSION when starting docker compose — the running container
self-reported whatever was last written into the VM's .env at
provisioning time, not the SHA actually running. /health (soon
/version) on the real deployed VM was disconnected from reality.
- Write SERVICE_VERSION=tree-${TREE_SHA} into the VM's .env before
`docker compose up -d`, matching the image tag exactly.
- After the existing health check, curl /version and assert its
`service` field matches the deployed tag — a deploy that "succeeds"
but reports the wrong version now fails CD instead of shipping
silently.
- Add a workflow_dispatch trigger with an optional tree_sha input so a
known-good prior release can be redeployed on demand (rollback)
without reverting commits — same approval gate and health/version
checks as a normal deploy. The automatic workflow_run path is
unchanged when no input is given.
- Adopt the resolve-tree-sha composite action (previous commit) for
the "compute from current HEAD" path; the manual tree_sha input
bypasses it entirely when supplied.
…e wrong field
render_freshness_heatmap read SourceProfile.updated_at as "last probe
time" — that field only changes when the source's registration record
itself is edited (name, interval, etc.), not when the scheduler
actually probes it. Discovered live: a source with probes firing every
60s showed a frozen "Last probe" timestamp from registration time and
a permanent "stale" badge. /health/jobs-metrics already tracks the
real per-job last_run_at (used elsewhere via HealthResource) — wire
that in instead, keyed by the same "probe_source_{id}" job name the
scheduler registers.
Also fixes a latent sign inconsistency in the same function: the
string-timestamp branch computed (last - now), the datetime-object
branch computed (now - last) — opposite signs meaning a stale source
and a fresh one could get swapped depending on which type flowed
through. Both branches now consistently compute (now - last), and
naive ISO timestamps (the API returns no "Z"/offset) are explicitly
treated as UTC instead of being silently misinterpreted as local time.
Read jinfo.get("total_executions", 0) and jinfo.get("error_count", 0)
— neither key exists in /health/jobs-metrics' actual response shape
(success_count/failure_count, per health_ingestion_jobs.py). .get()'s
default silently masked the mismatch, so every job showed zero runs
regardless of real activity, even ones firing every 60s. Discovered
live: probe_source_1 clearly had multiple successful runs (confirmed
via logs and the now-fixed freshness heatmap) but this panel still
read 0.
register_source_probe_jobs was only ever called once, at app startup (main.py lifespan) — sources created afterward via POST /api/v1/sources were silently never scheduled until the next restart. Discovered live during an e2e walkthrough: a newly-registered source's scorecard stayed at sample_count=0 despite the source being reachable, because no probe job existed for it yet. Wire the scheduler into source_registry.py the same way it's already injected into health_ingestion_jobs.py (module-level set_scheduler() called from the lifespan), then call the existing (idempotent — already skips jobs that exist) register_source_probe_jobs() right after a source is created. Best-effort: a registration failure here logs a warning but doesn't fail the source-creation request, matching the same tolerance the startup path already has.
JobScheduler.job() (the @scheduler.job(...) decorator) only ever wrote to self._jobs — the actual APScheduler.add_job() wiring happened exclusively inside start()'s one-time loop at boot. A previous commit made register_source_probe_jobs() callable dynamically (after a source is registered via the API, post-startup), but that only updated the bookkeeping dict; the live scheduler engine never learned about the new job. Confirmed live: registering a second source showed total_scheduler_jobs go from 4 to 6 in the logs, but the new probe_source_2/contract_snapshot_source_2 jobs never actually fired. Extract the per-job wiring (trigger-disabled check, handler wrapping, add_job, logging) out of start()'s loop into _activate_job(), and have the job() decorator call it immediately whenever the scheduler is already running — so a job registered post-boot is live immediately instead of only taking effect on the next process restart. Added a regression test that registers a job after start() and asserts it's present in the live APScheduler engine (not just self._jobs) — verified it fails on the old code (AssertionError: assert None is not None) and passes with this fix.
…ritical drift
Phase 1 of the AI-augmented observatory plan: unifies the drift-detection
pipeline with the Observation/AgentRun models instead of leaving them as two
disconnected demos. create_contract_snapshot now creates an Observation
(tags=["incident", severity]) + a pending AgentRun in the same transaction
as the drift event, whenever severity=="critical" or event_type=="breaking".
No agent processing happens yet — that lands in Phase 3 once the LangGraph
graph exists to actually consume pending runs.
Also: named Postgres volume in docker-compose.dev.yml so local data persists
across restarts (base docker-compose.yml stays cloud-deploy-ready), and Bruno
collection fixes/additions (contracts/4 had body:none silently dropping its
payload, contracts/5 had a malformed meta block that crashed the CLI runner,
both referenced a stale {{token}} var) so the drift-trigger flow is
verifiable end-to-end via `bru run`.
uv.lock relocked by this sandbox's uv/Python 3.14 environment (drops a
stale exclude-newer options block); unrelated to Phase 1 dependencies.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d pgvector Postgres Phase 2 of the AI-augmented observatory plan. New services/inference/ — FastAPI on :8001 with POST /index and POST /search matching the contract services/ingestor/vector_search.py already expected, so /analyze's RAG context retrieval is real instead of silently falling back to "no context available". Backed by pgvector, not Qdrant: embeddings via fastembed (ONNX Runtime, CPU-only) rather than sentence-transformers, since this environment's torch build pulls a full CUDA dependency chain (~2GB) unconditionally on Linux — disproportionate for a free-tier Azure VM. Runs on its own dedicated Postgres instance (inference-db, port 5433) rather than sharing the ingestor's db — real per-service database ownership, own Alembic migration history. Documentation debt closed along the way: ADR-002 (the old "Qdrant primary" decision) turned out to be orphaned — it referenced a parent doc that only exists in docs/_archive/, left over from an abandoned pre-MVP design that was never cleaned up. Superseded by ADR-015, which documents the actual decision and explicitly leaves Qdrant on the table as a future option once there's a concrete need, not gated on budget alone. mvp-roadmap.md, decisions.md, and application-architecture.md's container diagram updated to match reality. Also de-linked three pre-existing dead ADR references (005/006/008) in decisions.md that docs-quality-markdown caught once this file was touched. Verified live end-to-end multiple ways: local uvicorn, the real Docker container, and the ingestor's actual vector_search.search_observation_documents() call over the Docker network — not just the inference service in isolation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 3 of the AI-augmented observatory plan. New services/ingestor/agent/ —
five linear nodes (classify_severity -> retrieve_similar_incidents ->
draft_analysis -> human_review -> notify), checkpointed to the ingestor's
own Postgres via langgraph-checkpoint-postgres so the human-in-the-loop
pause survives process restarts. Auto-triggered by critical/breaking drift
events (fire-and-forget from contract_drift.py, lazy-imported so a missing
`ai` extra degrades to a warning, not a crash). Resume via
POST /api/v1/agent/runs/{run_id}/resume.
Anthropic, not OpenAI: the plan's "structured OpenAI output" assumed a key
that isn't available here — claude-haiku-4-5 for the cheap classify step,
claude-sonnet-4-5 for the analysis a human actually reviews. New
anthropic_* settings mirror the existing openai_* pattern exactly.
Verified live end-to-end against the real stack with a real API key, twice
(approve + reject): incident triggered -> AgentRun reached awaiting_review
with a coherent root-cause analysis -> resumed via a separate HTTP call ->
status correctly flipped. RAG paid off organically: the second incident's
analysis referenced the first one by name, retrieved from what the first
run had indexed moments earlier.
Also: fixed a cross-suite test bug found while combining regression runs —
services/inference/tests/'s cleanup fixture did DROP SCHEMA public CASCADE,
silently wiping the ingestor's own tables when both suites share the
session-scoped test Postgres; scoped it to just its own tables instead, and
fixed the mirror-image gap in the ingestor's own fixture (missing
CREATE EXTENSION vector after its schema reset). Docker: added explicit
`AS runtime` stage aliases to the two Dockerfiles that were still missing
them (services/inference/Dockerfile already had it from Phase 2).
Docs: corrected ADR-012 (was live, unlike the orphaned ADR-002 — fixed
stale OpenAI model names and a "conditional routing" description that was
never actually built), mvp-roadmap.md, audit-gaps.md gap #7 -> resolved,
application-architecture.md, and this plan's own Phase 3 section text.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 4 of the AI-augmented observatory plan - closes audit-gaps.md gap 🟠#6 for these two routers using the jwt_role_guard pattern already established in contract_drift.py/source_registry.py/scorecards.py (JwtDep for reads, WriterJwtDep/ReviewerJwtDep for writes, AdminJwtDep for hard-delete). The dedicated auth-mechanism teaching routes in observations.py (/secure/*, /auth/login, /batch/protected) and all of observations_v2.py are left untouched by design. Also spoof-proofs the HITL resume endpoint: reviewer_user_id is no longer a client-supplied body field. It's derived server-side from the caller's authenticated JWT (sub -> User.id via the same lookup routers/auth.py already uses), degrading to None rather than trusting any client input when the subject has no matching user. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d/docs 4 Streamlit session-state properties in adapter.py (agent_run_id, agent_result, agent_hitl_paused, agent_stream_events) were scaffolding for the same abandoned /enrich//stream design as the just-deleted bruno/z-agent/ collection - confirmed unreferenced anywhere in the dashboard and not part of the UIAdapter Protocol contract. learning-paths.md's "Agent Enrichment -> HITL Review tab" claim was false (no such tab exists) and its other agent-track bullets described the same aspirational design rather than the real Phase 3 nodes/flow; corrected to point at the real API flow and the working bruno/agent/ collection. Also fixed a stale "4 services" compose count (Phase 2 added inference/inference-db). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The client fixture tried to force docs-auth off for tests via
os.environ.setdefault(...), but load_dotenv() runs first and already
populates those keys from .env, so setdefault silently no-ops -
API_V1_BEARER_TOKEN had no such attempt at all. Any dev following the
documented `cp .env.example .env` setup got 4 spurious local test
failures (docs/redoc/openapi/batch-protected assuming auth was
disabled) that stayed invisible in CI, where .env doesn't exist.
Force these to blank instead, matching the plain-assignment pattern
already used for ENVIRONMENT/CACHE_ENABLED two lines above. Also flip
.env.example's own DOCS_USERNAME/DOCS_PASSWORD default to blank,
matching its pre-existing comment ("leave empty to allow public
access") that the shipped default never actually followed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…script CLAUDE.md duplicated the global ACROSS principles verbatim; replace with a pointer to ~/.claude/CLAUDE.md. Docker sandboxes only read project-level config, so add scripts/refresh-sandbox-claude.sh to regenerate a self-contained .claude/CLAUDE.md (gitignored) by concatenating the global file with this repo's root CLAUDE.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New services/mcp/ - a stdio FastMCP server wrapping 11 tools (sources, scorecards, contract drift, agent-run status/approve-reject) as thin authenticated HTTP calls against the ingestor's real API, never in-process repository imports. Logs in as a dedicated mcp-service account via the actual /auth/token flow (auth_client.py), dogfooding Phase 4's JWT auth rather than bypassing it - per-loop cached token with expiry tracking, single forced re-login+retry on a 401. scripts/register_mcp_service_user.py handles the one-time account setup (idempotent). No docker-compose entry or port for v1 (stdio only, port 8006 reserved for a future streamable-http mode). Verified two ways: 16 unit tests (respx-mocked HTTP) plus a real end-to-end run against the live stack - real login, real reads, and a real 409 correctly propagated from resume_agent_run on a non-approvable run. Along the way, fixed a real bug in this script (a .local email placeholder failing strict validation) and flagged (but did not fix, out of scope) a pre-existing auth.py bug where login 500s if Cache is merely unreachable instead of failing open. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
create_session() only guarded against _session_client being None (Cache never connected) — a live client whose write raises (e.g. Cache merely unreachable) was uncaught, so POST /auth/token 500s on any transient Cache outage even though the JWT itself doesn't need Cache. create_refresh_token() right next to it already wraps its Cache call the same way; this brings create_session() in line with that existing fail-open pattern. Found live while verifying the new MCP server (services/mcp/) against the real stack. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…jobs without ACR creds Unit test collection failed because nodes.py imported langgraph at module scope unguarded, unlike the existing aiokafka optional-import pattern in events.py. Guarded it the same way, and marked the two graph-execution tests in test_graph.py with importorskip (they need a real langgraph runtime, not just an unblocked import). Docker build/push/scan jobs were hard-failing since no ACR secrets/vars are configured yet. They now skip cleanly via `if: vars.ACR_LOGIN_SERVER != ''` and the CI gate accepts skipped as well as success for those three checks — CodeQL and integration tests remain strictly required. Building will resume automatically once Azure credentials are added. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…ream CodeQL flagged py/stack-trace-exposure: on failure, the SSE error event yielded the raw OpenAI SDK exception (str(exc)) straight into the client-facing response body. The real exception is already logged server-side (logger.error); the client now gets a generic message. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CodeQL flagged py/clear-text-logging-sensitive-data on generate-secrets.py: _print_secret printed each freshly-generated secret value straight to stdout, which can land in terminal scrollback, screen-recording tools, or shared sessions. Values now go to .generated.secrets.env (mode 0600, truncated at the start of each run so stale secrets from a prior invocation never linger), gitignored so it can never be committed. Only the env var name and a "wrote to <path>" confirmation print to stdout. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
| global _output_file_started | ||
| mode = "w" if not _output_file_started else "a" | ||
| with _OUTPUT_ENV_PATH.open(mode, encoding="utf-8") as f: | ||
| f.write(f"{env_var}={value}\n") |
|
|
||
|
|
||
| def _print_secret(env_var: str, value: str, label: str, desc: str) -> None: | ||
| print(f"# ── {label} ──") |
|
|
||
| def _print_secret(env_var: str, value: str, label: str, desc: str) -> None: | ||
| print(f"# ── {label} ──") | ||
| print(f"# {desc}") |
| print(f"# ── {label} ──") | ||
| print(f"# {desc}") | ||
| _write_secret_line(env_var, value) | ||
| print(f"# Wrote {env_var} to {_OUTPUT_ENV_PATH}") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings
mainup to currentdevelop— 95 commits since the last release (PR #15, 2026-06-14). Highlights:mainuntil Azure Container Registry secrets are added to the repoTest plan
DATABASE_URL_TEST=sqlite+aiosqlite:///:memory: uv run pytest -q -m unit)skipped(no ACR credentials configured yet — expected, not a failure)baseline-checklist.md,application-architecture.md,mvp-roadmap.mdall updated in the same commit that shipped the MCP server🤖 Generated with Claude Code