docs: release-readiness sweep — spec + populate changelog, refresh guides, future-proof upgrade test (#585)

* docs(changelog): spec a human-readable changelog + populate [Unreleased]

Add specs/release/changelog.spec.yaml (release-changelog): every entry is a
user-facing sentence (>=5 words, no commit-subject prefix), Keep a Changelog
categories, dated version sections, no emoji. Enforced by
packaging/tests/changelog_test.go (source-inspection, AC-01..05), scoped to
the actively-edited [Unreleased] section so history is grandfathered.

Populate [Unreleased] with the post-rc.7 work in that style: Settings
activation (users, notifications, security/SSO), per-host SSH auth/sudo
learning, one-command safe upgrade, airgap + fresh-install fixes, and the
pre-release security-hardening batch (#584).

* docs: refresh hardening + install guides; de-hardcode upgrade test

- SECURITY_HARDENING.md: drop the 'not yet implemented' rate-limit/headers
  callout (shipped in #584); document auth rate limiting, CSRF, and security
  headers as live; note always-on breach screening; add an outbound-SSH
  persistent known-hosts subsection; version line to 0.2.0 rc series.
- install_guide.md: fix the stale DEB version example (rc.5 -> rc.7) and add an
  'Upgrading' section documenting the one-command auto-migrate + backup +
  fail-safe path and /etc/openwatch/upgrade.conf.
- upgrade-container-test.sh / run-upgrade-container-test.sh: stop hardcoding
  migration 34->35 and the host_connection_profile table. The host driver now
  derives the head migration's goose version_id and its -- +goose Down SQL and
  passes them in; the test reverses exactly the head migration and asserts a
  generic prior->head advance, so it survives every future migration.

(CLAUDE.md was also refreshed locally for AI sessions but is intentionally
gitignored per commit 7a73353, so it is not part of this commit.)

* test(packaging): pin upgrade-test driver to the host RPM arch

dist/ can hold a leftover cross-built arm64 RPM (the packaging Go suite
cross-builds it for AC-14), and the rockylinux:9 container runs the host
platform, so the previous `cp dist/openwatch-*-1.*.rpm` swept in both arches
and `rpm -i` collided on /usr/bin/openwatch. Glob a single host-arch RPM.

Validated end-to-end: the container test now passes — real `rpm -U` runs the
%post helper ($1=2), which stops the service, takes a pg_dump restore point,
applies the head migration (35 -> 36), and restarts.

Author: remyluslosius

.secrets.baseline

@@ -165,14 +165,14 @@
         "filename": "docs/engineering/install_guide.md",
         "hashed_secret": "c99a970222c9f5d73283b8be8021086dd666620b",
         "is_verified": false,
-        "line_number": 139
+        "line_number": 146
       },
       {
         "type": "Basic Auth Credentials",
         "filename": "docs/engineering/install_guide.md",
         "hashed_secret": "9d4e1e23bd5b727046a9e3b4b7db57bd8d6ee684",
         "is_verified": false,
-        "line_number": 370
+        "line_number": 377
       }
     ],
     "docs/engineering/prototypes/openwatch-v1/Host Management.html": [
@@ -2867,5 +2867,5 @@
       }
     ]
   },
-  "generated_at": "2026-06-15T21:15:28Z"
+  "generated_at": "2026-06-17T01:00:11Z"
 }

CHANGELOG.md

@@ -10,6 +10,74 @@ Versioning: [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+Settings became a working control panel, OpenWatch started learning how to
+reach each host over SSH, package upgrades became a single safe command, and a
+pre-release security review closed a batch of perimeter and access-control
+gaps.
+
+### Added
+
+- Settings > Users now lets you invite people, manage their accounts, and
+  assign roles from the UI instead of showing a placeholder (#552, #553).
+- Settings > Notifications can now send compliance alerts to Slack, a generic
+  webhook, or email over SMTP, and each channel can be edited after creation
+  (#554, #555).
+- Settings > Security is live end to end: scoped API tokens you can create and
+  revoke, an authentication policy (password strength and session timeouts),
+  and single sign-on through an OIDC identity provider (#556, #557, #558).
+- Settings > Audit and About now browse the audit log in-app and show the live
+  license and build details instead of static text (#552).
+- Each host's actions menu now has Edit and Delete entries so you can correct
+  or remove a host without leaving the list (#560).
+- OpenWatch now learns how to reach each host: it records which SSH
+  authentication method and sudo style actually worked and reuses them on the
+  next discovery, intelligence, and liveness pass, so it stops retrying
+  combinations that already failed (#566, #575, #576).
+
+### Changed
+
+- Upgrading is now one command. `dnf update` (or `apt upgrade`) applies any
+  pending database migrations automatically, takes a full database backup
+  first, and on a failed migration leaves the service stopped with clear
+  recovery steps instead of running against a half-migrated schema. The
+  PostgreSQL engine upgrade itself stays an operator-supervised step (#569).
+- Web fonts now ship inside the application instead of loading from a font CDN,
+  so the interface renders completely in air-gapped deployments (#561).
+- Updated the frontend build and CI tooling (Vite, Vitest, lucide-react, zod,
+  and several GitHub Actions) to current major versions (#571, #572, #573).
+
+### Fixed
+
+- A fresh install now boots on the first try: the Kensa rule corpus and the
+  server identity keys are provisioned during installation rather than failing
+  at first startup (#564).
+- The SMTP notification channel edit form now pre-fills its current settings
+  instead of opening blank (#561).
+- Removed leftover demo and sample data that could appear on the dashboard,
+  the activity feed, and the host lists (#562).
+
+### Security
+
+A pre-release security review (six parallel audit dimensions, every
+high-severity finding re-verified by hand) closed eight findings (#584):
+
+- State-changing requests made with a session cookie are now CSRF-protected
+  with a double-submit token; a request without a matching token is rejected.
+- The login and MFA-verify endpoints are now rate-limited per client address
+  to slow online password and one-time-code guessing.
+- Every response now carries security headers: HSTS, a content-security policy
+  that forbids framing, no-sniff, and a strict referrer policy.
+- SSH host keys learned on first connection are now stored in the database, so
+  a restart no longer re-trusts every host and a changed host key is detected
+  across restarts.
+- New passwords are now screened against a built-in list of common and breached
+  passwords, with an option to point at a full breach corpus; the check now
+  runs in production instead of being silently skipped.
+- Reading the audit-event API now requires the audit-read permission instead of
+  being open to any caller.
+- Creating an API token or assigning a role can no longer grant more access
+  than the caller already holds.
+
 ---
 
 ## [0.2.0-rc.7] Eyrie — 2026-06-14

docs/engineering/install_guide.md

@@ -247,7 +247,7 @@ PGPASSWORD='replace-with-a-strong-password' \
 ### Step 3 — Install the packages
 
 ```bash
-sudo apt install -y ./openwatch_0.2.0-rc.5_amd64.deb ./kensa-rules_0.4.3_all.deb
+sudo apt install -y ./openwatch_0.2.0-rc.7_amd64.deb ./kensa-rules_0.4.3_all.deb
 ```
 
 Install **both** files together — `openwatch` `Depends` on `kensa-rules` (the
@@ -404,6 +404,63 @@ the underlying error.
 
 ---
 
+## Upgrading
+
+Upgrading is one command. Download the newer `openwatch` package (and the newer
+`kensa-rules` package if the rule corpus moved) and install it the same way you
+did originally:
+
+```bash
+# RHEL family
+sudo dnf install -y ./openwatch-<new>.x86_64.rpm ./kensa-rules-<new>.noarch.rpm
+
+# Debian / Ubuntu
+sudo apt install -y ./openwatch_<new>_amd64.deb ./kensa-rules_<new>_all.deb
+```
+
+On an upgrade (and only on an upgrade — never on a fresh install) the package
+post-install step runs the upgrade helper, which:
+
+1. Checks the database is reachable. If it is not, it leaves the service alone,
+   prints how to finish later (`openwatch migrate && systemctl restart
+   openwatch`), and does **not** fail the package transaction.
+2. Stops the service so the old binary never runs against a half-migrated
+   schema.
+3. Takes a full `pg_dump` restore point into `/var/lib/openwatch/backups/`
+   before touching the schema. If the backup fails, it aborts **without**
+   migrating (fail-closed) — your data is untouched.
+4. Applies any pending migrations, then starts the service again.
+
+If a migration fails, the helper leaves the service **stopped** and exits
+non-zero so the package manager surfaces the problem, and it prints the restore
+path. Your data is intact (each migration runs in its own transaction and rolls
+back on error). After fixing the cause:
+
+```bash
+openwatch migrate            # re-apply; reads the same DSN from secrets.env
+sudo systemctl start openwatch
+```
+
+To preview what an upgrade would apply without changing anything:
+
+```bash
+sudo -u openwatch openwatch migrate --status
+```
+
+Tunables live in `/etc/openwatch/upgrade.conf` (a `noreplace` config file):
+`AUTO_BACKUP=yes|no` toggles the pre-migration dump, and
+`BACKUP_RETENTION_DAYS` controls pruning. A `systemd` timer
+(`openwatch-backup-cleanup.timer`) prunes old dumps daily but **always keeps the
+most recent one** regardless of age.
+
+> Scope: this automates the OpenWatch **application** schema only. A PostgreSQL
+> **engine** major-version upgrade (for example PostgreSQL 15 -> 16) is a
+> separate, operator-supervised `pg_upgrade` and is never triggered from a
+> package scriptlet. See `specs/release/upgrade.spec.yaml` for the full
+> contract.
+
+---
+
 ## Uninstall
 
 ### RPM

docs/guides/SECURITY_HARDENING.md

@@ -1,6 +1,6 @@
 # OpenWatch security hardening guide
 
-**Applies to:** OpenWatch 0.2.0-rc.5 (Go single-binary build; pre-release)
+**Applies to:** OpenWatch 0.2.0 pre-release (rc series; Go single-binary build)
 **Audience:** System administrators, security engineers, compliance officers
 
 This guide covers the security controls you operate when you deploy OpenWatch
@@ -62,6 +62,20 @@ Hardening steps you perform at the host level:
 Source: `packaging/common/openwatch.service`,
 `docs/engineering/install_guide.md` (Step 2).
 
+### Outbound SSH to managed hosts
+
+When OpenWatch connects to a managed host it validates the host key on a
+trust-on-first-use basis and **persists** the accepted key in PostgreSQL
+(`ssh_known_hosts`, migration 0036). The first connection records the key; every
+later connection compares against it and is rejected if the key changed
+(`ErrHostKeyMismatch`). Because the store is durable, a service restart does not
+re-trust hosts, so an attacker cannot MITM the first scan after a restart to
+harvest credentials. Presented keys are also strength-validated per NIST SP
+800-57 (RSA >= 2048, Ed25519 always accepted). To rotate a host's key
+intentionally (a re-provisioned host), delete its row from `ssh_known_hosts` so
+the next connection re-learns it. Source: `internal/knownhosts/store.go`,
+`internal/ssh/`.
+
 ---
 
 ## 3. Transport security (TLS)
@@ -164,7 +178,7 @@ Source: `packaging/common/openwatch.service`, `internal/config/config.go`
 | Session inactivity timeout | 15 minutes | `internal/identity/sessions.go` (`SessionInactivityWindow`) |
 | Session absolute timeout | 12 hours | `internal/identity/sessions.go` (`SessionAbsoluteWindow`) |
 | Password policy | Length only — 8 chars (regular), 15 chars (admin), max 128; NIST SP 800-63B | `internal/identity/password.go` |
-| Breach check | Optional corpus lookup rejects known-compromised passwords | `internal/identity/password.go` |
+| Breach check | Always-on in production: new passwords are screened against an embedded common/breached corpus (airgap-safe); point `OPENWATCH_BREACH_CORPUS_FILE` at a full HIBP list to extend it | `internal/identity/password.go`, `internal/identity/breach_corpus_default.go` |
 | MFA | TOTP enrollment and verification | `internal/identity/mfa.go` |
 
 The password policy is deliberately length-based with no character-class rules,
@@ -173,12 +187,12 @@ per NIST SP 800-63B. The first admin is created out-of-band with
 
 Source: `internal/identity/`, `cmd/openwatch/main.go` (`cmdCreateAdmin`).
 
-> Not yet implemented: there is no failed-login throttle, account lockout, or
-> per-IP brute-force backoff in the auth handlers. The Argon2id cost (~50–100 ms
-> per verification) is the only built-in slow-down on online guessing. Until
-> rate limiting lands (Section 9), protect `/api/v1/auth/login` with an upstream
-> control (a reverse proxy with rate limiting, or network ACLs) if you expose
-> 8443 beyond a trusted network. Source: `internal/server/auth_handlers.go`.
+`/api/v1/auth/login` and `/auth/mfa:verify` are rate-limited per client IP
+(Section 10), which throttles online guessing in addition to the Argon2id cost
+(~50-100 ms per verification). There is still no per-account lockout after N
+failed attempts, so for an internet-facing 8443 a reverse proxy or network ACL
+is still worthwhile as defense in depth. Source:
+`internal/server/auth_handlers.go`, `internal/server/ratelimit.go`.
 
 ---
 
@@ -297,7 +311,7 @@ Hardening steps:
 
 ---
 
-## 10. Rate limiting and request controls — current state
+## 10. Rate limiting, CSRF, and security headers
 
 The HTTP server sets request-hardening timeouts and size limits:
 
@@ -309,15 +323,22 @@ The HTTP server sets request-hardening timeouts and size limits:
 | `IdleTimeout` | 120 s | `internal/server/server.go` |
 | `MaxHeaderBytes` | 64 KiB | `internal/server/server.go` |
 
-> Not yet implemented: there is no per-user or per-IP HTTP rate-limiting
-> middleware, and no HTTP security-header middleware (HSTS, CSP, `X-Frame-Options`,
-> `X-Content-Type-Options`, `Referrer-Policy`, `Permissions-Policy`). The
-> `RateLimit` constants in the codebase belong to the intelligence and discovery
-> schedulers (how many hosts to enqueue per tick), not to the HTTP surface.
-> Until these land, enforce request rate limiting and inject security response
-> headers at an upstream reverse proxy if you expose 8443 publicly. Source:
-> `internal/server/server.go`, `internal/intelligence/scheduler/service.go`,
-> `internal/intelligence/discovery/scheduler/service.go`.
+The single binary serves the SPA and the API from one origin with no required
+edge proxy, so the perimeter controls run in the application itself:
+
+| Control | Behavior | Source |
+|---------|----------|--------|
+| Auth rate limiting | Per-client-IP sliding window on `POST /api/v1/auth/login` and `/auth/mfa:verify`; over the limit returns `429` + `Retry-After` and skips the credential check. The key is the direct connection address (`RemoteAddr`), not a client-supplied `X-Forwarded-For`. | `internal/server/ratelimit.go` |
+| CSRF | Double-submit token: login and refresh set a non-HttpOnly `XSRF-TOKEN` cookie, and unsafe (POST/PUT/PATCH/DELETE) cookie-authenticated requests must echo it in `X-CSRF-Token` (constant-time compare) or get `403 authz.csrf_invalid`. Bearer/token requests and `/api/v1/auth/*` are exempt. | `internal/server/csrf.go` |
+| Security headers | Every response carries HSTS (>=1 year, includeSubDomains), a Content-Security-Policy that denies framing (`frame-ancestors 'none'`, `default-src 'self'`; `/docs` relaxes script/style for Swagger but still denies framing), `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, and `Referrer-Policy: no-referrer`. | `internal/server/security_headers.go` |
+
+> Scope notes: auth rate limiting covers the credential-guessing surface, not
+> every route — there is still no general per-route HTTP limiter, and no request
+> body-size cap (`http.MaxBytesReader`) on JSON endpoints. If you expose 8443
+> publicly, an upstream reverse proxy is still useful for global rate limiting
+> and body-size enforcement. The scheduler `RateLimit` constants are unrelated
+> (they bound how many hosts the intelligence and discovery schedulers enqueue
+> per tick), see `internal/intelligence/scheduler/service.go`.
 
 ---