feat: production hardening — persistent keypairs, metrics, integrity verification

Persist Ed448+ML-DSA-87 hybrid keypairs to disk for both CLI (~/.wharf/keys/)
and yacht-agent (/etc/wharf/keys/) with 0600 permissions, replacing ephemeral
generation. Wire Prometheus metrics and stats endpoints to real counters. Connect
mooring verify to BLAKE3 manifest verification. Add HTTP client timeouts and
connection pooling. Update all documentation and create TOPOLOGY.md.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: Jonathan D.A. Jewell

.machine_readable/META.scm

@@ -21,4 +21,12 @@
     (adr "ast-aware-sql-proxy"
       (status accepted)
       (description "Database proxy parses SQL AST to enforce table allowlists and block dangerous operations.")
-      (rationale "Prevents SQL injection at the wire protocol level, not just application level."))))
+      (rationale "Prevents SQL injection at the wire protocol level, not just application level."))
+    (adr "persistent-hybrid-keypairs"
+      (status accepted)
+      (description "CLI and agent persist Ed448+ML-DSA-87 keypairs to disk with restrictive file permissions.")
+      (rationale "Stable identity is required for trust establishment. Ephemeral keys prevent the yacht from recognizing returning controllers."))
+    (adr "ebpf-xdp-firewall"
+      (status accepted)
+      (description "Kernel-level packet filtering via eBPF XDP with nftables fallback.")
+      (rationale "XDP processes packets before the kernel network stack for lowest latency. Cascade to nftables ensures protection without CAP_BPF."))))

.machine_readable/STATE.scm

@@ -3,20 +3,27 @@
   (project-context
     (name "project-wharf")
     (purpose "Sovereign Web Hypervisor — immutable CMS infrastructure with offline admin (Wharf) and online runtime (Yacht)")
-    (completion-percentage 75))
+    (completion-percentage 90))
   (route-to-mvp
     (milestone "crypto-overhaul" (status complete) (description "Ed448 + ML-DSA-87 hybrid signatures, SHAKE3-512, XChaCha20-Poly1305, HKDF, Argon2id"))
     (milestone "http-mooring-client" (status complete) (description "MooringClient HTTP API: init → verify → rsync → commit"))
     (milestone "identity-file" (status complete) (description "SSH identity resolution: yacht → fleet → ~/.ssh/id_ed448 → agent"))
     (milestone "container-compliance" (status complete) (description "Chainguard bases, Containerfile naming, selur-compose orchestration"))
     (milestone "demo-script" (status complete) (description "End-to-end mooring + SQL injection blocking + file tampering detection"))
-    (milestone "production-hardening" (status pending) (description "Persist keypairs, eBPF loader, real database proxy, certificate management")))
+    (milestone "production-hardening" (status in-progress) (description "Production hardening milestone")
+      (sub-milestone "keypair-persistence" (status complete) (description "Persistent hybrid keypairs for CLI and yacht-agent with file permissions"))
+      (sub-milestone "ebpf-loader" (status complete) (description "Full XDP kernel program + userspace loader in crates/wharf-ebpf/"))
+      (sub-milestone "database-proxy" (status complete) (description "Full AST parser via sqlparser 0.39 in db_policy.rs"))
+      (sub-milestone "certificate-management" (status complete) (description "1024-line Nebula module in nebula.rs with CA, cert signing, IP allocation, revocation"))
+      (sub-milestone "metrics" (status complete) (description "Prometheus metrics and stats endpoints wired to real counters"))
+      (sub-milestone "http-resilience" (status complete) (description "Timeouts, connection pooling on mooring HTTP client"))
+      (sub-milestone "integrity-verification" (status complete) (description "Yacht agent mooring verify wired to BLAKE3 manifest verification"))
+      (sub-milestone "ed448-audit" (status pending) (description "ed448-goldilocks v0.14.0-pre.10 needs third-party audit before production"))))
   (components
     (component "wharf-core" (path "crates/wharf-core") (role "shared library: crypto, fleet, integrity, mooring, sync"))
     (component "wharf-cli" (path "bin/wharf-cli") (role "offline admin CLI: moor, integrity, fleet management"))
     (component "yacht-agent" (path "bin/yacht-agent") (role "online runtime: HTTP API, DB proxy, firewall, integrity enforcement"))
     (component "wharf-ebpf" (path "crates/wharf-ebpf") (role "eBPF programs for XDP firewall (optional, excluded from default build)"))
     (component "xtask" (path "xtask") (role "build tasks: eBPF compilation")))
   (blockers-and-issues
-    (blocker "ed448-goldilocks-pre-release" (severity medium) (description "v0.14.0-pre.10 is unaudited; needs audit before production"))
-    (blocker "keypair-persistence" (severity medium) (description "CLI generates ephemeral keypairs; need disk persistence with file permissions"))))
+    (blocker "ed448-goldilocks-pre-release" (severity medium) (description "v0.14.0-pre.10 is unaudited; needs third-party audit before production deployment"))))

CHANGELOG.adoc

@@ -13,19 +13,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Rust workspace with wharf-core, wharf-cli, and yacht-agent crates
 - DNS zone templates (simple, shared, standard, maximalist)
 - Nickel configuration schemas for security policies
-- Database policy engine with AST-based SQL filtering
+- Database policy engine with AST-based SQL filtering (sqlparser 0.39)
 - CMS adapters for WordPress, Drupal, Joomla, Moodle
 - Shell scripts for zone rendering, auditing, and deployment
 - Comprehensive Justfile with 50+ recipes
 - RSR (Rhodium Standard Repository) compliance documentation
 
+=== Production Hardening (2026-02-14)
+- *Persistent hybrid keypairs*: Wharf CLI (`~/.wharf/keys/wharf.key`) and Yacht Agent (`/etc/wharf/keys/yacht.key`) now persist Ed448 + ML-DSA-87 keypairs across restarts with proper file permissions (0600 private, 0644 public)
+- *Keypair serialization*: Custom binary format with `WHRF` magic, versioning, and optional password encryption via HKDF-SHAKE512 + XChaCha20-Poly1305
+- *eBPF XDP firewall*: Full kernel-level packet filtering with XDP programs and userspace loader (`crates/wharf-ebpf/`)
+- *Certificate management*: Complete Nebula certificate lifecycle — CA creation, cert signing, IP allocation, and revocation (`nebula.rs`)
+- *Prometheus metrics*: Real-time counters for queries (allowed/blocked/audited), mooring sessions, integrity checks, and moored status at `GET /metrics`
+- *Statistics endpoint*: `GET /stats` now returns actual counters instead of placeholder zeros
+- *File integrity verification*: Yacht Agent mooring verify phase now checks BLAKE3 manifests against on-disk files when `site_root` is configured
+- *HTTP client resilience*: Mooring client configured with 30s timeout, 10s connect timeout, and connection pooling (2 idle per host)
+- *Configuration*: Added `key_store_dir` (CLI + agent) and `site_root` (agent) config options
+
 === Security
 - Implemented AST-based SQL query analysis (not regex)
 - Added HTTP header airlock configuration
 - Defined filesystem immutability policies
 - Configured Nebula mesh network policies
 - Added FIDO2/WebAuthn authentication framework
 
+=== Known Issues
+- `ed448-goldilocks` v0.14.0-pre.10 is unaudited — needs third-party audit before production deployment
+
 == [0.1.0] - 2025-11-26
 
 === Added
@@ -47,6 +61,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 | Version | Date | Highlights |
 |---------|------|------------|
+| Unreleased | 2026-02-14 | Production hardening: persistent keypairs, eBPF, metrics, integrity verification |
 | 0.1.0 | 2025-11-26 | Initial release |
 
 == Upgrade Guide

README.adoc

@@ -60,7 +60,7 @@ Wharf inverts this:
 
 === Database "Virtual Sharding"
 
-The Yacht Agent acts as a SQL proxy, parsing queries using an *Abstract Syntax Tree* (not regex!) to enforce security policies:
+The Yacht Agent acts as a SQL proxy, parsing queries using an *Abstract Syntax Tree* (via `sqlparser` 0.39, not regex!) to enforce security policies:
 
 * *Mutable Tables* (Blue Zone): Content like comments and orders—allowed to write
 * *Immutable Tables* (Red Zone): Config like users and plugins—blocked unless from Wharf
@@ -76,6 +76,23 @@ INSERT INTO wp_users (user_login, user_pass) VALUES ('hacker', 'password')
 -- Error: Policy violation: write to immutable table 'wp_users'
 ----
 
+=== Persistent Hybrid Keypairs
+
+Both the Wharf CLI and Yacht Agent persist their Ed448 + ML-DSA-87 hybrid keypairs to disk, establishing stable identities across restarts.
+
+|===
+| Component | Key Location | Permissions
+
+| Wharf CLI | `~/.wharf/keys/wharf.key` | `0600` (private), `0644` (public)
+| Yacht Agent | `/etc/wharf/keys/yacht.key` | `0600`
+|===
+
+Keys use a custom binary format (`WHRF` magic + version + length-prefixed key material). Password-encrypted serialization is available via HKDF-SHAKE512 + XChaCha20-Poly1305 for offline storage.
+
+=== eBPF XDP Firewall
+
+The Yacht Agent supports kernel-level packet filtering via eBPF XDP programs for high-performance network enforcement. Falls back to nftables when CAP_BPF is unavailable.
+
 === Filesystem Immutability
 
 The Yacht filesystem is mounted read-only with specific writable "playgrounds":
@@ -88,6 +105,21 @@ The Yacht filesystem is mounted read-only with specific writable "playgrounds":
 | `/wp-content/plugins` | OverlayFS | "The Lie"—plugins think they write, but it's ephemeral
 |===
 
+=== File Integrity Verification
+
+The Yacht Agent verifies incoming BLAKE3 file manifests against on-disk state during the mooring verify phase. Mismatched, missing, and unexpected files are reported back to the Wharf CLI before any rsync transfer.
+
+=== Prometheus Metrics
+
+The Yacht Agent exposes real-time operational counters at `GET /metrics` (Prometheus format):
+
+* `yacht_queries_total{status="allowed|blocked|audited"}` — database proxy traffic
+* `yacht_mooring_sessions_total` — mooring session count
+* `yacht_integrity_checks_total` — integrity verification runs
+* `yacht_moored` — whether the agent is currently moored
+
+Statistics are also available as JSON via `GET /stats`.
+
 === HTTP Header Airlock
 
 A Rust-based HTTP proxy strips dangerous headers and injects security headers:
@@ -106,7 +138,7 @@ A Rust-based HTTP proxy strips dangerous headers and injects security headers:
 
 === Zero Trust Networking (Nebula)
 
-Admin ports are *invisible* to the public internet. The Yacht's management API (port 9000) only responds to devices with valid Nebula certificates.
+Admin ports are *invisible* to the public internet. The Yacht's management API (port 9001) only responds to devices with valid Nebula certificates. Full certificate lifecycle management (CA, signing, IP allocation, revocation) is built into `nebula.rs`.
 
 == Quick Start
 
@@ -236,6 +268,8 @@ The Wharf controller syncs state to the Yacht via a four-phase HTTP protocol:
 
 All init and commit requests carry hybrid signatures. The Yacht verifies both Ed448 and ML-DSA-87 components before accepting.
 
+The mooring HTTP client uses a 30-second request timeout, 10-second connect timeout, and connection pooling (2 idle connections per host) for resilient communication.
+
 === Threat Model
 
 Wharf assumes:
@@ -287,7 +321,9 @@ just sign-images
 
 == Configuration Reference
 
-Wharf uses https://nickel-lang.org/[Nickel] for declarative configuration:
+Wharf uses https://nickel-lang.org/[Nickel] for declarative configuration and TOML for runtime settings:
+
+=== Policy Files (Nickel)
 
 * `configs/fleet.ncl` - Define your Yachts
 * `configs/policies/database.ncl` - Database virtual sharding rules
@@ -296,6 +332,13 @@ Wharf uses https://nickel-lang.org/[Nickel] for declarative configuration:
 * `configs/policies/auth.ncl` - FIDO2 and session policies
 * `configs/policies/network.ncl` - Nebula mesh and firewall rules
 
+=== Runtime Configuration (TOML)
+
+Configuration is loaded with the following precedence (highest first): CLI args > environment variables > project-local (`.wharf/config.toml`) > user config (`~/.config/wharf/`) > system config (`/etc/wharf/`) > defaults.
+
+* `yacht-agent.toml` — Yacht Agent runtime settings (db proxy, firewall, API, site root, key store)
+* `wharf-cli.toml` — CLI settings (paths, build, mooring, state, key store)
+
 == Justfile Commands
 
 [source]

TOPOLOGY.md

@@ -0,0 +1,95 @@
+<!-- SPDX-License-Identifier: PMPL-1.0-or-later -->
+# TOPOLOGY.md - Project Wharf
+
+## System Architecture
+
+```
+                     THE WHARF (Offline Controller)
+                    +---------------------------------+
+                    |  wharf-cli                      |
+                    |  +-----------+  +-----------+   |
+                    |  | moor.rs   |  | fleet.rs  |   |
+                    |  | (keypair  |  | (yacht    |   |
+                    |  |  persist) |  |  config)  |   |
+                    |  +-----+-----+  +-----+-----+   |
+                    |        |              |          |
+                    |  +-----v--------------v-----+   |
+                    |  |       wharf-core          |   |
+                    |  | crypto | integrity | sync |   |
+                    |  | mooring_client | config   |   |
+                    |  | db_policy | fleet         |   |
+                    |  +-------------+-------------+   |
+                    +-----------------|--+--------------+
+                                     |  |
+              Mooring Protocol (HTTP) |  | rsync (SSH)
+              Ed448+ML-DSA-87 signed  |  |
+                                     |  |
+                    +-----------------|--+--------------+
+                    |  THE YACHT (Online Runtime)       |
+                    |  yacht-agent                      |
+                    |  +-----------+  +-------------+  |
+                    |  | Mooring   |  | DB Proxy    |  |
+                    |  | API       |  | (sqlparser  |  |
+                    |  | (init/    |  |  AST filter)|  |
+                    |  |  verify/  |  +------+------+  |
+                    |  |  commit)  |         |         |
+                    |  +-----+-----+  +------v------+  |
+                    |        |        | Shadow DB   |  |
+                    |  +-----v-----+  +-------------+  |
+                    |  | Integrity |                    |
+                    |  | (BLAKE3)  |  +-------------+  |
+                    |  +-----------+  | Firewall    |  |
+                    |                 | eBPF/nftab  |  |
+                    |  +-----------+  +-------------+  |
+                    |  | Metrics   |                    |
+                    |  | /metrics  |  +-------------+  |
+                    |  | /stats    |  | Nebula Mesh |  |
+                    |  +-----------+  +-------------+  |
+                    +----------------------------------+
+
+    +-----------------------------------------------------------+
+    |  wharf-ebpf (optional, excluded from default build)       |
+    |  XDP kernel program + userspace loader (aya 0.12)         |
+    +-----------------------------------------------------------+
+```
+
+## Completion Dashboard
+
+| Component              | Progress                     | Status |
+|------------------------|------------------------------|--------|
+| **wharf-core**         | `[██████████]` 100%          | Complete |
+| - crypto (Ed448+ML-DSA)| `[██████████]` 100%         | Hybrid sigs, HKDF, XChaCha20, Argon2id, keypair serialization |
+| - db_policy (AST)      | `[██████████]` 100%         | sqlparser 0.39, policy engine |
+| - integrity (BLAKE3)   | `[██████████]` 100%         | Manifest gen/verify, remote SSH verify |
+| - mooring protocol     | `[██████████]` 100%         | Init/verify/commit/abort, canonical signing |
+| - mooring_client       | `[██████████]` 100%         | HTTP client with timeouts + pooling |
+| - fleet/sync/config    | `[██████████]` 100%         | Fleet TOML, rsync, config hierarchy |
+| **wharf-cli**          | `[█████████░]` 90%           | Near-complete |
+| - moor operations      | `[██████████]` 100%         | Full mooring flow, persistent keypairs |
+| - fleet management     | `[██████████]` 100%         | Add/remove/list yachts |
+| - integrity audit      | `[████████░░]` 80%          | Local + remote SSH, API mode pending |
+| **yacht-agent**        | `[█████████░]` 90%           | Near-complete |
+| - DB proxy             | `[██████████]` 100%         | MySQL + PostgreSQL wire protocol |
+| - mooring API          | `[██████████]` 100%         | Init/verify/commit with sig verification |
+| - integrity verify     | `[██████████]` 100%         | BLAKE3 manifest verification wired |
+| - firewall (nftables)  | `[██████████]` 100%         | Rule generation, validation, runtime updates |
+| - metrics/stats        | `[██████████]` 100%         | Prometheus + JSON with real counters |
+| - keypair persistence  | `[██████████]` 100%         | /etc/wharf/keys/yacht.key |
+| **wharf-ebpf**         | `[████████░░]` 80%           | XDP program + loader, needs production testing |
+| **nebula.rs**          | `[██████████]` 100%          | CA, cert signing, IP allocation, revocation |
+| **Overall**            | `[█████████░]` 90%           | ed448 audit pending |
+
+## Key Dependencies
+
+| Dependency | Version | Purpose |
+|------------|---------|---------|
+| ed448-goldilocks | 0.14.0-pre.10 | Classical hybrid signature component (UNAUDITED) |
+| pqcrypto-mldsa | 0.1 | Post-quantum ML-DSA-87 signatures |
+| sqlparser | 0.39 | AST-based SQL query analysis |
+| axum | 0.7 | Yacht agent HTTP API |
+| reqwest | 0.12 | Mooring HTTP client |
+| blake3 | 1.5 | File integrity hashing |
+| chacha20poly1305 | 0.10 | Symmetric encryption |
+| aya | 0.12 | eBPF userspace loader |
+| tokio | 1.x | Async runtime |
+| clap | 4.4 | CLI argument parsing |