Skip to content

Commit c8cf146

Browse files
OnlineChefcodesmith-bot
OnlineChef
and
codesmith-bot
authored
feat(ops): bc-scan-arm ChefVault fail-closed packaging (#188)
* feat(ops): add bc-scan-arm ChefVault system packaging Add a system-service layout for always-on Kater on bc-scan-arm: dedicated kater user paths under /opt /etc /var, non-secret kater.conf, and fail-closed startup via kater-with-chefvault.py. Documents cutover gates so laptop joep is no longer the canonical runtime. * fix(ops): scrub org and data-plane identifiers from packaging examples Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com> * fix(ops): chown release, provision .kater, keep chef-vault profile Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com> --------- Co-authored-by: OnlineChef <280567955+OnlineChef@users.noreply.github.com> Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
1 parent abaef6d commit c8cf146

5 files changed

Lines changed: 273 additions & 0 deletions

File tree

‎docs/deploy-server.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,11 @@
22

33
Run Kater on a machine you control. **Always enable auth before public exposure.**
44

5+
For the ChefGroep always-on runtime target (`bc-scan-arm`), prefer the system
6+
service packaging in `docs/ops/bc-scan-arm-runtime.md` and
7+
`scripts/systemd/kater-system.service.example` (dedicated `kater` user, `/opt/kater`,
8+
ChefVault fail-closed bootstrap). Keep this page for generic self-managed deploys.
9+
510
## Quick start (Tailscale / private network)
611

712
```bash

‎docs/ops/bc-scan-arm-runtime.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Kater runtime on bc-scan-arm
2+
3+
Target: move the canonieke Kater runtime off laptop `joep` onto always-on
4+
`bc-scan-arm`, with ChefVault fail-closed secret bootstrap and no
5+
`/home/<person>` paths.
6+
7+
This is packaging + runbook only. Do **not** cut over production traffic until
8+
shadow deploy + soaktest pass.
9+
10+
## Layout
11+
12+
```text
13+
/opt/kater/releases/<git-sha>/
14+
/opt/kater/current -> /opt/kater/releases/<git-sha>
15+
/etc/kater/kater.conf # non-secret config (0640, root:kater)
16+
/etc/kater/broker-token # ChefVault consumer token (0600, kater:kater)
17+
/var/lib/kater # SQLite / settings state
18+
/var/cache/kater # fleet cache and other regenerable data
19+
/var/log/kater # optional file logs (journald remains primary)
20+
/run/kater # RuntimeDirectory (tmpfs)
21+
```
22+
23+
## Service identity
24+
25+
```bash
26+
sudo useradd --system --home /var/lib/kater --shell /usr/sbin/nologin kater
27+
sudo install -d -o kater -g kater -m 0750 /var/lib/kater /var/cache/kater /var/log/kater
28+
sudo install -d -o root -g kater -m 0750 /etc/kater
29+
```
30+
31+
## Non-secret config
32+
33+
```bash
34+
sudo install -o root -g kater -m 0640 \
35+
/opt/kater/current/scripts/systemd/kater.conf.example \
36+
/etc/kater/kater.conf
37+
```
38+
39+
Edit host-specific values:
40+
41+
- `UTRECHT_FLEET_INVENTORY_PATH=/var/cache/kater/utrecht-fleet/inventory/fleet.json`
42+
- keep `KATER_HOST=127.0.0.1` while `KATER_AUTH_MODE=none`
43+
- keep CI SSH target pointing at `ubuntu@bc-scan-2`
44+
45+
## ChefVault broker token
46+
47+
```bash
48+
sudo install -o kater -g kater -m 0600 /dev/null /etc/kater/broker-token
49+
sudo -u kater tee /etc/kater/broker-token >/dev/null <<'EOF'
50+
<kater broker token>
51+
EOF
52+
sudo chmod 0600 /etc/kater/broker-token
53+
```
54+
55+
Token scope:
56+
57+
- profile `kater-dev-tools/ops`
58+
- only `Kater/*` collections needed by that profile
59+
- no admin / unrelated collections
60+
61+
Broker URL (private, not public):
62+
63+
```bash
64+
# example in /etc/kater/kater.conf
65+
CHEF_VAULT_BROKER_URL=http://127.0.0.1:8322
66+
```
67+
68+
## Install release
69+
70+
```bash
71+
SHA="$(git -C /path/to/checkout rev-parse HEAD)"
72+
sudo mkdir -p "/opt/kater/releases/$SHA"
73+
sudo rsync -a --delete \
74+
--exclude .git --exclude .venv --exclude .kater \
75+
/path/to/checkout/ "/opt/kater/releases/$SHA/"
76+
sudo chown -R kater:kater "/opt/kater/releases/$SHA"
77+
# Runtime state dir the unit lists in ReadWritePaths; systemd needs it to exist
78+
# before ExecStart, and rsync excluded it from the release.
79+
sudo install -d -o kater -g kater -m 0700 "/opt/kater/releases/$SHA/.kater"
80+
sudo ln -sfn "/opt/kater/releases/$SHA" /opt/kater/current
81+
sudo -u kater bash -lc 'cd /opt/kater/current && uv sync --frozen'
82+
```
83+
84+
The release must be owned by `kater` before `uv sync`: the virtualenv is created
85+
project-local at `/opt/kater/current/.venv`, and the unit's `ExecStart` depends on
86+
`/opt/kater/current/.venv/bin/python` existing.
87+
88+
## Fleet cache bootstrap
89+
90+
```bash
91+
sudo -u kater bash -lc '
92+
mkdir -p /var/cache/kater
93+
if [ ! -d /var/cache/kater/utrecht-fleet/.git ]; then
94+
git clone --depth 1 git@github.com:<org>/<fleet-inventory-repo>.git \
95+
/var/cache/kater/utrecht-fleet
96+
else
97+
git -C /var/cache/kater/utrecht-fleet pull --ff-only
98+
fi
99+
test -f /var/cache/kater/utrecht-fleet/inventory/fleet.json
100+
'
101+
```
102+
103+
Prefer atomic refresh later (clone/pull into temp dir → validate → rename).
104+
105+
## Systemd unit
106+
107+
```bash
108+
sudo install -m 0644 \
109+
/opt/kater/current/scripts/systemd/kater-system.service.example \
110+
/etc/systemd/system/kater.service
111+
sudo systemctl daemon-reload
112+
sudo systemctl enable --now kater.service
113+
```
114+
115+
The unit:
116+
117+
- runs as `User=kater`
118+
- uses `EnvironmentFile=/etc/kater/kater.conf` for non-secrets
119+
- starts via `scripts/kater-with-chefvault.py serve ...` (fail-closed)
120+
- binds loopback only
121+
- hardens filesystem with `ProtectHome=true` and `ProtectSystem=strict`
122+
123+
## Acceptance checks (shadow)
124+
125+
```bash
126+
systemctl is-active kater.service
127+
curl --fail --silent http://127.0.0.1:9091/health/live
128+
curl --silent http://127.0.0.1:9091/health/ready
129+
journalctl -u kater.service -n 100 --no-pager | rg -i 'token|secret|ghp_|cfut_|cfat_' || true
130+
```
131+
132+
Negative tests (must fail closed / degrade predictably):
133+
134+
1. missing `/etc/kater/broker-token`
135+
2. broker unreachable
136+
3. missing fleet.json
137+
4. `bc-scan-2` SSH unreachable
138+
5. host reboot → service returns without laptop `joep`
139+
140+
## Cutover gate
141+
142+
Only after:
143+
144+
- no runtime path under `/home/joep`
145+
- secrets materialize via ChefVault, not a durable plaintext env of provider keys
146+
- health/live always up when process is up
147+
- health/ready reports component degradation clearly
148+
- soaktest on `bc-scan-arm` (memory/CPU/FDs/orphans) passes
149+
- laptop runtime kept as rollback for the observation window

‎docs/ops/chefvault.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,20 @@ Keep independent Vaultwarden items per provider (`primary`, `fallback`, `read-on
6262
not require editing Kater configuration: restart through the bootstrap and the fresh bundle
6363
is resolved before the gateway starts.
6464

65+
## Production systemd (bc-scan-arm)
66+
67+
For the always-on runtimehost, do **not** start Kater from a durable plaintext
68+
provider-key env file under a personal home directory.
69+
70+
Use:
71+
72+
- unit: `scripts/systemd/kater-system.service.example`
73+
- non-secret config: `scripts/systemd/kater.conf.example` → `/etc/kater/kater.conf`
74+
- broker token file: `/etc/kater/broker-token` (mode `0600`, owner `kater`)
75+
- bootstrap: `scripts/kater-with-chefvault.py` via `ExecStart`
76+
77+
Full host layout and cutover gates: `docs/ops/bc-scan-arm-runtime.md`.
78+
6579
## Failure behavior
6680

6781
The startup fails before Kater launches when the broker token is missing, the profile is not

‎scripts/systemd/kater-system.service.example‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
[Unit]
2+
Description=Kater MCP Gateway (system runtime)
3+
Documentation=file:///opt/kater/current/docs/ops/bc-scan-arm-runtime.md
4+
After=network-online.target
5+
Wants=network-online.target
6+
7+
[Service]
8+
Type=simple
9+
User=kater
10+
Group=kater
11+
WorkingDirectory=/opt/kater/current
12+
13+
# Non-secret host config only. Secrets come from kater-with-chefvault.py.
14+
EnvironmentFile=-/etc/kater/kater.conf
15+
Environment=KATER_HOST=127.0.0.1
16+
Environment=KATER_PUBLIC=0
17+
Environment=KATER_AUTH_MODE=none
18+
Environment=KATER_RATE_LIMIT=120
19+
Environment=CHEF_VAULT_BROKER_TOKEN_FILE=/etc/kater/broker-token
20+
Environment=CHEF_VAULT_RUNTIME_DIR=/run/kater/chefvault
21+
# uv must be on PATH for scripts/kater-with-chefvault.py (adjust if installed elsewhere).
22+
Environment=PATH=/usr/local/bin:/usr/bin:/bin
23+
24+
# Fail closed: materialize ChefVault secrets, then exec kater serve.
25+
# Missing broker token / profile / required item aborts before the gateway starts.
26+
# The explicit --profile wins over the wrapper's KATER_PROFILE, so chef-vault
27+
# must be listed here or the ChefVault MCP source stays gated off.
28+
ExecStart=/opt/kater/current/.venv/bin/python /opt/kater/current/scripts/kater-with-chefvault.py serve --profile utrecht,ops,cloud,research,code,web,chef-vault --host 127.0.0.1
29+
30+
Restart=on-failure
31+
RestartSec=5
32+
TimeoutStartSec=60
33+
TimeoutStopSec=30
34+
KillMode=control-group
35+
36+
# Hardening. Browser/MCP sidecars may require relaxing ProtectHome / PrivateUsers
37+
# after soak testing on the target host; start strict and open only what breaks.
38+
NoNewPrivileges=true
39+
PrivateTmp=true
40+
ProtectSystem=strict
41+
ProtectHome=true
42+
ProtectKernelTunables=true
43+
ProtectKernelModules=true
44+
ProtectControlGroups=true
45+
RestrictSUIDSGID=true
46+
LockPersonality=true
47+
UMask=0077
48+
CapabilityBoundingSet=
49+
AmbientCapabilities=
50+
# /opt/kater/current/.kater must exist before the unit starts: systemd sets up the
51+
# ProtectSystem=strict namespace before ExecStart, and the wrapper only creates the
52+
# directory once it is already running. The release install step provisions it as
53+
# kater:kater 0700 (see docs/ops/bc-scan-arm-runtime.md).
54+
ReadWritePaths=/var/lib/kater /var/cache/kater /run/kater /opt/kater/current/.kater
55+
RuntimeDirectory=kater
56+
RuntimeDirectoryMode=0700
57+
StateDirectory=kater
58+
CacheDirectory=kater
59+
60+
# Logging stays in journald; do not echo secrets.
61+
StandardOutput=journal
62+
StandardError=journal
63+
SyslogIdentifier=kater
64+
65+
[Install]
66+
WantedBy=multi-user.target

‎scripts/systemd/kater.conf.example‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Non-secret Kater runtime configuration for /etc/kater/kater.conf
2+
# Install with: install -m 0640 -o root -g kater kater.conf.example /etc/kater/kater.conf
3+
#
4+
# Secrets MUST NOT live in this file. Use ChefVault bootstrap:
5+
# ExecStart=/opt/kater/current/scripts/kater-with-chefvault.py serve ...
6+
# and a broker token at /etc/kater/broker-token (mode 0600, owner kater).
7+
8+
KATER_HOST=127.0.0.1
9+
KATER_PUBLIC=0
10+
KATER_AUTH_MODE=none
11+
KATER_RATE_LIMIT=120
12+
KATER_PROXY=1
13+
KATER_PROFILE=utrecht,ops,cloud,research,code,web,chef-vault
14+
15+
# Canonical PR/repo identity (not a local checkout path).
16+
KATER_PR_REPO=<org>/<data-plane-repo>
17+
18+
# Fleet inventory: host-local cache under /var/cache/kater (refreshed separately).
19+
UTRECHT_FLEET_INVENTORY_PATH=/var/cache/kater/utrecht-fleet/inventory/fleet.json
20+
21+
# Remote CI health plane (bc-scan-2 remains execution SSOT).
22+
UTRECHT_CI_HEALTH_SSH_TARGET=ubuntu@bc-scan-2
23+
UTRECHT_CI_HEALTH_REMOTE_REPO=/home/ubuntu/<data-plane-repo>
24+
25+
# Never point UTRECHT_MCP_URL at this Kater instance (self-loop).
26+
# UTRECHT_MCP_URL=
27+
28+
# Optional local data-plane checkout for CLI tools only. Leave unset on runtime hosts.
29+
# UTRECHT_REPO_PATH=
30+
31+
# ChefVault broker (secrets come from Vaultwarden via chefvault-profile).
32+
CHEF_VAULT_BROKER_URL=http://127.0.0.1:8322
33+
CHEF_VAULT_BROKER_TOKEN_FILE=/etc/kater/broker-token
34+
CHEF_VAULT_RUNTIME_DIR=/run/kater/chefvault
35+
CHEF_VAULT_PROFILE_COMMAND=chefvault-profile
36+
37+
# Extensions: use ChefVault when bootstrapped; Utrecht overlay is separate package.
38+
# KATER_EXTENSIONS_MODULE is set by kater-with-chefvault.py to kater.chefvault_extension.
39+
# For Utrecht tools on top, load a combined extensions module in the release image.

0 commit comments

Comments
 (0)