Skip to content

Commit 05560e7

Browse files
committed
docs(admin): fix audit log shapes and split login 401 vs 403
Two findings from the latest claude review on #674: 1. **Audit log shapes (Bug A)**: the previous example claimed `admin_audit ... operation=delete_table table=orders` was a leader-direct sample, but `DynamoHandler.handleDelete` (dynamo_handler.go:379-396) does not emit any handler-level admin_audit — that shape only appears via `ForwardServer` on the forwarded path. It also omitted the `Audit` middleware line (middleware.go:206-232) which wraps every non-GET admin request with method= / path= / status= / remote= / duration=. An operator building a parser against the doc would have missed half of the actual lines. Rewrote the section as three explicit shapes — Audit middleware (always), S3Handler op line (only emits from s3_handler.go:299 /333/355), ForwardServer op line (covers both Dynamo and S3 forwarded paths). Removed the misleading `delete_table` leader-direct sample and replaced the Dynamo examples with forwarded-only ones, which is where they actually originate. Added a one-sentence note that a single request typically produces two lines (op-specific + middleware) so log parsers know to treat the key as a union. 2. **Login 401 vs 403 (Codex P2)**: the troubleshooting entry said "both produce 401" but auth_handler.go:336-345 returns 403 forbidden when the credentials match but the access key is not in `-adminFullAccessKeys` / `-adminReadOnlyAccessKeys`. Split into two entries: 401 invalid_credentials (bad access_key or secret) and 403 forbidden (valid SigV4 creds, but no admin role assignment), with the remediation specific to each case. No functional changes. Doc-only.
1 parent b9b0b0c commit 05560e7

1 file changed

Lines changed: 50 additions & 19 deletions

File tree

docs/admin.md

Lines changed: 50 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -174,32 +174,53 @@ should look at Raft leader-churn logs first.
174174

175175
## Audit log
176176

177-
Every state-changing admin request emits a structured slog line at
178-
`INFO` level on the leader's stdout (or wherever the process slog
179-
handler is wired). Two shapes:
177+
Every state-changing admin request emits structured slog lines at
178+
`INFO` level under the `admin_audit` key on the leader's stdout (or
179+
wherever the process slog handler is wired). A single state-changing
180+
request typically produces **two** audit lines: one operation-specific
181+
line from the source that performed the mutation, plus one generic
182+
HTTP-shaped line from the `Audit` middleware. The shapes differ by
183+
source — log parsers should treat the `admin_audit` key as a union
184+
and dispatch on the fields present.
185+
186+
**`Audit` middleware** — emitted for every non-GET/HEAD/OPTIONS
187+
request that reaches the admin mux on this node, regardless of which
188+
handler served it. Always present on the node that received the HTTP
189+
request (which may be a follower if the request was then forwarded):
180190

181-
**Leader-direct write** (`DynamoHandler` / `S3Handler` after a
182-
successful mutation):
191+
```
192+
admin_audit actor=AKIA_ADMIN role=full method=POST path=/admin/api/v1/buckets status=201 remote=10.0.0.7:51234 duration=8.2ms
193+
```
194+
195+
**`S3Handler` operation line** — emitted on the leader after a
196+
successful bucket mutation. Only the S3 admin path emits these; the
197+
DynamoDB admin path relies on the middleware line plus the forwarded
198+
line below for its audit trail:
183199

184200
```
185201
admin_audit actor=AKIA_ADMIN role=full operation=create_bucket bucket=my-bucket
186-
admin_audit actor=AKIA_ADMIN role=full operation=delete_table table=orders
202+
admin_audit actor=AKIA_ADMIN role=full operation=put_bucket_acl bucket=my-bucket acl=public-read
203+
admin_audit actor=AKIA_ADMIN role=full operation=delete_bucket bucket=my-bucket
187204
```
188205

189-
**Forwarded write** (logged by the leader's `ForwardServer` after a
190-
follower handed the request off):
206+
**`ForwardServer` operation line** — emitted on the leader when a
207+
follower forwarded the request via `AdminForward`. Carries the
208+
originating follower's node ID in `forwarded_from`. Covers both
209+
DynamoDB and S3 admin operations:
191210

192211
```
212+
admin_audit actor=AKIA_ADMIN role=full forwarded_from=n2 operation=create_table table=orders
213+
admin_audit actor=AKIA_ADMIN role=full forwarded_from=n2 operation=delete_table table=orders
193214
admin_audit actor=AKIA_ADMIN role=full forwarded_from=n2 operation=put_bucket_acl bucket=my-bucket acl=public-read
194215
```
195216

196-
`forwarded_from` is the originating follower's node ID. CR and LF
197-
in that field are stripped at the entry point — a hostile follower
198-
cannot split a single audit line into two by smuggling control
199-
characters into its node ID.
217+
CR and LF in `forwarded_from` are stripped at the entry point — a
218+
hostile follower cannot split a single audit line into two by
219+
smuggling control characters into its node ID.
200220

201-
Login and logout emit their own audit lines (`action=login` /
202-
`action=logout`) so the JWT's lifetime can be correlated with the
221+
Login and logout emit their own `admin_audit` lines with
222+
`action=login` / `action=logout` (plus `actor`, `claimed_actor`,
223+
`remote`, `status`) so the JWT's lifetime can be correlated with the
203224
mutations it authorised.
204225

205226
## Troubleshooting
@@ -220,11 +241,21 @@ read the TLS section above before doing so).
220241

221242
### Login returns 401 invalid_credentials
222243

223-
The access key + secret pair did not match the credentials file, or
224-
the key is not listed in `-adminFullAccessKeys` /
225-
`-adminReadOnlyAccessKeys`. The dashboard does not distinguish the
226-
two cases on the wire — both produce 401 — but the leader's audit
227-
log shows the precise reason.
244+
The access key + secret pair did not match an entry in
245+
`-s3CredentialsFile`. Either the access key is unknown or the secret
246+
is wrong. Verify the credentials file is the one the running process
247+
loaded (it is read once at startup) and that the secret matches
248+
exactly — secrets are compared with `subtle.ConstantTimeCompare`, so
249+
trailing whitespace counts.
250+
251+
### Login returns 403 forbidden
252+
253+
The credentials matched, but the access key is not listed in either
254+
`-adminFullAccessKeys` or `-adminReadOnlyAccessKeys`. This is a
255+
distinct case from the 401 above: the operator has valid SigV4
256+
credentials for the data plane but no admin role assignment. Add the
257+
key to one of the role flags and **restart every node** so each
258+
node's live role index picks up the change.
228259

229260
### Write returns 403 forbidden
230261

0 commit comments

Comments
 (0)