refactor(brain-server): organize src/ into thematic sub-modules

Eliminates flat src/ root files in favour of grouped sub-modules,
modelled after recall-server's app/bootstrap/http layout. Every
former root file now lives under a dedicated folder:

  src/
  ├── main.rs                  binary entry
  ├── admin/mod.rs             was admin.rs       (admin HTTP server)
  ├── bootstrap/
  │   ├── mod.rs               new
  │   ├── shutdown.rs          was shutdown.rs    (signal + drain)
  │   └── tls.rs               was tls.rs         (rustls provider)
  ├── config/mod.rs            was config.rs      (TOML schema)
  ├── llm/                     unchanged          (summarizer adapters)
  ├── network/
  │   ├── mod.rs               new
  │   ├── connection.rs        was connection.rs  (Tokio listener)
  │   ├── dispatch.rs          was dispatch.rs    (frame FSM)
  │   ├── routing.rs           was routing.rs     (RoutingTable)
  │   └── subscribe.rs         was subscribe.rs   (SUBSCRIBE bridge)
  └── shard/
      ├── mod.rs               was shard.rs       (per-shard runtime)
      └── adapters.rs          was shard_adapters.rs (worker glue)

Why this layout

- main.rs declares the five top-level modules; each is a folder.
  No source file sits directly in src/ except main.rs.
- main.rs adds `use bootstrap::{shutdown, tls};` and
  `use network::{connection, dispatch, routing, subscribe};`
  at crate-root scope so every internal `crate::tls::*`,
  `crate::connection::*` reference keeps working without churning
  ~30 sites.
- Tests use `#[path = "../src/<group>/<file>.rs"] mod <name>;`
  to load source files at their own top level. The crate-root
  aliases above match the names tests use, so the same source
  file compiles cleanly in both contexts.
- `shard/mod.rs` declares `pub mod adapters;`; the old
  `mod shard_adapters;` in tests was removed (clippy::duplicate_mod
  rejected loading the same file twice through both paths).

Compared with recall-server's pattern (see
`.claude/plans/recall-server-comparison.md`), brain-server now
matches the "every concern in a folder" property, while keeping
the binary-protocol-specific structure (single connection
listener, frame dispatcher, per-shard Glommio executor) that
brain's spec mandates.

Pure relocation + import scoping. No semantic edits. The
findings doc lists what else recall-server has that we
intentionally don't copy (Tower middleware, routes, services —
those are HTTP/REST-shaped and don't translate to our binary
protocol).

docker-verify green.

Author: nirajgeorgian

.claude/plans/recall-server-comparison.md

@@ -0,0 +1,214 @@
+# recall-server organization — what to borrow, what to skip
+
+`arc-labs/recall/crates/recall-server` is a mature axum-based REST
+server (163 source files). It demonstrates a clean way to organize
+a Rust server crate. Below: which of its ideas apply to brain-server,
+which don't, and why.
+
+---
+
+## recall-server's organizing skeleton
+
+```
+src/
+├── main.rs              32 LOC — runtime + bootstrap::run() only
+├── lib.rs                    — declares the modules
+├── app/                      — Application + Router + AppState
+│   ├── application.rs        Application::{build, run}
+│   ├── router.rs             build_router() with Tower middleware stack
+│   ├── state.rs              AppState<S> generic over storage backend
+│   └── mod.rs
+├── bootstrap/                — startup orchestration
+│   ├── mod.rs::run()         single public entry-point
+│   ├── observability.rs      tracing subscriber init
+│   ├── otel.rs               OTLP exporter
+│   ├── pools.rs              storage pool construction
+│   ├── providers.rs          LLM / embedder selection
+│   └── worker.rs             background worker spawn
+├── config/                   — env-var-loaded config, split by section
+│   ├── deployment.rs
+│   ├── log.rs
+│   ├── providers.rs
+│   ├── secrets.rs
+│   ├── server.rs
+│   └── mod.rs
+├── http/                     — HTTP-layer infrastructure
+│   ├── constants/            api / namespace / plan
+│   ├── cursor.rs             opaque pagination encoding
+│   ├── error/                ApiError, codes, impls
+│   ├── extractors/           pagination, scope_filter, time_range, …
+│   ├── middleware/           auth, idempotency, rate_limit, security_headers
+│   ├── response.rs           ApiJson, PaginatedApiJson envelopes
+│   └── span.rs               handler_event! macros
+├── routes/                   — one sub-dir per resource group
+│   ├── memories/   …
+│   ├── entities/   …
+│   ├── pipeline/   …
+│   └── router.rs             merge all sub-routers
+├── domain/                   — JSON request/response body types
+├── services/                 — multi-step orchestration
+├── identity/                 — auth + tenancy
+├── messaging/                — control-plane event bus, write-job SSE
+├── email/                    — sender + templates
+└── crypto.rs                 — AES-256-GCM helpers
+```
+
+The headline pattern: **main.rs is 32 lines.** Every concern has a
+home; the entry point doesn't reach for any of them directly.
+
+---
+
+## What translates to brain-server
+
+Brain-server is a binary-protocol server (Glommio + Tokio bridge), not
+REST. Most of recall's layers don't map. But two big wins do:
+
+### 1. **`main.rs` → `bootstrap/` + thin main** (high value)
+
+brain-server's `main.rs` is **488 LOC**. It performs:
+
+1. CLI arg parsing.
+2. Tracing subscriber init (fmt + optional JSON, optional OTel).
+3. Config load + validation.
+4. Summarizer factory wiring.
+5. Tokio runtime construction.
+6. Inside the runtime: shutdown signal channel, signal listener spawn,
+   TLS build, ConnectionMetrics construction, AdminServer construction
+   + bind + spawn, ConnectionListener construction + bind + spawn,
+   serve() await, admin drain timeout, shard graceful shutdown.
+
+That's 5+ orthogonal concerns. recall's pattern splits this into:
+
+```
+bootstrap/
+├── mod.rs              run() — orchestration only
+├── tracing.rs          init_tracing(LogConfig)
+├── tls.rs              build_tls(TlsConfig) — currently in tls.rs root
+├── shards.rs           spawn_shards(cfg, summarizer) + ShardSpawn helpers
+├── admin.rs            build_admin(cfg, state) → AdminServer (binds inside)
+├── listener.rs         build_listener(cfg, tls, topology, metrics) → ConnectionListener
+├── shutdown.rs         signal handler + graceful drain — currently in shutdown.rs
+└── summarizer.rs       summarizer factory — currently inside main.rs
+```
+
+After: `main.rs` would be ~40 LOC (CLI parse + Tokio runtime build +
+`bootstrap::run(cfg).await`). The existing `tls.rs`, `shutdown.rs`,
+`llm/` would slot under bootstrap/ naturally.
+
+**Risk**: low. Pure relocation. The runtime model and shard
+discipline don't change.
+
+### 2. **`config.rs` → `config/`** (high value)
+
+brain-server's `config.rs` is **578 LOC** of `[server]`, `[storage]`,
+`[shard]`, `[hnsw]`, `[embedder]`, `[workers]`, `[logging]`,
+`[tracing]`, `[auth]`, `[summarizer]` sections plus parsers
+(`parse_human_bytes`, `coerce_leaf`, `set_path`, `deserialize_*`).
+
+Recall's `config/` mirrors its env-var sections one-per-file. Brain
+can mirror its TOML sections:
+
+```
+config/
+├── mod.rs              Config struct + load + validation
+├── server.rs           ServerConfig + TlsConfig
+├── storage.rs          StorageConfig + ShardConfig
+├── index.rs            HnswConfig + EmbedderConfig
+├── workers.rs          WorkersConfig
+├── observability.rs    LoggingConfig + TracingConfig
+├── auth.rs             AuthConfig
+├── summarizer.rs       SummarizerConfig (currently inline)
+└── parse.rs            parse_human_bytes, coerce_leaf, set_path
+```
+
+Each section file: ~50–80 LOC. mod.rs: ~100 LOC for the top-level
+`Config` + `load_from_file` + `apply_overrides`.
+
+**Risk**: low. Section structs are independent; only `parse.rs`
+helpers cross-cut.
+
+### 3. **`admin.rs` (463 LOC) — partial: keep monolithic but lift constants** (low value)
+
+recall's `http/constants/`, `http/error/`, `http/response.rs` are
+beautiful in an HTTP REST context. brain-server's admin.rs is a
+hand-rolled HTTP/1.1 implementation intentionally kept tight (no
+hyper, no Tower). The constants we *do* have (path strings, status
+codes, JSON body shapes) could live in
+`admin/{routes,response,handlers}.rs`, but the win is small.
+
+**Recommendation: skip**. The cost of admin/ split outweighs the
+~30% navigability gain. Revisit if admin grows past 1k LOC.
+
+---
+
+## What does NOT translate
+
+| recall pattern | Why it doesn't fit brain |
+|---|---|
+| `app/router.rs` Tower middleware stack | Brain's connection-listener has a single FSM dispatch (HELLO → AUTH → Established → frame-dispatch). The "stack" is opcodes + idempotency lookup + auth-phase guard, all already in `dispatch.rs`. Wrapping in a Tower-like layer would force boxed-Service ceremony for zero behaviour gain. |
+| `routes/{memories,entities,pipeline,…}` | Brain has 30 wire opcodes dispatched in `dispatch.rs`. They're not "routes" — they're stream operations with shard-affinity. Splitting `dispatch.rs` further would over-fragment the FSM. |
+| `services/` | Brain's "service layer" is the brain-ops crate. Don't duplicate. |
+| `domain/{request,response}` JSON types | Brain has the brain-protocol crate (rkyv-encoded binary). The recently-shipped refactor D already gave it `requests/` + `responses/` sub-modules. |
+| `http/middleware/{auth,rate_limit,idempotency,security_headers}` | Brain's auth is per-connection FSM, idempotency lives in brain-ops, rate-limiting is a connection-limit (spec §03/06). No mid-stream middleware needed. |
+| `http/extractors/` | Brain wire frames are rkyv-decoded; there are no query strings to extract. |
+| `identity/`, `tenancy/`, `email/`, `crypto/` | Out of spec scope for v1 (auth is `AuthMethod::None` per spec §03/06 §3.1). |
+| `messaging/` (in-process event bus + write-job SSE) | Brain has `brain-ops/subscribe` (SUBSCRIBE bridge). Don't duplicate. |
+| OTLP / `otel.rs` | Brain has it in cargo deps (`opentelemetry`) but no exporter yet — that's a Phase 14 observability task. |
+
+---
+
+## Spec considerations
+
+Any refactor must preserve:
+
+- **Single-writer-per-shard** discipline (CLAUDE.md §5 inv. 2).
+- **Glommio executor per shard** model (CLAUDE.md §9: no Tokio inside a shard).
+- **ConnectionListener** structure (spec §10).
+- **Hand-rolled HTTP/1.1 for admin** (decided in 9.13 — no hyper).
+- **Tokio↔Glommio boundary** at `dispatch.rs` (sub-task 9.10).
+
+The two recommended refactors (bootstrap/, config/) touch only
+main.rs, config.rs, and the standalone helper files. They don't
+cross any of the above lines.
+
+---
+
+## Recommended sequence (if you choose to act)
+
+| # | Refactor | LOC moved | Files added | Risk |
+|---|---|---|---|---|
+| 1 | `main.rs` → `bootstrap/` + thin main | ~450 | ~6 | Low |
+| 2 | `config.rs` → `config/` | ~580 | ~8 | Low |
+| 3 | (skip) `admin.rs` split | — | — | — |
+
+Both refactors are pure relocation with import-scoping. ~2 commits,
+~30 min each, docker-verify between.
+
+After: brain-server's `src/` becomes:
+
+```
+src/
+├── main.rs               ~40 LOC
+├── lib.rs                module declarations
+├── bootstrap/            6 files — startup orchestration
+├── config/               8 files — config sections + parsers
+├── shard.rs              ~975 LOC (core shard runtime)
+├── shard_adapters.rs     ~735 LOC (worker adapter glue)
+├── connection.rs         ~780 LOC (Tokio listener)
+├── dispatch.rs           ~700 LOC (frame FSM + boundary)
+├── subscribe.rs          ~430 LOC (SUBSCRIBE bridge)
+├── admin.rs              ~460 LOC (admin HTTP/1.1)
+├── routing.rs            ~280 LOC (RoutingTable)
+├── llm/                  6 files (summarizer adapters, already split)
+└── tests/                in-process E2E fixtures
+```
+
+The "infra" (bootstrap, config) is grouped; the per-feature concerns
+(shard, connection, dispatch, subscribe, admin, routing) stay at the
+root as cohesive units — which matches recall's pattern where
+`app/`, `bootstrap/`, `http/`, `routes/` are dirs but
+single-concern feature files stay at the root.
+
+---
+
+*Awaiting direction on whether to execute refactor 1, 2, or both.*

crates/brain-server/src/admin.rs crates/brain-server/src/admin/mod.rscrates/brain-server/src/admin.rs renamed to crates/brain-server/src/admin/mod.rs

@@ -0,0 +1,8 @@
+//! Startup orchestration: TLS provider build, signal handling, and
+//! graceful shard drain. The pieces here are wired together by
+//! `main::run`.
+
+#[cfg(target_os = "linux")]
+pub mod shutdown;
+#[cfg(target_os = "linux")]
+pub mod tls;

crates/brain-server/src/bootstrap/mod.rs

@@ -11,26 +11,29 @@
 
 #[cfg(target_os = "linux")]
 mod admin;
+mod bootstrap;
 mod config;
 #[cfg(target_os = "linux")]
-mod connection;
-#[cfg(target_os = "linux")]
-mod dispatch;
-#[cfg(target_os = "linux")]
 mod llm;
-mod routing;
+#[cfg(target_os = "linux")]
+mod network;
 #[cfg(target_os = "linux")]
 #[allow(dead_code)] // consumed by the connection layer in sub-task 9.10.
 mod shard;
+
+// Crate-root aliases. The folder reorg moved each module into a
+// thematic sub-module (`bootstrap::tls`, `network::connection`, …),
+// but every file references its peers by the historical top-level
+// name (`crate::tls`, `crate::connection`, …). Re-exporting at the
+// crate root preserves those paths and matches the way integration
+// tests load source files via `#[path]` + `mod xxx;`.
 #[cfg(target_os = "linux")]
-#[allow(dead_code)] // adapters wired into the per-shard scheduler in 9.8.
-mod shard_adapters;
-#[cfg(target_os = "linux")]
-mod shutdown;
+use bootstrap::{shutdown, tls};
 #[cfg(target_os = "linux")]
-mod subscribe;
+use network::{connection, dispatch, routing, subscribe};
 #[cfg(target_os = "linux")]
-mod tls;
+#[allow(unused_imports)] // re-export kept for symmetry; binary doesn't reach it directly
+use shard::adapters as shard_adapters;
 
 use std::env;
 use std::path::PathBuf;

crates/brain-server/src/shutdown.rs …s/brain-server/src/bootstrap/shutdown.rscrates/brain-server/src/shutdown.rs renamed to crates/brain-server/src/bootstrap/shutdown.rs crates/brain-server/src/shutdown.rs renamed to crates/brain-server/src/bootstrap/shutdown.rs

@@ -0,0 +1,13 @@
+//! Tokio connection layer: the per-listener accept loop
+//! (`connection`), the Tokio↔Glommio frame dispatcher (`dispatch`),
+//! the agent→shard routing table (`routing`), and the SUBSCRIBE bridge
+//! (`subscribe`).
+
+#[cfg(target_os = "linux")]
+pub mod connection;
+#[cfg(target_os = "linux")]
+pub mod dispatch;
+#[cfg(target_os = "linux")]
+pub mod routing;
+#[cfg(target_os = "linux")]
+pub mod subscribe;