|
| 1 | +# Plan: Project-Local and Server ADF from `.terraphim` |
| 2 | + |
| 3 | +Date: 2026-05-20 10:01 BST |
| 4 | +Status: planning |
| 5 | +Scope: design and implementation plan only; no runtime changes in this document. |
| 6 | + |
| 7 | +## Goal |
| 8 | + |
| 9 | +Create a new ADF mode where a repository can define its own agent fleet under `.terraphim/`, and the same definition can run either: |
| 10 | + |
| 11 | +- locally, for developer-triggered project agents; or |
| 12 | +- on the ADF server, as part of the long-running multi-project fleet. |
| 13 | + |
| 14 | +The core design principle is to extend the existing project-discovery and orchestrator configuration paths rather than create a second ADF configuration system. |
| 15 | + |
| 16 | +## Current Evidence |
| 17 | + |
| 18 | +### Gitea Issue Signals |
| 19 | + |
| 20 | +Recent ADF-related issue and PR signals show the main pressure points: |
| 21 | + |
| 22 | +- `#1674` asks for project-level `.terraphim/` config discovery and is the direct prerequisite for project-local ADF. |
| 23 | +- `#1714`, `#1715`, `#1761`, and `#1760` show PR gate fragility around `adf/build`, `adf/pr-reviewer`, status posting, and auto-merge confidence. |
| 24 | +- `#1754` through `#1762` are repeated `[ADF] PR gate remediation` issues, indicating the server ADF needs better dedupe and per-project gate semantics. |
| 25 | +- `#1709` and PR `#1491` point to sandbox/executor hardening needed for agents that run code locally or on the server. |
| 26 | +- PR `#1739` adds worktree ownership manifest protection, which should be a required part of any local/server worktree lifecycle. |
| 27 | +- PR `#1514` adds strict ADF permission checks; project-local config must support the same validation path. |
| 28 | +- `#1675`, `#1718`, `#1664`, and `#1655` show recurring secret-redaction risks in config, webhook, RLM, and ADF output paths. |
| 29 | + |
| 30 | +### Codebase Signals |
| 31 | + |
| 32 | +- `terraphim_config::project::discover()` already walks upward to find `.terraphim/` and returns its canonical path. |
| 33 | +- `ProjectConfig` currently supports only `global_shortcut` and role overrides loaded from `.terraphim/config.json`. |
| 34 | +- `OrchestratorConfig` already supports multi-project ADF via `[[projects]]`, per-project `working_dir`, per-project Gitea settings, include fragments, per-project PR dispatch, provider budgets, learning, evolution, and validation. |
| 35 | +- `adf --check CONFIG` currently validates and prints the routing table for an explicit TOML config path. |
| 36 | +- Orchestrator runtime already injects `ADF_PROJECT_ID`, `ADF_WORKING_DIR`, `GITEA_OWNER`, and `GITEA_REPO` into agent spawn contexts. |
| 37 | + |
| 38 | +## Proposed Concept |
| 39 | + |
| 40 | +Introduce **Project ADF**: a `.terraphim/adf.toml` file that is valid both as a developer-local ADF entrypoint and as a server-side project fragment. |
| 41 | + |
| 42 | +The file is not a replacement for `OrchestratorConfig`. It is a project-scoped layer that compiles into the existing orchestrator model. |
| 43 | + |
| 44 | +## Proposed `.terraphim` Layout |
| 45 | + |
| 46 | +```text |
| 47 | +.terraphim/ |
| 48 | +├── config.json # existing Terraphim role/search config |
| 49 | +├── adf.toml # project-local ADF declaration |
| 50 | +├── agents/ # optional agent prompt/task templates |
| 51 | +├── skills/ # optional project-local skills |
| 52 | +├── personas/ # optional project-local personas |
| 53 | +└── adf.local.toml # ignored/local overrides, optional |
| 54 | +``` |
| 55 | + |
| 56 | +`adf.local.toml` should be ignored by git and used only for developer-local paths, tokens, and machine-specific limits. Secrets must still be supplied through environment variables or `op` injection, not stored directly. |
| 57 | + |
| 58 | +## Proposed `adf.toml` Shape |
| 59 | + |
| 60 | +```toml |
| 61 | +[adf] |
| 62 | +id = "terraphim-ai" |
| 63 | +mode = "hybrid" # local | server | hybrid |
| 64 | +working_dir = "." # resolved relative to repository root |
| 65 | +skill_data_dir = ".terraphim/skills" |
| 66 | +persona_data_dir = ".terraphim/personas" |
| 67 | + |
| 68 | +[adf.gitea] |
| 69 | +base_url = "https://git.terraphim.cloud" |
| 70 | +owner = "terraphim" |
| 71 | +repo = "terraphim-ai" |
| 72 | +token = "${GITEA_TOKEN}" |
| 73 | + |
| 74 | +[adf.local] |
| 75 | +enabled = true |
| 76 | +max_concurrent_agents = 2 |
| 77 | +worktree_root = ".worktrees/adf-local" |
| 78 | +allowed_agents = ["build-runner", "pr-reviewer", "security-sentinel"] |
| 79 | + |
| 80 | +[adf.server] |
| 81 | +enabled = true |
| 82 | +max_concurrent_agents = 4 |
| 83 | +schedule_offset_minutes = 0 |
| 84 | +post_status_context_prefix = "adf" |
| 85 | + |
| 86 | +[[adf.agents]] |
| 87 | +name = "build-runner" |
| 88 | +layer = "Safety" |
| 89 | +model = "sonnet" |
| 90 | +run_policy = "on-pr-open" |
| 91 | +local_command = "cargo test --workspace" |
| 92 | +server_command = "cargo test --workspace" |
| 93 | + |
| 94 | +[[adf.agents]] |
| 95 | +name = "pr-reviewer" |
| 96 | +layer = "Review" |
| 97 | +model = "zai-coding-plan/glm-5.1" |
| 98 | +run_policy = "on-pr-open" |
| 99 | + |
| 100 | +[adf.pr_dispatch] |
| 101 | +agents_on_pr_open = [ |
| 102 | + { name = "build-runner", required = true }, |
| 103 | + { name = "pr-reviewer", required = true }, |
| 104 | +] |
| 105 | +``` |
| 106 | + |
| 107 | +This shape is intentionally project-centred and smaller than full `OrchestratorConfig`, but every field should map deterministically into the existing orchestrator TOML structures. |
| 108 | + |
| 109 | +## Configuration Resolution |
| 110 | + |
| 111 | +Resolution order should be explicit: |
| 112 | + |
| 113 | +1. CLI `--config PATH`, if provided. |
| 114 | +2. `.terraphim/adf.toml`, discovered upward from the current directory. |
| 115 | +3. Existing `/opt/ai-dark-factory/orchestrator.toml` server default. |
| 116 | + |
| 117 | +For project-local mode, `adf` should accept: |
| 118 | + |
| 119 | +```text |
| 120 | +adf --local |
| 121 | +adf --local --check |
| 122 | +adf --local --agent build-runner |
| 123 | +adf --local --issue 1234 |
| 124 | +adf --local --pr 42 |
| 125 | +``` |
| 126 | + |
| 127 | +For server mode, existing `adf CONFIG` continues to work, but the server can include project fragments generated from each repository's `.terraphim/adf.toml`. |
| 128 | + |
| 129 | +## Compilation Model |
| 130 | + |
| 131 | +Add a small conversion layer: |
| 132 | + |
| 133 | +```text |
| 134 | +.terraphim/adf.toml |
| 135 | + -> ProjectAdfConfig |
| 136 | + -> OrchestratorConfig / IncludeFragment-compatible structs |
| 137 | + -> existing validate() |
| 138 | + -> existing AgentOrchestrator |
| 139 | +``` |
| 140 | + |
| 141 | +The compiler should: |
| 142 | + |
| 143 | +- canonicalise the project root via `terraphim_config::project::discover()`; |
| 144 | +- set `Project.id` from `[adf].id`; |
| 145 | +- set `Project.working_dir` to the repository root unless overridden; |
| 146 | +- map `[adf.gitea]` to `GiteaOutputConfig`; |
| 147 | +- map `[[adf.agents]]` to `AgentDefinition` with `project = adf.id`; |
| 148 | +- map `[adf.pr_dispatch]` to `pr_dispatch_per_project[adf.id]`; |
| 149 | +- resolve `.terraphim/skills` and `.terraphim/personas` relative to project root; |
| 150 | +- preserve the same provider allow-list checks already used by `OrchestratorConfig::validate()`. |
| 151 | + |
| 152 | +## Local Runtime Semantics |
| 153 | + |
| 154 | +Local ADF should be a bounded, foreground-first runner: |
| 155 | + |
| 156 | +- runs from the project repository root; |
| 157 | +- uses a local worktree root under `.worktrees/adf-local` by default; |
| 158 | +- limits concurrency aggressively, default `1` or `2`; |
| 159 | +- does not auto-merge; |
| 160 | +- may post Gitea comments/statuses only when `GITEA_TOKEN` is present; |
| 161 | +- prints a routing table and resolved config on `--check`; |
| 162 | +- fails closed on invalid config, banned providers, unsafe paths, and world-readable sensitive local overrides. |
| 163 | + |
| 164 | +This supports developer workflows such as: |
| 165 | + |
| 166 | +```text |
| 167 | +adf --local --agent build-runner |
| 168 | +adf --local --pr 42 |
| 169 | +adf --local --issue 1714 |
| 170 | +``` |
| 171 | + |
| 172 | +## Server Runtime Semantics |
| 173 | + |
| 174 | +Server ADF should continue using the existing long-running orchestrator, but with one additional source of project fragments: |
| 175 | + |
| 176 | +- server scans configured repositories for `.terraphim/adf.toml`; |
| 177 | +- server converts each file into project/agent fragments; |
| 178 | +- server validates the merged fleet once before startup; |
| 179 | +- server owns PR-gate status posting and auto-merge decisions; |
| 180 | +- server applies pause flags and project circuit breakers per project id; |
| 181 | +- server never reads developer-local `adf.local.toml`. |
| 182 | + |
| 183 | +## Security Requirements |
| 184 | + |
| 185 | +- No secrets in `.terraphim/adf.toml`; use `${VAR}` placeholders. |
| 186 | +- Warn or fail on group/world-readable `.terraphim/adf.local.toml` and any loaded token files. |
| 187 | +- Redact webhook URLs, API keys, Gitea tokens, RLM keys, and provider credentials from all debug output. |
| 188 | +- Restrict local command execution to the project root or managed worktrees. |
| 189 | +- Reject path traversal in project-relative paths. |
| 190 | +- Reuse the provider allow-list in `terraphim_orchestrator::config`. |
| 191 | +- Treat local mode as non-privileged; only the server systemd lifecycle may run privileged cleanup. |
| 192 | + |
| 193 | +## Implementation Phases |
| 194 | + |
| 195 | +### Phase 1: Research and Schema |
| 196 | + |
| 197 | +- Define `ProjectAdfConfig` in `terraphim_orchestrator::project_adf` or `terraphim_config::project` if shared outside ADF. |
| 198 | +- Document the `.terraphim/adf.toml` schema. |
| 199 | +- Add JSON/TOML examples for local-only, server-only, and hybrid projects. |
| 200 | +- Decide whether `adf.local.toml` is merged by the CLI only or by a dedicated `--local-overrides` flag. |
| 201 | + |
| 202 | +### Phase 2: Loader and Validation |
| 203 | + |
| 204 | +- Add `ProjectAdfConfig::discover_and_load(start_dir)` using existing `.terraphim` discovery. |
| 205 | +- Add conversion into an `OrchestratorConfig` or include-fragment-compatible type. |
| 206 | +- Reuse `OrchestratorConfig::validate()` for duplicate project ids, banned providers, mixed mode, and project references. |
| 207 | +- Extend `adf --check` so `adf --local --check` prints the same routing table plus resolved project root. |
| 208 | + |
| 209 | +### Phase 3: Local Runner |
| 210 | + |
| 211 | +- Add CLI parsing for `--local`, `--agent`, `--issue`, and `--pr`. |
| 212 | +- Build a single-project `AgentOrchestrator` from `.terraphim/adf.toml`. |
| 213 | +- Add a foreground execution path for one agent without starting the full reconcile loop. |
| 214 | +- Ensure local mode does not auto-merge or post statuses unless explicitly enabled. |
| 215 | + |
| 216 | +### Phase 4: Server Import |
| 217 | + |
| 218 | +- Add a server-side scanner that reads `.terraphim/adf.toml` from configured repositories. |
| 219 | +- Convert each project file into include fragments. |
| 220 | +- Preserve current `include = ["conf.d/*.toml"]` behaviour for existing deployments. |
| 221 | +- Add clear conflict rules when both server TOML and project `.terraphim/adf.toml` define the same project id. |
| 222 | + |
| 223 | +### Phase 5: PR Gate and Status Contract |
| 224 | + |
| 225 | +- Standardise status contexts: `adf/build`, `adf/pr-reviewer`, `adf/security`, and project-qualified variants when needed. |
| 226 | +- Require non-empty agent output for success to avoid `empty_success` cases. |
| 227 | +- Add dedupe keys for repeated `[ADF] PR gate remediation` issues. |
| 228 | +- Ensure auto-merge confidence can reach threshold only when required contexts are posted by the correct local/server actor. |
| 229 | + |
| 230 | +### Phase 6: Worktree and Sandbox Hardening |
| 231 | + |
| 232 | +- Make project-local worktrees use ownership manifests from PR `#1739` once merged. |
| 233 | +- Require local runner cleanup on normal exit and startup sweep for residue. |
| 234 | +- For server mode, keep root-owned cleanup in systemd `ExecStartPre` only. |
| 235 | +- Integrate Docker/RLM sandbox constraints from `#1709` and PR `#1491` before enabling arbitrary build agents by default. |
| 236 | + |
| 237 | +## Test Plan |
| 238 | + |
| 239 | +- Unit test `.terraphim` discovery for nested directories and symlinks. |
| 240 | +- Unit test `ProjectAdfConfig` parsing with minimal and full examples. |
| 241 | +- Unit test conversion from `ProjectAdfConfig` to `OrchestratorConfig`. |
| 242 | +- Integration test `adf --local --check` from a fixture repository with `.terraphim/adf.toml`. |
| 243 | +- Integration test local single-agent execution with a harmless command. |
| 244 | +- Integration test server import of two fixture repositories with distinct project ids. |
| 245 | +- Regression test banned provider rejection in project-local config. |
| 246 | +- Regression test secret redaction in `Debug` output. |
| 247 | +- Regression test that `adf.local.toml` is ignored by server imports. |
| 248 | + |
| 249 | +## Acceptance Criteria |
| 250 | + |
| 251 | +- A developer can run `adf --local --check` anywhere inside a repository and see the resolved project root, agents, models, and Gitea target. |
| 252 | +- A developer can run one configured local agent without a server orchestrator running. |
| 253 | +- The server can import the same project's `.terraphim/adf.toml` and run its agents as part of the existing multi-project fleet. |
| 254 | +- `OrchestratorConfig::validate()` remains the single validation gate for provider allow-list and project-agent references. |
| 255 | +- No secrets are printed by `--check`, debug formatting, status comments, or failure issues. |
| 256 | +- Repeated PR gate failures produce deduped remediation issues instead of issue storms. |
| 257 | + |
| 258 | +## Open Decisions |
| 259 | + |
| 260 | +- Should `.terraphim/adf.toml` be TOML only, or should JSON be supported for symmetry with `.terraphim/config.json`? |
| 261 | +- Should local mode be able to post commit statuses by default, or require `--post-status`? |
| 262 | +- Should local mode support long-running watch/reconcile mode, or only foreground one-shot agent runs initially? |
| 263 | +- Should server import pull project config from the checked-out repository path, Gitea raw files, or both? |
| 264 | +- How should conflicts be resolved when server `conf.d/<project>.toml` and project `.terraphim/adf.toml` both define the same agent? |
| 265 | + |
| 266 | +## Recommended First Slice |
| 267 | + |
| 268 | +Implement the smallest useful vertical slice: |
| 269 | + |
| 270 | +1. Add `ProjectAdfConfig` and parse `.terraphim/adf.toml`. |
| 271 | +2. Convert it into a single-project `OrchestratorConfig`. |
| 272 | +3. Add `adf --local --check`. |
| 273 | +4. Add tests for discovery, parsing, conversion, and banned provider validation. |
| 274 | + |
| 275 | +This validates the configuration model without changing server runtime behaviour. Local one-agent execution and server import should follow after the schema proves stable. |
0 commit comments