Skip to content

Commit 8d5bd59

Browse files
darkspockclaude
darkspock
and
claude
committed
EPIC-12 revised + EPIC-13 created (exec method)
EPIC-12 (Worker Relay) revised with validated decisions: - Auth via /workers/{id}/relay endpoint (worker credential, no task keys) - Full relay: heartbeats, results, chat, artifacts - Breaking change: CRONCONTROL_API_URL → CRONCONTROL_URL - Depends on EPIC-13 (exec method is the primary use case) EPIC-13 (Local Exec Method) created: - New exec execution method for local subprocess execution - Start/Poll/Kill with PID handle, SIGTERM/SIGKILL - Env injection: CRONCONTROL_URL, API_KEY, RUN_ID, JOB_ID - Worker-only registration (not on control plane) - method_config: command, working_dir, timeout, shell, user Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent d55249b commit 8d5bd59

2 files changed

Lines changed: 451 additions & 142 deletions

File tree

‎docs/epics/epic-12-worker-relay.md‎

Lines changed: 111 additions & 142 deletions
Original file line numberDiff line numberDiff line change
@@ -2,111 +2,89 @@
22

33
## Outcome
44

5-
The CronControl worker optionally exposes a local HTTP endpoint that acts as a relay/proxy for SDK calls (heartbeats, results, chat, artifacts). Tasks running on the same host as the worker can send telemetry to `localhost` instead of directly to the control plane, gaining resilience, lower latency, and buffering. The relay is opt-in and only benefits tasks that execute on the worker host.
5+
The CronControl worker optionally exposes a local HTTP endpoint that acts as a relay for all SDK write calls (heartbeats, results, chat, artifacts). Tasks running as subprocesses on the worker host send telemetry to `localhost` instead of directly to the control plane, gaining resilience, lower latency, and buffering.
66

77
## Why This Epic Exists
88

9-
Today, SDKs call the CronControl control plane directly for heartbeats, results, artifacts, and chat. This works but has three weaknesses:
9+
When a worker runs a local subprocess (EPIC-13 `exec` method), the subprocess uses the CronControl SDK to report heartbeats, set results, post chat messages, and upload artifacts. These calls go directly to the control plane over the network. If the control plane is temporarily unavailable, the SDK call blocks or fails — impacting the running task.
1010

11-
1. **Fragility**: If the control plane is temporarily unavailable (deploy, network blip), the SDK call blocks or fails inside the running task, potentially affecting its execution.
12-
2. **Latency**: Every heartbeat is a remote HTTP call (50-200ms). For tasks reporting progress every second, this adds up.
13-
3. **Network isolation**: In private networks, tasks may not have direct route to the control plane. The worker already has connectivity — tasks should leverage it.
11+
The relay buffers, batches, and retries these calls locally. The task always gets a fast `202 Accepted` from localhost and continues working.
1412

15-
## Scope: Which Execution Methods Benefit
13+
## Scope
1614

17-
The relay only works when the task process runs **on the same host** as the worker binary.
15+
The relay benefits tasks that run **on the same host** as the worker:
1816

19-
| Execution Method | Relay Works? | Why |
20-
|-----------------|-------------|-----|
21-
| HTTP (sync) | No | HTTP tasks call external URLs, no subprocess spawned |
22-
| HTTP (async_tracked) | No | Same — the "task" is a remote HTTP server |
23-
| SSH (foreground) | No | Task runs on remote SSH host, not on worker host |
24-
| SSH (detached) | No | Same — remote host |
25-
| SSM | No | Task runs on EC2 instance, not on worker host |
26-
| K8s | Partial | Only if worker and task pod share network (sidecar pattern) |
27-
| Container (Docker) | Yes | If worker runs on Swarm manager, containers share host network |
28-
| Future: subprocess/exec | Yes | Task is a child process of the worker |
29-
30-
**Key insight**: The relay is most useful for a future subprocess/exec execution method where the worker spawns tasks as local child processes. For SSH/SSM, tasks must continue calling the control plane directly.
17+
| Execution Method | Relay Works? | Notes |
18+
|-----------------|-------------|-------|
19+
| `exec` (EPIC-13) | **Yes** | Primary use case — subprocess on worker host |
20+
| Container (Docker) | Yes | If containers share host network |
21+
| K8s (sidecar) | Partial | Only with sidecar deployment pattern |
22+
| HTTP / SSH / SSM | No | Tasks run on remote hosts |
3123

3224
## How It Works
3325

3426
### Architecture
3527

3628
```
37-
Task Process (same host) Worker (localhost:9091) CronControl
29+
Subprocess (same host) Worker Relay (localhost:9091) CronControl
3830
| | |
3931
|-- POST /heartbeat ---------->| |
4032
|<-- 202 Accepted -------------| |
41-
| |-- (buffer, batch, retry) --->|
42-
| |<-- 200 OK -------------------|
33+
| |-- batch flush every 3s ----->|
4334
| | |
4435
|-- PATCH /runs/{id}/result -->| |
4536
|<-- 202 Accepted -------------| |
46-
| |-- forward (with API key) --->|
37+
| |-- POST /workers/{id}/relay ->|
38+
| | (worker credential auth) |
39+
| |<-- 200 OK -------------------|
4740
```
4841

49-
### Opt-in Activation
42+
### Authentication
5043

51-
Disabled by default. Enabled via worker flag:
44+
No task-scoped API keys. The relay uses the **worker credential** to forward all calls through a single control plane endpoint:
5245

53-
```bash
54-
croncontrol-worker --url https://croncontrol.io --credential wrk_cred_abc \
55-
--relay-port 9091
46+
```
47+
POST /api/v1/workers/{id}/relay
48+
X-Worker-Credential: wrk_cred_abc123
49+
Content-Type: application/json
50+
51+
{
52+
"method": "PATCH",
53+
"path": "/api/v1/runs/run_01ABC/result",
54+
"body": {"data": {"deleted": 42}}
55+
}
5656
```
5757

58-
When enabled:
59-
- Worker starts a local HTTP server on `--relay-port` (default: 0 = disabled)
60-
- Worker injects `CRONCONTROL_RELAY_URL=http://localhost:9091` into task environment
61-
- SDKs check `CRONCONTROL_RELAY_URL` first, fall back to `CRONCONTROL_URL`
58+
The control plane verifies the worker credential, checks the run/job belongs to the worker's workspace, and executes the operation internally.
6259

63-
### Authentication Model
60+
**Heartbeats** are the exception — they are unauthenticated on both relay and control plane, so they are forwarded directly to `POST /api/v1/heartbeat` without the relay endpoint.
6461

65-
The control plane includes a **task-scoped API key** in the task payload dispatched to the worker. This key has the same permissions as the workspace API key but is short-lived (expires when the task finishes).
62+
### Opt-in Activation
6663

67-
```
68-
Control Plane Worker Task
69-
| | |
70-
|-- dispatch(task, api_key) -->| |
71-
| |-- inject RELAY_URL ------->|
72-
| | store api_key locally |
73-
| | |
74-
| |<-- POST /heartbeat --------|
75-
| | (no auth, unauthenticated endpoint)
76-
| | |
77-
| |<-- PATCH /result ----------|
78-
|<-- forward + X-API-Key ------| (relay adds stored key) |
64+
```bash
65+
croncontrol-worker --url https://croncontrol.io --credential wrk_cred_abc \
66+
--relay-port 9091
7967
```
8068

81-
**Heartbeats**: unauthenticated (both on relay and control plane). No key needed.
82-
**Results, chat, artifacts**: relay injects the task-scoped API key from the stored payload.
83-
84-
This requires:
85-
- New field `TaskAPIKey string` in `worker.Task` struct
86-
- Control plane generates a short-lived key on dispatch, includes it in the task payload
87-
- Worker stores it per active task and adds `X-API-Key` header when forwarding
88-
89-
### Environment Variable Naming
69+
When enabled:
70+
- Worker starts local HTTP server on `--relay-port` (default: 0 = disabled)
71+
- Worker injects `CRONCONTROL_RELAY_URL=http://localhost:9091` into subprocess env
72+
- SDKs detect `CRONCONTROL_RELAY_URL` and route write operations through it
9073

91-
Current state (inconsistent):
92-
- Worker injects: `CRONCONTROL_API_URL`
93-
- SDKs read: `CRONCONTROL_URL`
94-
- Relay adds: `CRONCONTROL_RELAY_URL`
74+
### Environment Variables (Breaking Change)
9575

96-
**Resolution**: Standardize on `CRONCONTROL_URL` everywhere. The worker should inject `CRONCONTROL_URL` (not `CRONCONTROL_API_URL`). SDKs already read it. The relay URL is a separate variable:
76+
Rename `CRONCONTROL_API_URL` → `CRONCONTROL_URL` everywhere:
9777

9878
| Variable | Set By | Used For |
9979
|----------|--------|----------|
10080
| `CRONCONTROL_URL` | Worker or user | Base URL for all SDK calls |
10181
| `CRONCONTROL_RELAY_URL` | Worker (when relay enabled) | Override for write operations |
10282
| `CRONCONTROL_API_KEY` | Worker or user | Authentication |
83+
| `CRONCONTROL_RUN_ID` | Worker | Current run ID |
10384

104-
### SDK Changes
105-
106-
Both `CRONCONTROL_RELAY_URL` detection and write routing need to be added. This is ~10 lines per SDK, not 1 line.
85+
### SDK Changes (~10 lines per SDK)
10786

10887
```python
109-
# Python SDK
11088
class CronControl:
11189
def __init__(self, base_url=None, api_key=None):
11290
self.relay_url = os.environ.get('CRONCONTROL_RELAY_URL')
@@ -115,122 +93,113 @@ class CronControl:
11593

11694
def _request(self, method, path, body=None, use_relay=False):
11795
base = self.relay_url if (use_relay and self.relay_url) else self.base_url
118-
# ... rest unchanged
96+
# ... existing logic
11997

98+
# Write operations → relay
12099
def heartbeat(self, run_id, total, current, message=''):
121100
return self._request('POST', '/heartbeat', body={...}, use_relay=True)
122-
123101
def set_result(self, run_id, data):
124102
return self._request('PATCH', f'/runs/{run_id}/result', body=data, use_relay=True)
103+
def post_chat(self, orchestra_id, content, msg_type='text'):
104+
return self._request('POST', f'/orchestras/{orchestra_id}/chat', body={...}, use_relay=True)
125105

126-
# Read operations always go direct
106+
# Read operations → direct
127107
def get_result(self, run_id):
128-
return self._request('GET', f'/runs/{run_id}/result', use_relay=False)
108+
return self._request('GET', f'/runs/{run_id}/result')
129109
```
130110

131111
### Relayed Endpoints
132112

133-
| Endpoint | Buffered | Batched | Auth Required | Notes |
134-
|----------|----------|---------|---------------|-------|
135-
| `POST /heartbeat` | Yes | Yes (aggregate by run_id) | No | Most frequent |
136-
| `PATCH /runs/{id}/result` | Yes | No | Yes (task key) | Forward immediately |
137-
| `POST /orchestras/{id}/chat` | Yes | No | Yes (task key) | Forward immediately |
138-
| `POST /runs/{id}/artifacts` | Yes | No | Yes (task key) | Stream-forward |
139-
| `GET /*` | No | No | — | Pass-through or reject |
113+
| Endpoint | Buffered | Batched | Forwarded Via |
114+
|----------|----------|---------|---------------|
115+
| `POST /heartbeat` | Yes | Yes (latest per run_id, flush 3s) | Direct (unauthenticated) |
116+
| `PATCH /runs/{id}/result` | Yes | No | `/workers/{id}/relay` |
117+
| `POST /orchestras/{id}/chat` | Yes | No | `/workers/{id}/relay` |
118+
| `POST /runs/{id}/artifacts` | Yes | No | `/workers/{id}/relay` |
140119

141-
### Buffering Strategy
120+
### Buffering
142121

143122
```
144123
Heartbeats:
145-
- Buffer in memory: map[run_id] → latest heartbeat
146-
- Flush every 3 seconds (configurable)
147-
- On flush: POST /heartbeat for each buffered entry
148-
- On control plane error: retain buffer, retry next flush
149-
- Max buffer: 1000 entries (oldest evicted on overflow)
124+
- map[run_id] → latest heartbeat
125+
- Flush every 3s (configurable)
126+
- On error: retain, retry next flush
127+
- Max 1000 entries (evict oldest on overflow)
150128
151129
Results / Chat / Artifacts:
152-
- Forward immediately (no aggregation)
153-
- On failure: retry 3x with 1s backoff
154-
- On persistent failure: return 502 to SDK (task can handle or ignore)
130+
- Forward immediately, retry 3x with 1s backoff
131+
- On persistent failure: return 502 to SDK
155132
```
156133

157-
## Configuration
158-
159-
```bash
160-
# Worker flags
161-
--relay-port 9091 # Enable relay on this port (0 = disabled, default)
162-
--relay-flush-interval 3s # Heartbeat flush interval (default 3s)
163-
--relay-max-buffer 1000 # Max buffered heartbeats (default 1000)
164-
--relay-retry-count 3 # Retries for forwarded calls (default 3)
165-
```
166-
167-
No control plane configuration needed beyond generating the task-scoped API key on dispatch.
134+
## Control Plane: Relay Endpoint
168135

169-
## Data Model
170-
171-
### Task struct change
136+
New endpoint on the control plane:
172137

173138
```go
174-
type Task struct {
175-
// ... existing fields
176-
TaskAPIKey string `json:"task_api_key,omitempty"` // short-lived workspace key for relay forwarding
139+
// POST /api/v1/workers/{id}/relay
140+
// Auth: worker credential
141+
func (s *Service) WorkerRelay(w http.ResponseWriter, r *http.Request) {
142+
worker := authenticateWorkerRequest(r)
143+
144+
var req struct {
145+
Method string `json:"method"`
146+
Path string `json:"path"`
147+
Body json.RawMessage `json:"body"`
148+
}
149+
150+
// Validate path is in allowlist
151+
// Execute the operation internally with workspace context
152+
// Return result to relay
177153
}
178154
```
179155

180-
### Task-scoped API key
181-
182-
Generated on dispatch, stored in `api_keys` table with:
183-
- `name`: `task_{run_id}` or `task_{job_id}`
184-
- `role`: `operator`
185-
- `expires_at`: task timeout or 24h max
186-
- Deleted on task completion
187-
188-
No schema migration needed — uses existing `api_keys` table.
189-
190-
## Implementation Notes
156+
Allowlisted paths:
157+
- `PATCH /api/v1/runs/*/result`
158+
- `POST /api/v1/orchestras/*/chat`
159+
- `POST /api/v1/runs/*/artifacts`
191160

192-
### SSH/SSM Detached Env Gap
161+
## Configuration
193162

194-
The SSH detached start path (`ssh.go` line 140) does not pass `params.Environment` to `runRemoteCommand`. This is a pre-existing gap. Even without the relay, `CRONCONTROL_URL` and custom env vars are not available to detached SSH tasks. This should be fixed independently of EPIC-12.
163+
```bash
164+
--relay-port 9091 # 0 = disabled (default)
165+
--relay-flush-interval 3s # Heartbeat flush interval
166+
--relay-max-buffer 1000 # Max buffered heartbeats
167+
--relay-retry-count 3 # Retries for forwarded calls
168+
```
195169

196-
### K8s Sidecar Pattern
170+
## Data Model
197171

198-
For K8s, the relay can work if the worker runs as a sidecar container in the same pod as the task. In this case, `localhost` correctly resolves within the pod network. This is an advanced deployment pattern documented separately.
172+
No database changes. No new tables. The relay endpoint uses existing worker authentication.
199173

200174
## Security
201175

202-
- Relay listens on `localhost` only — not exposed to external network
203-
- No authentication on relay endpoints (same-host trust model)
204-
- Relay authenticates to control plane using task-scoped API key (not worker credential)
205-
- Task-scoped API key is short-lived and auto-deleted on task completion
206-
- Artifacts forwarded with task-scoped key (workspace-scoped permissions)
207-
208-
## Out of Scope
209-
210-
- Persistent buffering (disk-based) — memory only
211-
- Relay for SSH/SSM tasks (they run on remote hosts)
212-
- Multi-worker relay aggregation
213-
- WebSocket/SSE relay for real-time chat streaming
214-
- Subprocess/exec execution method (separate epic)
176+
- Relay listens on localhost only
177+
- No auth on relay (same-host trust)
178+
- Relay → control plane uses worker credential (existing auth)
179+
- Control plane validates workspace ownership on every relayed operation
180+
- Allowlisted paths prevent arbitrary API access through relay
215181

216182
## Dependencies
217183

218-
- EPIC-11 (async execution handles — worker control poll)
219-
- Worker binary (`cmd/croncontrol-worker`)
184+
- **EPIC-13** (exec method — primary use case for relay)
185+
- EPIC-11 (Start/Poll/Kill contract)
220186
- All 5 SDKs
221-
- Fix: rename `CRONCONTROL_API_URL` → `CRONCONTROL_URL` in worker (pre-requisite)
187+
188+
## Out of Scope
189+
190+
- Persistent buffering (disk)
191+
- Relay for SSH/SSM tasks (remote hosts)
192+
- WebSocket/SSE relay
222193

223194
## Acceptance Criteria
224195

225196
- [ ] Worker `--relay-port` flag enables local HTTP relay
226197
- [ ] Heartbeats buffered and flushed in batch every 3s
227-
- [ ] Results, chat, artifacts forwarded immediately with task-scoped API key
228-
- [ ] Task-scoped API key generated on dispatch, deleted on completion
229-
- [ ] `CRONCONTROL_RELAY_URL` injected into task environment
230-
- [ ] SDKs detect relay URL and use it for write operations (heartbeat, result, chat, artifacts)
231-
- [ ] SDKs use direct URL for read operations
232-
- [ ] Relay gracefully handles control plane unavailability (buffer, retry, 502)
233-
- [ ] Relay listens on localhost only
234-
- [ ] No impact when relay is disabled (default behavior unchanged)
235-
- [ ] Worker logs relay stats (buffered count, flush success/failure)
236-
- [ ] Environment variable renamed: `CRONCONTROL_API_URL` → `CRONCONTROL_URL`
198+
- [ ] Results, chat, artifacts forwarded via `/workers/{id}/relay`
199+
- [ ] Control plane relay endpoint with worker credential auth and path allowlist
200+
- [ ] `CRONCONTROL_RELAY_URL` injected into subprocess env
201+
- [ ] SDKs detect relay URL for write operations
202+
- [ ] Relay handles control plane unavailability (buffer, retry, 502)
203+
- [ ] localhost only binding
204+
- [ ] No impact when disabled
205+
- [ ] Breaking change: `CRONCONTROL_API_URL` → `CRONCONTROL_URL`

0 commit comments

Comments
 (0)