Skip to content

Commit 5cb6afe

Browse files
kasvithkasvith
committed
docs: split WORKPLAN.md into per-milestone planning documents
Replace monolithic WORKPLAN.md with ROADMAP.md index and 7 milestone docs (M01-M07) covering graph visualization, developer tooling, testing framework, documentation/release, message bus, Phoenix dashboard, and scalability adapters. Each milestone includes architecture deep-dives, implementation plans, and acceptance criteria.
1 parent 0b0aeee commit 5cb6afe

11 files changed

+1610
-1
lines changed

‎agents/ROADMAP.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# Durable — Roadmap
2+
3+
**Last Updated:** 2026-03-09
4+
**Reference:** See `arch.md` for technical architecture, `WORKPLAN_ARCHIVED.md` for historical detail
5+
6+
---
7+
8+
## Quick Stats
9+
10+
| Metric | Value |
11+
|--------|-------|
12+
| Source Modules | 42 |
13+
| Passing Tests | ~291 |
14+
| Documentation Guides | 6 |
15+
| Lines of Code | ~11,000 |
16+
17+
---
18+
19+
## Completed Work
20+
21+
| Area | Key Features | Tests |
22+
|------|-------------|-------|
23+
| **Foundation** | Mix project, Ecto schemas (6), programmatic migrations, NimbleOptions config, CI (`mix precommit`), Credo strict | — |
24+
| **Core MVP** | `workflow`/`step`/`decision` macros, context management, executor, retry/backoff, PostgreSQL queue (FOR UPDATE SKIP LOCKED), query API, time helpers | ~65 |
25+
| **Wait Primitives** | `sleep`, `schedule_at`, `wait_for_event`, `wait_for_any/all`, `wait_for_input`, `wait_for_approval`, `wait_for_choice`, `wait_for_text`, `wait_for_form`, timeouts | 52 |
26+
| **Branching** | `branch on:` macro with pattern matching, default clause, multi-step branches | 19 |
27+
| **Parallel Execution** | `parallel do` macro, `__results__` model, `into:` merge, `returns:` option, error strategies, resume durability | 20 |
28+
| **Compensation/Saga** | `compensate` macro, `step :name, compensate:`, reverse-order execution | 10 |
29+
| **Cron Scheduling** | `@schedule` decorator, scheduler GenServer, multi-node safety, timezone support, CRUD API, enable/disable, manual trigger | 49 |
30+
| **Workflow Orchestration** | `call_workflow/3`, `start_workflow/3`, parallel call_workflow, cascade cancellation, parent notifications, `list_children/2` | 12 |
31+
| **Observability** | Logger handler, IO capture, log buffer | 13 |
32+
| **Developer Experience** | `mix durable.gen.migration`, `mix durable.install`, DataCase, @moduledoc/@doc/@spec, 6 guides | — |
33+
34+
**Total Tests:** ~291
35+
36+
---
37+
38+
## Active Milestones
39+
40+
| ID | Title | Priority | Effort | Status | Dependencies |
41+
|----|-------|----------|--------|--------|--------------|
42+
| [M01](milestones/M01-graph-visualization.md) | Graph Visualization | Medium | 1 week | Not Started | None |
43+
| [M02](milestones/M02-developer-tooling.md) | Developer Tooling | High | 1 week | Not Started | None |
44+
| [M03](milestones/M03-testing-framework.md) | Testing Framework | Medium | 1 week | Not Started | None |
45+
| [M04](milestones/M04-documentation-release.md) | Documentation & Release | High | 1 week | Not Started | M02, M03 (soft) |
46+
| [M05](milestones/M05-message-bus.md) | Message Bus | Medium | 1.5 weeks | Not Started | None |
47+
| [M06](milestones/M06-phoenix-dashboard.md) | Phoenix Dashboard | Low | 2 weeks | Not Started | M01, M05 |
48+
| [M07](milestones/M07-scalability-adapters.md) | Scalability Adapters | Low | 2 weeks | Not Started | M05 (soft) |
49+
50+
### Recommended Execution Order
51+
52+
```
53+
M02 (Developer Tooling) ─┐
54+
M03 (Testing Framework) ─┤──→ M04 (Documentation & Release)
55+
M01 (Graph Visualization) ┘
56+
──→ M05 (Message Bus)
57+
├──→ M06 (Phoenix Dashboard)
58+
└──→ M07 (Scalability Adapters)
59+
```
60+
61+
**Phase A** (parallel, no dependencies): M01, M02, M03
62+
**Phase B** (needs Phase A): M04
63+
**Phase C** (independent): M05
64+
**Phase D** (needs M01 + M05): M06, M07
65+
66+
---
67+
68+
## Backlog
69+
70+
Lower-priority items not warranting standalone milestone documents:
71+
72+
| Item | Notes |
73+
|------|-------|
74+
| Switch/Case macro | Low priority — `branch` covers most use cases |
75+
| Pipe-based API | Functional workflow composition — nice-to-have |
76+
| Example Project | Standalone demo app with common patterns |
77+
| Redis message bus | Quick follow-up after M05 + M07 |
78+
| RabbitMQ queue adapter | Niche — add if requested |
79+
80+
---
81+
82+
## Test Coverage
83+
84+
| Test File | Tests | Area |
85+
|-----------|-------|------|
86+
| wait_test.exs | 52 | Wait primitives |
87+
| scheduler_test.exs | 49 | Cron scheduling |
88+
| parallel_test.exs | 20 | Parallel execution |
89+
| branch_test.exs | 19 | Branch macro |
90+
| postgres_test.exs | 16 | Queue adapter |
91+
| decision_test.exs | 14 | Decision steps |
92+
| log_capture_test.exs | 13 | Log/IO capture |
93+
| orchestration_test.exs | 12 | Workflow orchestration |
94+
| integration_test.exs | 11 | End-to-end flows |
95+
| validation_test.exs | 10 | Input validation |
96+
| context_test.exs | 10 | Context management |
97+
| compensation_test.exs | 10 | Saga pattern |
98+
| durable_test.exs | 10 | Core API |
99+
| handler_test.exs | 8 | Log handler |
100+
| io_server_test.exs | 7 | IO capture |
101+
| resume_edge_cases_test.exs | 5 | Resume edge cases |
102+
| log_capture/integration_test.exs | 5 | Log capture integration |
103+
| Other | ~20 | Misc |
104+
| **Total** | **~291** | |
105+
106+
---
107+
108+
## Known Limitations
109+
110+
1. Wait primitives not supported in parallel blocks
111+
2. Child workflows with waits (`sleep`, `wait_for_event`) not supported in parallel blocks
112+
3. No backward jumps in decision steps (forward-only by design)
113+
4. Context is single-level atomized (top-level keys only)
114+
5. No workflow versioning
115+
6. No foreach/loop DSL primitives (use Elixir's `Enum` functions)

agents/WORKPLAN.md renamed to agents/WORKPLAN_ARCHIVED.md

File renamed without changes.

‎agents/context-index.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,10 +121,20 @@ agents/
121121
│ ├── README.md # Topic overview
122122
│ ├── sessions/ # Individual session records
123123
│ └── implementation-plan.md
124+
├── milestones/ # Per-feature planning documents
125+
│ ├── _TEMPLATE.md # Milestone template
126+
│ ├── M01-graph-visualization.md
127+
│ ├── M02-developer-tooling.md
128+
│ ├── M03-testing-framework.md
129+
│ ├── M04-documentation-release.md
130+
│ ├── M05-message-bus.md
131+
│ ├── M06-phoenix-dashboard.md
132+
│ └── M07-scalability-adapters.md
124133
├── context-index.md # This file
125134
├── .archived-topics.json # Machine-readable metadata
126135
├── arch.md # Architecture & technical reference
127-
├── WORKPLAN.md # Current status & work planning
136+
├── ROADMAP.md # Current status & milestone index
137+
├── WORKPLAN_ARCHIVED.md # Historical (replaced by ROADMAP.md + milestones/)
128138
└── IMPLEMENTATION_PLAN_ARCHIVED.md # Historical (no longer maintained)
129139
```
130140

‎agents/milestones/M01-graph-visualization.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# M01: Graph Visualization
2+
3+
**Status:** Not Started
4+
**Priority:** Medium
5+
**Effort:** 1 week
6+
**Dependencies:** None
7+
8+
## Motivation
9+
10+
Complex workflows with branches, parallel blocks, and compensation edges are hard to reason about from code alone. Graph visualization provides a structural view of workflow definitions and, with execution state overlay, lets developers debug running/failed workflows visually. This also feeds into M06 (Phoenix Dashboard) as the primary workflow visualization component.
11+
12+
## Scope
13+
14+
### In Scope
15+
16+
- Graph data structure (`%Graph{nodes, edges}`) from workflow definitions
17+
- Builder that walks `Durable.Definition` structs
18+
- Export to DOT (Graphviz), Mermaid, and Cytoscape.js JSON formats
19+
- Execution state overlay (join graph nodes with `step_executions` data)
20+
- Support for all node types: step, decision, branch, parallel (fork/join), compensation
21+
22+
### Out of Scope
23+
24+
- Live/real-time graph updates (deferred to M05/M06 — needs message bus)
25+
- Visual layout engine (consumers handle layout — DOT/Mermaid/Cytoscape all have their own)
26+
- Phoenix LiveView rendering (M06)
27+
- WebSocket streaming (M06)
28+
29+
## Architecture & Design
30+
31+
### Module Layout
32+
33+
```
34+
lib/durable/graph.ex # Public API: generate/2, with_execution_state/3
35+
lib/durable/graph/builder.ex # Walks Definition structs → %Graph{}
36+
lib/durable/graph/export/dot.ex # DOT format export
37+
lib/durable/graph/export/mermaid.ex # Mermaid format export
38+
lib/durable/graph/export/cytoscape.ex # Cytoscape.js JSON export
39+
```
40+
41+
### Data Structures
42+
43+
```elixir
44+
defmodule Durable.Graph do
45+
defstruct [:workflow_name, :workflow_module, nodes: [], edges: []]
46+
47+
defmodule Node do
48+
defstruct [:id, :type, :label, :metadata]
49+
# type: :start | :end | :step | :decision | :branch_fork | :branch_join
50+
# | :parallel_fork | :parallel_join | :compensation
51+
end
52+
53+
defmodule Edge do
54+
defstruct [:from, :to, :label, :type]
55+
# type: :normal | :branch | :compensation | :parallel
56+
end
57+
end
58+
```
59+
60+
### Builder Logic
61+
62+
The builder walks the list of `%Durable.Definition.Step{}` structs returned by `Module.__workflow_definition__/1`:
63+
64+
1. Add `:start` node
65+
2. Iterate steps in order:
66+
- **Regular step** → add node + edge from previous
67+
- **Decision** → add diamond node, edges for each `{:goto, target}`
68+
- **Branch** → add fork node, walk each branch's steps, add join node
69+
- **Parallel** → add fork node, one edge per parallel step, add join node
70+
- **Step with `compensate:`** → add dashed compensation edge back to compensate target
71+
3. Add `:end` node
72+
73+
### Execution State Overlay
74+
75+
```elixir
76+
Durable.Graph.with_execution_state(graph, workflow_id, opts)
77+
```
78+
79+
Queries `step_executions` via `Durable.Query` and merges status/timing into node metadata:
80+
81+
```elixir
82+
%Node{
83+
id: :charge_payment,
84+
type: :step,
85+
metadata: %{
86+
execution: %{
87+
status: :completed,
88+
attempt: 2,
89+
duration_ms: 1234,
90+
started_at: ~U[...],
91+
completed_at: ~U[...]
92+
}
93+
}
94+
}
95+
```
96+
97+
### Export Format Details
98+
99+
**DOT**: Standard Graphviz digraph. Nodes styled by type (diamonds for decisions, boxes for steps, etc.). Execution state colors: green=completed, blue=running, red=failed, gray=pending.
100+
101+
**Mermaid**: `flowchart TD` format. Branch/parallel rendered as subgraphs. Compatible with GitHub markdown rendering.
102+
103+
**Cytoscape.js**: JSON elements array with `{ data: { id, label, type, ... } }` for nodes and `{ data: { source, target, label } }` for edges. Ready for frontend consumption.
104+
105+
## Implementation Plan
106+
107+
1. **Graph struct & public API** — `lib/durable/graph.ex`
108+
- Define `%Graph{}`, `%Node{}`, `%Edge{}` structs
109+
- Public functions: `generate/2`, `with_execution_state/3`
110+
- Delegate to builder/query internally
111+
112+
2. **Graph builder** — `lib/durable/graph/builder.ex`
113+
- `build/1` takes a workflow module, returns `%Graph{}`
114+
- Walk `__workflow_definition__/1` step list
115+
- Handle each step type (step, decision, branch, parallel, compensation)
116+
- Pure function — no side effects, no DB access
117+
118+
3. **DOT exporter** — `lib/durable/graph/export/dot.ex`
119+
- `to_dot/1` takes `%Graph{}`, returns DOT string
120+
- Node shape mapping: step→box, decision→diamond, start/end→circle
121+
- Optional execution state coloring
122+
123+
4. **Mermaid exporter** — `lib/durable/graph/export/mermaid.ex`
124+
- `to_mermaid/1` takes `%Graph{}`, returns Mermaid string
125+
- Use `flowchart TD` direction
126+
- Subgraphs for parallel blocks
127+
128+
5. **Cytoscape exporter** — `lib/durable/graph/export/cytoscape.ex`
129+
- `to_cytoscape/1` takes `%Graph{}`, returns JSON-encodable map
130+
- Standard Cytoscape elements format
131+
132+
6. **Execution overlay** — integrate into `lib/durable/graph.ex`
133+
- Query step executions for a workflow_id
134+
- Merge execution data into node metadata
135+
136+
## Testing Strategy
137+
138+
- `test/durable/graph/builder_test.exs` — test graph generation for each workflow pattern:
139+
- Linear steps
140+
- Decision with goto
141+
- Branch with multiple paths
142+
- Parallel block
143+
- Compensation edges
144+
- Mixed (branch + parallel + compensation)
145+
- `test/durable/graph/export/dot_test.exs` — validate DOT output format
146+
- `test/durable/graph/export/mermaid_test.exs` — validate Mermaid output format
147+
- `test/durable/graph/export/cytoscape_test.exs` — validate Cytoscape JSON structure
148+
- `test/durable/graph/execution_state_test.exs` — test overlay with DB fixtures (use DataCase)
149+
- Define test workflow modules in `test/support/` for predictable graph structures
150+
151+
## Acceptance Criteria
152+
153+
- [ ] `Durable.Graph.generate(MyWorkflow)` returns `%Graph{}` with correct nodes/edges
154+
- [ ] All step types produce expected graph structures (step, decision, branch, parallel, compensation)
155+
- [ ] DOT export produces valid Graphviz syntax (verify with `dot -Tsvg` if available)
156+
- [ ] Mermaid export produces valid Mermaid syntax
157+
- [ ] Cytoscape export produces valid JSON elements
158+
- [ ] Execution state overlay merges step status into node metadata
159+
- [ ] All exports handle execution state coloring/annotation
160+
- [ ] No duplicated nodes or edges for converging branches
161+
- [ ] `mix credo --strict` passes
162+
- [ ] All new tests pass
163+
164+
## Open Questions
165+
166+
- Should `generate/2` accept a workflow name filter for modules with multiple workflows?
167+
- Should execution overlay include step logs summary (log count, last error)?
168+
- Do we need a `to_ascii/1` exporter for terminal output in mix tasks (M02)?
169+
170+
## References
171+
172+
- `agents/arch.md` — "Graph Visualization" section for target API and data structures
173+
- `lib/durable/definition.ex` — `%Step{}` struct that builder walks
174+
- `lib/durable/query.ex` — query functions for execution state overlay
175+
- `lib/durable/executor.ex` — understanding of step type handling

0 commit comments

Comments
 (0)