Merge pull request #4 from wavezync/feat/call-workflows

feat: workflow orchestration (call_workflow / start_workflow)

Author: kasvith

README.md

@@ -14,6 +14,7 @@ A durable, resumable workflow engine for Elixir. Similar to Temporal/Inngest.
 - **Compensations** - Saga pattern with automatic rollback
 - **Cron Scheduling** - Recurring workflows with cron expressions
 - **Reliability** - Automatic retries with exponential/linear/constant backoff
+- **Orchestration** - Parent/child workflow composition
 - **Persistence** - PostgreSQL-backed execution state
 
 ## Installation
@@ -439,6 +440,38 @@ hours(2)      # 7_200_000 ms
 days(7)       # 604_800_000 ms
 ```
 
+### Orchestration
+
+```elixir
+use Durable.Orchestration
+
+# Synchronous: call child and wait for result
+case call_workflow(MyApp.PaymentWorkflow, %{"amount" => 100}, timeout: hours(1)) do
+  {:ok, result} -> {:ok, assign(data, :payment, result)}
+  {:error, reason} -> {:error, reason}
+end
+
+# Fire-and-forget: start child and continue
+{:ok, child_id} = start_workflow(MyApp.EmailWorkflow, %{"to" => email}, ref: :welcome)
+
+# call_workflow also works inside parallel blocks (executed inline)
+parallel do
+  step :payment, fn data ->
+    case call_workflow(MyApp.PaymentWorkflow, %{"amount" => data.total}, ref: :pay) do
+      {:ok, result} -> {:ok, assign(data, :payment, result)}
+      {:error, reason} -> {:error, reason}
+    end
+  end
+
+  step :shipping, fn data ->
+    case call_workflow(MyApp.ShippingWorkflow, %{"id" => data.order_id}, ref: :ship) do
+      {:ok, result} -> {:ok, assign(data, :shipping, result)}
+      {:error, reason} -> {:error, reason}
+    end
+  end
+end
+```
+
 ### API
 
 ```elixir
@@ -449,6 +482,7 @@ Durable.list_executions(workflow: Module, status: :running)
 Durable.cancel(id, "reason")
 Durable.send_event(id, "event", payload)
 Durable.provide_input(id, "input_name", data)
+Durable.list_children(parent_id)
 ```
 
 ## Guides
@@ -457,10 +491,10 @@ Durable.provide_input(id, "input_name", data)
 - [Parallel](guides/parallel.md) - Concurrent execution
 - [Compensations](guides/compensations.md) - Saga pattern
 - [Waiting](guides/waiting.md) - Sleep, events, human input
+- [Orchestration](guides/orchestration.md) - Parent/child workflow composition
 
 ## Coming Soon
 
-- Workflow orchestration (parent/child workflows)
 - Phoenix LiveView dashboard
 
 ## License

agents/IMPLEMENTATION_PLAN.md agents/IMPLEMENTATION_PLAN_ARCHIVED.mdagents/IMPLEMENTATION_PLAN.md renamed to agents/IMPLEMENTATION_PLAN_ARCHIVED.md

@@ -1,5 +1,13 @@
 # DurableWorkflow Implementation Plan
 
+> **⚠️ ARCHIVED (2026-01-23):** This document is no longer maintained.
+> See `WORKPLAN.md` for current status and `arch.md` for technical reference.
+>
+> Key changes since this document was written:
+> - ForEach primitive was **removed** (use `Enum.map` instead)
+> - Parallel execution uses new results model (`__results__`, `into:`, `returns:`)
+> - Loop primitive was **never implemented** (use step retries or `Enum` functions)
+
 ## Executive Summary
 
 This document outlines the complete implementation plan for **Durable**, a durable, resumable workflow engine for Elixir.

agents/WORKPLAN.md

@@ -1,19 +1,19 @@
 # Durable Workflow Engine - Work Plan
 
-**Last Updated:** 2026-01-03
+**Last Updated:** 2026-03-08
 **Overall Progress:** ~75% Complete
-**Reference:** See `IMPLEMENTATION_PLAN.md` for detailed design and code examples
+**Reference:** See `arch.md` for technical architecture
 
 ---
 
 ## Quick Stats
 
 | Metric | Value |
 |--------|-------|
-| Source Modules | 41 |
-| Passing Tests | 214 |
+| Source Modules | 42 |
+| Passing Tests | ~291 |
 | Documentation Guides | 6 |
-| Lines of Code | ~8,500 |
+| Lines of Code | ~11,000 |
 
 ---
 
@@ -24,8 +24,8 @@
 | 0 | Project Foundation | Complete | 100% |
 | 1 | Core MVP | Complete | 100% |
 | 2 | Observability | Partial | 40% |
-| 3 | Advanced Features | Mostly Complete | 85% |
-| 4 | Scalability | Not Started | 0% |
+| 3 | Advanced Features | Mostly Complete | 90% |
+| 4 | Scalability | Partial | ~5% |
 | 5 | Developer Experience | Partial | 35% |
 
 ---
@@ -86,9 +86,9 @@
 
 ---
 
-## Phase 3: Advanced Features [85%]
+## Phase 3: Advanced Features [90%]
 
-### 3.1-3.3 Wait Primitives [COMPLETE - 46 tests]
+### 3.1-3.3 Wait Primitives [COMPLETE - 52 tests]
 
 | Feature | Status |
 |---------|--------|
@@ -105,7 +105,7 @@
 | Timeout handling | Complete |
 | Context preservation | Complete |
 
-### 3.4 Conditional Branching [COMPLETE - 10 tests]
+### 3.4 Conditional Branching [COMPLETE - 19 tests]
 
 | Feature | Status |
 |---------|--------|
@@ -117,32 +117,33 @@
 
 ### 3.5 Loops [SKIPPED]
 
-Intentionally skipped - use step-level retries or `foreach` instead.
+Intentionally skipped - use step-level retries or Elixir's `Enum` functions instead.
 
-### 3.6 Parallel Execution [COMPLETE - 13 tests]
+### 3.6 Parallel Execution [COMPLETE - 20 tests]
 
 | Feature | Status |
 |---------|--------|
 | `parallel do` macro | Complete |
-| Context merge strategies | Complete |
+| Results model (`__results__`) | Complete |
+| `into:` custom merge function | Complete |
+| `returns:` option | Complete |
 | Error strategies | Complete |
 | Resume durability | Complete |
 
-### 3.7 ForEach [COMPLETE - 13 tests]
+See `guides/parallel.md` for comprehensive documentation.
 
-| Feature | Status |
-|---------|--------|
-| `foreach` macro | Complete |
-| `current_item/0`, `current_index/0` | Complete |
-| Sequential/Concurrent modes | Complete |
-| `:collect_as` option | Complete |
-| `:on_error` handling | Complete |
+### 3.7 ForEach [REMOVED]
+
+**Decision (2026-01-23):** The `foreach` primitive was removed. Users should use
+Elixir's built-in enumeration tools (`Enum.map`, `Task.async_stream`) for batch
+processing instead. This simplifies the DSL while providing the same functionality
+through idiomatic Elixir.
 
 ### 3.8 Switch/Case [NOT STARTED]
 
 Low priority - `branch` macro covers most cases.
 
-### 3.9 Compensation/Saga [COMPLETE - 6 tests]
+### 3.9 Compensation/Saga [COMPLETE - 10 tests]
 
 | Feature | Status |
 |---------|--------|
@@ -151,7 +152,7 @@ Low priority - `branch` macro covers most cases.
 | Reverse-order execution | Complete |
 | CompensationRunner | Complete |
 
-### 3.10 Cron Scheduling [COMPLETE - 45 tests]
+### 3.10 Cron Scheduling [COMPLETE - 49 tests]
 
 | Feature | Status |
 |---------|--------|
@@ -165,22 +166,35 @@ Low priority - `branch` macro covers most cases.
 | Manual trigger | Complete |
 | Telemetry events | Complete |
 
+### 3.11 Workflow Orchestration [COMPLETE - 12 tests]
+
+| Feature | Status |
+|---------|--------|
+| `call_workflow/3` (synchronous) | Complete |
+| `start_workflow/3` (fire-and-forget) | Complete |
+| `call_workflow` in `parallel` blocks (inline execution) | Complete |
+| Idempotent resume | Complete |
+| Cascade cancellation | Complete |
+| Parent notification on child complete/fail | Complete |
+| Nested workflows (A → B → C) | Complete |
+| `Durable.list_children/2` API | Complete |
+
+See `guides/orchestration.md` for comprehensive documentation.
+
 ### Remaining Phase 3 Work
 
 | Feature | Priority | Complexity |
 |---------|----------|------------|
-| Workflow Orchestration (`call_workflow`) | High | Medium |
-| Parent-child tracking | High | Low |
 | Switch/Case macro | Low | Low |
 | Pipe-based API | Low | Medium |
 
 ---
 
-## Phase 4: Scalability [0%]
+## Phase 4: Scalability [~5%]
 
 | Feature | Priority | Complexity |
 |---------|----------|------------|
-| Queue Adapter Behaviour | Complete | - |
+| Queue Adapter Behaviour | **Complete** | - |
 | Redis Queue Adapter | Medium | Medium |
 | RabbitMQ Queue Adapter | Low | Medium |
 | Message Bus Behaviour | Medium | Low |
@@ -229,60 +243,91 @@ Note: Multi-node scheduling already works via `FOR UPDATE SKIP LOCKED`.
 ### High Priority
 
 1. Guide: Getting Started
-2. Workflow Orchestration (`call_workflow`)
-3. HexDocs Publishing
-4. `mix durable.status`
+2. HexDocs Publishing
+3. `mix durable.status`
 
 ### Medium Priority
 
-5. Guide: Testing Workflows
-6. `Durable.TestCase`
-7. Graph Generation
-8. `mix durable.list`
-9. pg_notify Message Bus
+4. Guide: Testing Workflows
+5. `Durable.TestCase`
+6. Graph Generation
+7. `mix durable.list`
+8. pg_notify Message Bus
 
 ### Lower Priority
 
-10. Switch/Case macro
-11. Redis Queue Adapter
-12. Phoenix Dashboard
-13. Example Project
-14. Pipe-based API
+9. Switch/Case macro
+10. Redis Queue Adapter
+11. Phoenix Dashboard
+12. Example Project
+13. Pipe-based API
 
 ---
 
 ## Test Coverage
 
 | Test File | Tests | Area |
 |-----------|-------|------|
-| scheduler_test.exs | 45 | Cron scheduling |
-| wait_test.exs | 46 | Wait primitives |
-| decision_test.exs | 13 | Decision steps |
-| parallel_test.exs | 13 | Parallel execution |
-| foreach_test.exs | 13 | ForEach iteration |
+| wait_test.exs | 52 | Wait primitives |
+| scheduler_test.exs | 49 | Cron scheduling |
+| parallel_test.exs | 20 | Parallel execution |
+| branch_test.exs | 19 | Branch macro |
+| postgres_test.exs | 16 | Queue adapter |
+| decision_test.exs | 14 | Decision steps |
 | log_capture_test.exs | 13 | Log/IO capture |
+| orchestration_test.exs | 12 | Workflow orchestration |
 | integration_test.exs | 11 | End-to-end flows |
-| branch_test.exs | 10 | Branch macro |
-| durable_test.exs | 8 | Core API |
-| compensation_test.exs | 6 | Saga pattern |
-| Other | ~36 | Queue, handlers, etc. |
-| **Total** | **214** | |
+| validation_test.exs | 10 | Input validation |
+| context_test.exs | 10 | Context management |
+| compensation_test.exs | 10 | Saga pattern |
+| durable_test.exs | 10 | Core API |
+| handler_test.exs | 8 | Log handler |
+| io_server_test.exs | 7 | IO capture |
+| resume_edge_cases_test.exs | 5 | Resume edge cases |
+| log_capture/integration_test.exs | 5 | Log capture integration |
+| Other | ~20 | Misc |
+| **Total** | **~291** | |
 
 ---
 
 ## Known Limitations
 
-1. Wait primitives not supported in parallel/foreach blocks
-2. No backward jumps in decision steps (forward-only by design)
-3. Context is single-level atomized (top-level keys only)
-4. No workflow versioning
+1. Wait primitives not supported in parallel blocks
+2. Child workflows with waits (`sleep`, `wait_for_event`) not supported in parallel blocks
+3. No backward jumps in decision steps (forward-only by design)
+4. Context is single-level atomized (top-level keys only)
+5. No workflow versioning
+6. No foreach/loop DSL primitives (use Elixir's `Enum` functions)
 
 ---
 
 ## Next Steps
 
 1. **Documentation** - Getting Started guide and HexDocs publishing
-2. **Workflow Orchestration** - Child workflow support (`call_workflow`)
-3. **Graph Visualization** - Understanding complex workflows
+2. **Graph Visualization** - Understanding complex workflows
+3. **Testing Helpers** - `Durable.TestCase` for easier workflow testing
+
+The existing ~291 tests provide good confidence in implemented features. Suitable for internal use; additional documentation needed before public release.
+
+---
 
-The existing 214 tests provide good confidence in implemented features. Suitable for internal use; additional documentation needed before public release.
+## Changelog
+
+### 2026-02-27
+- Added `call_workflow` support inside `parallel` blocks (inline synchronous execution)
+- Child workflows in parallel execute synchronously with process state save/restore
+- 3 new tests for parallel call_workflow (total: ~291)
+- Updated guides/orchestration.md, guides/parallel.md, and README.md
+- Added workflow orchestration: `call_workflow/3` (synchronous) and `start_workflow/3` (fire-and-forget)
+- Added `Durable.Orchestration` module with `use Durable.Orchestration` macro
+- Added cascade cancellation (cancelling parent cancels active children)
+- Added parent notification on child completion/failure
+- Added `Durable.list_children/2` API
+- Added `guides/orchestration.md` documentation
+- 12 new tests for orchestration
+
+### 2026-01-23
+- Removed `foreach` primitive (use `Enum.map` or `Task.async_stream` instead)
+- Updated parallel execution with new results model (`__results__`, `into:`, `returns:`)
+- Updated documentation in `guides/parallel.md`
+- Archived `IMPLEMENTATION_PLAN.md` (now `IMPLEMENTATION_PLAN_ARCHIVED.md`)