GitHub #60: Workflow Phase 3: Timers, schedules, and deterministic time (#529)

Author: durable-workflow-ops

docs/workflow/plan.md

@@ -336,6 +336,196 @@ Continue-as-new lineage entries are deliberately unaffected; the
 continued run owns the open children and re-emits the relevant child
 history events on its own line.
 
+## Time, Timer, and Schedule Determinism Contract
+
+This section pins the determinism rules for time, timers, timeouts,
+and named schedules. It expands the `Timers and sleep`, `Schedules`,
+and timeout-bearing rows of the Feature Compatibility Matrix and is
+the normative reference for what every implementation must preserve
+so that replay is not coupled to ambient wall-clock time.
+
+### Virtual time
+
+Workflow code reads time through `Workflow::now()` (or the
+namespaced `Workflow\V2\now()` helper). The reading is durable: it
+returns the deterministic event time that the executor seeded onto
+the workflow fiber from the latest history event consumed before
+the resume, not ambient wall-clock time.
+
+The contract pins:
+
+- The first resume of a run reads `workflow_runs.started_at`.
+- After an activity, child, signal, update, side-effect, version,
+  memo, or search-attribute history event drives a resume, the
+  fiber-local time advances to that event's `recorded_at`. Timer
+  resumes use `fired_at`; condition-wait resumes use the resolution
+  event's `recorded_at`.
+- Successful parallel groups advance the fiber-local time to the
+  latest leaf-completion `recorded_at` so a resume after a fan-out
+  reads the most recent observable progress on the workflow line.
+- Outside a workflow fiber (CLI, tests, server jobs, the matching
+  role) `Workflow::now()` falls back to wall-clock `now()` so
+  non-workflow code is not coupled to a fiber-local clock.
+- Workflow code MUST NOT call `Carbon::now()`,
+  `Carbon::setTestNow()`, the global `now()` helper, `microtime()`,
+  `time()`, `date()`, `new DateTime()`, or any other ambient-clock
+  primitive. Replay must produce identical decisions regardless of
+  host wall-clock state.
+
+The exit criterion "replay never depends on ambient `Carbon::now()`"
+is verified by `tests/Feature/V2/V2DeterministicTimeReplayTest.php`,
+which freezes `Carbon::setTestNow()` to a value far from the
+recorded history before replaying the run and asserts the replayed
+`Workflow::now()` reading still matches the seeded event time. The
+matching live-execution behavior is pinned by
+`tests/Feature/V2/V2DeterministicTimeTest.php`, and fiber-local
+seeding is pinned by
+`tests/Unit/V2/WorkflowFiberContextTimeTest.php`.
+
+### Timer lifecycle and supersession
+
+Timers are durable rows in `workflow_run_timers`/`workflow_run_timer_entries`
+with one logical `timer_id` per wait. The lifecycle is:
+
+- **Scheduled.** The run records `TimerScheduled` with the durable
+  `timer_id`, `delay_seconds`, and `fire_at`. A `workflow_tasks`
+  row with `task_type=Timer` and `available_at=fire_at` is created.
+- **Fired.** When the timer task is dispatched the run records
+  `TimerFired` with the same `timer_id`, and the fiber resumes at
+  `fired_at`.
+- **Cancelled.** When workflow code cancels a pending timer (scope
+  rollback, satisfied condition wait, supersession by a later
+  command) the run records `TimerCancelled` with the same
+  `timer_id` and the open `workflow_tasks` row is closed.
+- **Repaired.** When the watchdog recreates a missing timer task
+  from typed timer history, the repaired task preserves the
+  timer's durable `fire_at`. Future timers MUST NOT be snapped to
+  current wall-clock time.
+- **Transport chunking.** `TimerTransportChunker` may split the
+  scheduling/wake hops for very long sleeps that the queue
+  transport cannot represent in one entry, but the durable timer
+  row, the typed history events, and the resume value all keep one
+  stable `timer_id` end-to-end.
+
+Cancel and terminate commands targeting the run cancel every open
+timer, condition timeout, and signal timeout the same way: the run
+records `TimerCancelled` (or the corresponding wait-resolution
+event), the open `workflow_tasks` row is closed, and the fiber
+wakes through the cancellation/termination path rather than the
+timer-fired path.
+
+### Activity timeout taxonomy
+
+Every `activity_executions` row carries the four timeout deadlines
+as durable columns:
+
+- **`schedule_to_start`** — fails the activity if no worker claims
+  the task before the deadline. Retriable per the activity's retry
+  policy.
+- **`start_to_close`** (column `close_deadline_at`) — fails the
+  activity if the running attempt does not return before the
+  deadline. Retriable.
+- **`schedule_to_close`** — fails the activity if the total
+  scheduling+running time crosses the deadline. NOT retriable; the
+  enforcer terminates the activity even when the retry policy still
+  has remaining attempts.
+- **`heartbeat`** — fails the running attempt if no heartbeat is
+  recorded before the deadline. Retriable.
+
+`ActivityTimeoutEnforcer::enforce` is the single durable path that
+turns an expired deadline into a typed `ActivityTimedOut` event with
+the matching `timeout_kind`. The enforcer is covered by
+`tests/Feature/V2/V2ActivityTimeoutTest.php`.
+
+### Workflow execution-timeout, run-timeout, and workflow-task-timeout
+
+Workflow-level timeouts share the same separation:
+
+- **`execution_timeout_seconds`** — limits the total lifetime of a
+  workflow instance across continue-as-new chains. Breach records
+  `WorkflowTimedOut` with `timeout_kind=execution_timeout`.
+- **`run_timeout_seconds`** — limits a single run's lifetime.
+  Continue-as-new resets the run-timeout deadline. Breach records
+  `WorkflowTimedOut` with `timeout_kind=run_timeout`.
+- **`workflow_task_timeout_seconds`** — limits how long a single
+  workflow-task lease may be held by a worker. Lease expiry is
+  reclaimed by the watchdog and does not record a workflow-level
+  terminal event; it surfaces as a worker-plane redelivery.
+
+Both `execution_timeout` and `run_timeout` flow through
+`WorkflowExecutor::timeoutRun` and call
+`ParentClosePolicyEnforcer::enforce` (see the parent-close matrix
+above) so child disposition is consistent across timeout kinds.
+
+### Workflow-level retry first-release contract
+
+Top-level workflow runs do NOT retry on failure or timeout in the
+first v2 release. A run that records `WorkflowFailed` or
+`WorkflowTimedOut` is terminal; restarting a new run is the
+operator's or owner's responsibility. This is the explicit
+"first-release unsupported" contract: the durable engine does not
+expose a top-level workflow retry policy, and adding one is a
+contract change that must be reviewed under the rules in
+"Changing This Contract" below.
+
+Retry policies remain a first-class authoring surface for
+**activities** through `ActivityOptions` (`maxAttempts`, `backoff`,
+`nonRetryableErrorTypes`) and for **child workflows** through the
+`ChildWorkflowRetryPolicy` snapshot carried on the child-start
+command. Both default to no retry unless an explicit policy is set,
+matching the V2.0 default above.
+
+### Named schedule lifecycle
+
+`workflow_schedules` is the durable home for named schedules.
+`ScheduleManager` exposes the full lifecycle:
+
+- `create` / `createFromSpec` — register a schedule with a cron
+  expression, timezone, overlap policy, and visibility.
+- `pause` / `resume` — toggle `ScheduleStatus::Active` and
+  `ScheduleStatus::Paused` without losing schedule history.
+- `update` — change the cron, timezone, overlap policy, or paused
+  state with the next-fire time recomputed deterministically.
+- `trigger` — start a manual run of the schedule's workflow,
+  gated by `ScheduleOverlapPolicy`.
+- `tick` — two-phase fire evaluation: drain the buffered run
+  queue first, then evaluate due fires.
+- `describe` / `findByScheduleId` — read the schedule's
+  projection and audit history.
+- `delete` — mark the schedule `Deleted`; the row remains for
+  audit and is filtered out of evaluation.
+- `backfill` — replay `computeNextFireAt` between two boundary
+  instants to mass-trigger missed fires under the overlap policy.
+
+`ScheduleOverlapPolicy::Skip`, `BufferOne`, `AllowAll`,
+`CancelOther`, and `TerminateOther` are the five supported
+policies. The schedule history events (`ScheduleCreated`,
+`SchedulePaused`, `ScheduleResumed`, `ScheduleUpdated`,
+`ScheduleTriggered`, `ScheduleTriggerSkipped`, `ScheduleDeleted`)
+are the durable audit truth; schedule projections are derived from
+those events. Schedule-fire evaluation runs through the
+`workflow:v2:schedule-tick` artisan command (and the equivalent
+server endpoint), which is covered by
+`tests/Feature/V2/V2ScheduleTest.php`.
+
+### Replay tests for timeout helpers and timer ordering
+
+Replay-test coverage is required and pinned for:
+
+- Workflow timer ordering across parallel and sequential branches
+  (`tests/Feature/V2/V2GoldenHistoryReplayTest.php`,
+  `tests/Feature/V2/V2WorkflowReplayerTest.php`).
+- Activity timeout helpers
+  (`tests/Feature/V2/V2ActivityTimeoutTest.php`).
+- Deterministic time across live execution
+  (`tests/Feature/V2/V2DeterministicTimeTest.php`) and across
+  replay (`tests/Feature/V2/V2DeterministicTimeReplayTest.php`).
+- Schedule lifecycle
+  (`tests/Feature/V2/V2ScheduleTest.php`).
+
+Adding a new time-, timer-, or schedule-affecting code path that
+does not extend one of these test buckets is a release-blocker.
+
 ## Gap Analysis
 
 | Item | Classification | Decision |

tests/Feature/V2/V2DeterministicTimeReplayTest.php

@@ -0,0 +1,143 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Tests\Feature\V2;
+
+use Illuminate\Support\Carbon;
+use Illuminate\Support\Facades\Queue;
+use Tests\Fixtures\V2\TestReplayDeterministicTimeWorkflow;
+use Tests\TestCase;
+use Workflow\V2\Enums\HistoryEventType;
+use Workflow\V2\Enums\TaskStatus;
+use Workflow\V2\Enums\TaskType;
+use Workflow\V2\Jobs\RunActivityTask;
+use Workflow\V2\Jobs\RunWorkflowTask;
+use Workflow\V2\Models\WorkflowHistoryEvent;
+use Workflow\V2\Models\WorkflowRun;
+use Workflow\V2\Models\WorkflowTask;
+use Workflow\V2\Support\WorkflowReplayer;
+use Workflow\V2\WorkflowStub;
+
+/**
+ * Pins the Phase 3 exit criterion: replay never depends on ambient
+ * Carbon::now(). The test runs a workflow to completion at one frozen
+ * clock, then replays it under a wildly different ambient clock and
+ * asserts that `Workflow::now()` (Workflow\V2\now()) reads the
+ * deterministic event time the executor seeded — not the ambient
+ * Carbon::setTestNow() value.
+ */
+final class V2DeterministicTimeReplayTest extends TestCase
+{
+    protected function setUp(): void
+    {
+        parent::setUp();
+
+        config()
+            ->set('queue.default', 'redis');
+        config()
+            ->set('queue.connections.redis.driver', 'redis');
+        config()
+            ->set('workflows.v2.task_dispatch_mode', 'poll');
+        Queue::fake();
+    }
+
+    public function testReplayReadsSeededEventTimeAndIgnoresAmbientWallClock(): void
+    {
+        $startedAt = Carbon::parse('2026-02-01T12:00:00Z');
+        Carbon::setTestNow($startedAt);
+
+        $workflow = WorkflowStub::make(
+            TestReplayDeterministicTimeWorkflow::class,
+            'replay-deterministic-time-1',
+        );
+        $workflow->start('Taylor');
+
+        $this->runReadyTaskOfType(TaskType::Workflow);
+
+        $activityRecordedAt = $startedAt->copy()
+            ->addMinutes(5);
+        Carbon::setTestNow($activityRecordedAt);
+
+        $this->runReadyTaskOfType(TaskType::Activity);
+        $this->runReadyTaskOfType(TaskType::Workflow);
+
+        Carbon::setTestNow(null);
+
+        $this->assertTrue($workflow->refresh()->completed());
+
+        /** @var WorkflowRun $run */
+        $run = WorkflowRun::query()
+            ->findOrFail($workflow->runId());
+
+        /** @var WorkflowHistoryEvent $activityCompleted */
+        $activityCompleted = WorkflowHistoryEvent::query()
+            ->where('workflow_run_id', $run->id)
+            ->where('event_type', HistoryEventType::ActivityCompleted->value)
+            ->orderBy('sequence')
+            ->firstOrFail();
+
+        $expectedStartMs = $run->started_at?->getTimestampMs();
+        $expectedAfterActivityMs = $activityCompleted->recorded_at?->getTimestampMs();
+
+        $this->assertNotNull($expectedStartMs);
+        $this->assertNotNull($expectedAfterActivityMs);
+
+        $ambientFuture = Carbon::parse('2099-12-31T23:59:59Z');
+        Carbon::setTestNow($ambientFuture);
+
+        try {
+            $state = (new WorkflowReplayer())->replay($run);
+        } finally {
+            Carbon::setTestNow(null);
+        }
+
+        $this->assertInstanceOf(TestReplayDeterministicTimeWorkflow::class, $state->workflow);
+
+        $this->assertSame(
+            $expectedStartMs,
+            $state->workflow->observedStartMs,
+            'Replay must read the run started_at from history when sampling Workflow::now() at handle entry, even when Carbon::setTestNow() is set far in the future.',
+        );
+
+        $this->assertSame(
+            $expectedAfterActivityMs,
+            $state->workflow->observedAfterActivityMs,
+            'Replay must read the ActivityCompleted recorded_at from history when sampling Workflow::now() after the activity, even when Carbon::setTestNow() is set far in the future.',
+        );
+
+        $this->assertNotSame(
+            $ambientFuture->getTimestampMs(),
+            $state->workflow->observedStartMs,
+            'Replay must not read ambient wall-clock time at handle entry.',
+        );
+
+        $this->assertNotSame(
+            $ambientFuture->getTimestampMs(),
+            $state->workflow->observedAfterActivityMs,
+            'Replay must not read ambient wall-clock time after an activity completes.',
+        );
+    }
+
+    private function runReadyTaskOfType(TaskType $taskType): void
+    {
+        /** @var WorkflowTask|null $task */
+        $task = WorkflowTask::query()
+            ->where('status', TaskStatus::Ready->value)
+            ->where('task_type', $taskType->value)
+            ->orderBy('created_at')
+            ->first();
+
+        if (! $task instanceof WorkflowTask) {
+            $this->fail("Expected a ready task of type {$taskType->value}.");
+        }
+
+        $job = match ($task->task_type) {
+            TaskType::Workflow => new RunWorkflowTask($task->id),
+            TaskType::Activity => new RunActivityTask($task->id),
+            default => $this->fail("Unsupported task type {$task->task_type->value}."),
+        };
+
+        $this->app->call([$job, 'handle']);
+    }
+}

tests/Fixtures/V2/TestReplayDeterministicTimeWorkflow.php

@@ -0,0 +1,42 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Tests\Fixtures\V2;
+
+use function Workflow\V2\activity;
+use function Workflow\V2\now;
+
+use Workflow\V2\Workflow;
+
+/**
+ * Fixture used by V2DeterministicTimeReplayTest to verify that
+ * Workflow::now() during replay returns the deterministic event time
+ * seeded by the executor — not ambient Carbon::now() / wall-clock.
+ *
+ * The observed reads are stored on instance properties so the test can
+ * inspect them on the workflow returned from WorkflowReplayer::replay().
+ */
+final class TestReplayDeterministicTimeWorkflow extends Workflow
+{
+    public ?int $observedStartMs = null;
+
+    public ?int $observedAfterActivityMs = null;
+
+    public function handle(string $name): array
+    {
+        $timeAtStart = now();
+        $this->observedStartMs = $timeAtStart->getTimestampMs();
+
+        $greeting = activity(TestGreetingActivity::class, $name);
+
+        $timeAfterActivity = now();
+        $this->observedAfterActivityMs = $timeAfterActivity->getTimestampMs();
+
+        return [
+            'greeting' => $greeting,
+            'time_at_start_ms' => $timeAtStart->getTimestampMs(),
+            'time_after_activity_ms' => $timeAfterActivity->getTimestampMs(),
+        ];
+    }
+}