Skip to content

keep a reasonable history events order when time skipping happens#10912

Open
feiyang3cat wants to merge 3 commits into
temporalio:mainfrom
feiyang3cat:vts-fix-event-order
Open

keep a reasonable history events order when time skipping happens#10912
feiyang3cat wants to merge 3 commits into
temporalio:mainfrom
feiyang3cat:vts-fix-event-order

Conversation

@feiyang3cat

@feiyang3cat feiyang3cat commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

What changed?

change the time-skipping-transitioned event to be a buffered event

Why?

Currently, the time-skipping-transitioned event is not buffered, while the update-workflow-options event is. This inconsistency leads to two minor (non-critical) issues that are still worth addressing:

  • The workflow history can show time-skipping-transitioned events (order) before time skipping has actually been enabled.
  • When the fast-forward timer fires to disable time skipping while a workflow task has been scheduled but not yet completed, the time-skipping-transitioned event can appear within the workflow task window. This does not match the existing event ordering pattern.

Buffering the time-skipping-transitioned event resolves both issues while preserving the existing time-skipping behavior and logic.

How did you test it?

  • built
  • run locally and tested manually
  • covered by existing tests
  • added new unit test(s)
  • added new functional test(s)

@feiyang3cat feiyang3cat requested review from a team as code owners July 3, 2026 02:25
@feiyang3cat feiyang3cat force-pushed the vts-fix-event-order branch from 8469444 to 6a2e8b0 Compare July 3, 2026 02:27
@feiyang3cat feiyang3cat changed the title fix event order fix event order when time skipping happens Jul 3, 2026
@feiyang3cat feiyang3cat force-pushed the vts-fix-event-order branch from 6a2e8b0 to 0ac2732 Compare July 5, 2026 00:44
@feiyang3cat feiyang3cat changed the title fix event order when time skipping happens fix history events order when time skipping happens Jul 5, 2026
@feiyang3cat feiyang3cat force-pushed the vts-fix-event-order branch 3 times, most recently from c1f77ac to 895243e Compare July 5, 2026 01:13
case // Buffer this event to ensure:
// 1) The transition is emitted after WORKFLOW_EXECUTION_OPTIONS_UPDATED in the
// same transaction, so history reflects when time skipping actually becomes enabled.
// 2) The transition is not inserted between workflow task events when the

@feiyang3cat feiyang3cat Jul 5, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually think both are trivial OR maybe 2) is a serious problem for some history reasons?

…saction

A time-skipping transition must be the last event of the transaction that
produces it: it is a server-generated state change applied synchronously (it
bumps AccumulatedSkippedDuration and drives timer-task regeneration in the same
close-transaction), so it must order after every other event in that
transaction.

WORKFLOW_EXECUTION_OPTIONS_UPDATED is a buffered event type while the transition
previously was not. When an options update re-enables (or enables) time skipping
on an idle workflow with a pending timer, the close-transaction emits a
transition in the same transaction as the buffered OPTIONS_UPDATED. The
non-buffered transition was allocated an event ID immediately, before the
buffered OPTIONS_UPDATED was flushed in closeTransactionPrepareEvents, inverting
their order and producing non-monotonic event timestamps.

Make the transition a buffered event instead. Buffered events drain at the
transaction's end-of-close flush (Finish -> FlushBufferToCurrentBatch), appended
after every non-buffered event already in the batch and in insertion order among
themselves; reorderBuffer only moves activity/child/nexus completion events to
the end, and those cannot exist on a skippable (idle) workflow. So the
transition, added last, flushes last — OPTIONS_UPDATED then transition — with
monotonic IDs and timestamps, and no explicit FlushBufferedEvents call injected
into time-skipping logic.

Buffering also fixes the fast-forward corner (executeTimeSkippingTimerTask firing
while a workflow task is started): the non-buffered transition used to be wedged
inside the open WorkflowTaskStarted->WorkflowTaskCompleted window; as a buffered
event it instead follows standard semantics (the in-flight WT is failed with
UnhandledCommand and retried, so the worker processes the transition). The event
is never dropped: it only lingers in the buffer while a WT is started, and every
workflow-finishing path (worker complete via UnhandledCommand, terminate/timeout
via failWorkflowTask) flushes buffered events before writing the finish event.

Add TestTransitionEventIsLastInTransaction to TimeSkippingTestSuite guarding the
invariant.
@feiyang3cat feiyang3cat force-pushed the vts-fix-event-order branch from 895243e to 6891f33 Compare July 5, 2026 01:27
@feiyang3cat feiyang3cat force-pushed the vts-fix-event-order branch from 6891f33 to b985926 Compare July 5, 2026 02:04
@feiyang3cat feiyang3cat changed the title fix history events order when time skipping happens keep a reasonable history events order when time skipping happens Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant