Skip to content

Commit 7de87ac

Browse files
feat(runtime): journal-backed replay through subscribe/poll (#659) (#709)
Complete the final slice of #659: DurableBroker's subscribe/poll now fall back to the durable journal when a cursor predates the inner broker's in-memory retention window, with cursor_expired + oldest- available-cursor semantics carried through end to end and the identical subject_id filter applied to both live delivery and durable replay (spec 066 FR-003, FR-005, FR-007, FR-008). - EventBroker gains two additive, default-no-op methods: publish_with_cursor (lets a decorator inject a pre-assigned cursor instead of self-incrementing) and seed_restart_floor (lets a decorator establish a floor below which any cursor for an unobserved event type is treated as potentially predating this broker instance). InProcessBroker overrides both; every other EventBroker implementor is unaffected. - DurableBroker::publish now calls publish_with_cursor with the journal-assigned cursor, unifying the live and durable cursor spaces so a cursor obtained during live polling remains valid when later resolved through durable replay. - New JournalSource trait (replay_from) shared via Arc<RwLock<DurableEventJournal>> between the writer thread and the read path; DurableBroker::open is the new production constructor that wires sink and source to the same underlying journal storage. DurableBroker::new gains a `source` parameter for tests that need independent write/read doubles. - subscribe_for_subject tries the inner broker first; on CursorExpired it checks the journal and either creates a durable-mode subscription (own subscription table, "durable-sub-*" ids) or surfaces the journal's own CursorExpired. poll routes by id prefix: durable-mode subscriptions loop over replay_from (paging internally on partial batches) applying the same event_type/ subject_id filtering as live delivery; other ids delegate to the inner broker unchanged. - Fixed a real restart-continuity gap: a freshly constructed inner broker has no history for an event type it has never seen, so it optimistically accepted any cursor as non-expired. DurableBroker:: open now seeds the inner broker's restart floor from the journal's latest cursor at construction, so a resumed cursor from before a process restart is correctly deferred to durable replay instead of silently treated as caught-up. - 25 new tests (63 total in the events module, all at 100% line coverage for traverse-runtime): the fallback cascade, identical subject filtering through fallback, double-expiry surfacing the journal's oldest-available cursor, non-cursor journal read-error propagation (Io/InvalidCursor/Corrupt), paging and multi-batch scanning, cross-type filtering, real full-broker restart continuity, and a production SharedJournalSink revocation exercised through DurableBroker::open rather than a scripted test double. Closes #659 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent c1a9e78 commit 7de87ac

4 files changed

Lines changed: 1494 additions & 109 deletions

File tree

crates/traverse-runtime/src/events/broker.rs

Lines changed: 227 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,8 @@ struct BrokerState {
6969
buffers: HashMap<String, VecDeque<BufferedEvent>>,
7070
seen_event_ids: HashMap<String, HashSet<String>>,
7171
subscriptions: HashMap<SubscriptionId, SubscriptionState>,
72+
/// See [`EventBroker::seed_restart_floor`].
73+
restart_floor: u64,
7274
}
7375

7476
/// Synchronous, in-memory implementation of [`EventBroker`].
@@ -182,6 +184,104 @@ impl InProcessBroker {
182184
cursor: cursor_to_string(from_cursor),
183185
})
184186
}
187+
188+
/// Shared implementation for `publish` and `publish_with_cursor`.
189+
/// `assigned_cursor`, when given, is adopted as this event's cursor
190+
/// instead of self-incrementing the per-type counter (spec 066 FR-007:
191+
/// keeps live-delivery cursors consistent with the durable journal's
192+
/// cursor space). The per-type counter still tracks the highest cursor
193+
/// ever used so `validate_from_cursor`'s empty-buffer fallback remains
194+
/// correct regardless of cursor source.
195+
fn publish_internal(
196+
&self,
197+
event: &TraverseEvent,
198+
assigned_cursor: Option<u64>,
199+
) -> Result<(), EventError> {
200+
let entry = self
201+
.catalog
202+
.get(&event.event_type)
203+
.ok_or_else(|| EventError::UnregisteredEventType(event.event_type.clone()))?;
204+
205+
match entry.lifecycle_status {
206+
LifecycleStatus::Active => {}
207+
LifecycleStatus::Deprecated => {
208+
return Err(EventError::LifecycleViolation(format!(
209+
"event type '{}' is Deprecated and cannot be published",
210+
event.event_type
211+
)));
212+
}
213+
LifecycleStatus::Draft => {
214+
return Err(EventError::LifecycleViolation(format!(
215+
"event type '{}' is Draft and cannot be published",
216+
event.event_type
217+
)));
218+
}
219+
}
220+
221+
let now = self.clock.now();
222+
223+
let mut state = self
224+
.state
225+
.lock()
226+
.map_err(|_| EventError::LifecycleViolation("broker lock poisoned".to_owned()))?;
227+
228+
prune_expired(
229+
&mut state,
230+
&event.event_type,
231+
self.config.retention_window,
232+
now,
233+
);
234+
235+
let seen = state
236+
.seen_event_ids
237+
.entry(event.event_type.clone())
238+
.or_default();
239+
if seen.contains(&event.id) {
240+
// Duplicate emissions are silently discarded.
241+
return Ok(());
242+
}
243+
seen.insert(event.id.clone());
244+
245+
let next = state
246+
.next_cursor
247+
.entry(event.event_type.clone())
248+
.or_insert(0);
249+
let cursor = if let Some(assigned) = assigned_cursor {
250+
*next = (*next).max(assigned);
251+
assigned
252+
} else {
253+
*next = next.saturating_add(1);
254+
*next
255+
};
256+
257+
let buffered = BufferedEvent {
258+
cursor,
259+
published_at: now,
260+
event: event.clone(),
261+
};
262+
263+
state
264+
.buffers
265+
.entry(event.event_type.clone())
266+
.or_default()
267+
.push_back(buffered.clone());
268+
269+
for sub in state.subscriptions.values_mut() {
270+
if sub.event_type != event.event_type {
271+
continue;
272+
}
273+
if sub
274+
.subject_id
275+
.as_deref()
276+
.is_some_and(|subject_id| event.subject_id.as_deref() != Some(subject_id))
277+
{
278+
continue;
279+
}
280+
enqueue_with_drop_oldest(&mut sub.queue, self.config.max_queue_len, buffered.clone());
281+
}
282+
283+
Ok(())
284+
}
185285
}
186286

187287
fn parse_cursor(raw: &str) -> Result<u64, EventError> {
@@ -263,7 +363,12 @@ fn validate_from_cursor(
263363
return Ok(());
264364
}
265365

266-
let last_cursor = state.next_cursor.get(event_type).copied().unwrap_or(0);
366+
let last_cursor = state
367+
.next_cursor
368+
.get(event_type)
369+
.copied()
370+
.unwrap_or(0)
371+
.max(state.restart_floor);
267372
if let Some(buffer) = state.buffers.get(event_type)
268373
&& let Some(front) = buffer.front()
269374
{
@@ -299,92 +404,36 @@ impl EventBroker for InProcessBroker {
299404
self.subscribe_with_subject(event_type, from_cursor, subject_id)
300405
}
301406

407+
fn seed_restart_floor(&self, floor: u64) {
408+
if let Ok(mut state) = self.state.lock() {
409+
state.restart_floor = state.restart_floor.max(floor);
410+
}
411+
}
412+
302413
/// Publish `event` to all registered subscribers.
303414
///
304415
/// # Errors
305416
///
306417
/// - [`EventError::UnregisteredEventType`] if the event type is not in the catalog.
307418
/// - [`EventError::LifecycleViolation`] if the catalog entry is `Draft` or `Deprecated`.
308419
fn publish(&self, event: TraverseEvent) -> Result<(), EventError> {
309-
let entry = self
310-
.catalog
311-
.get(&event.event_type)
312-
.ok_or_else(|| EventError::UnregisteredEventType(event.event_type.clone()))?;
313-
314-
match entry.lifecycle_status {
315-
LifecycleStatus::Active => {}
316-
LifecycleStatus::Deprecated => {
317-
return Err(EventError::LifecycleViolation(format!(
318-
"event type '{}' is Deprecated and cannot be published",
319-
event.event_type
320-
)));
321-
}
322-
LifecycleStatus::Draft => {
323-
return Err(EventError::LifecycleViolation(format!(
324-
"event type '{}' is Draft and cannot be published",
325-
event.event_type
326-
)));
327-
}
328-
}
329-
330-
let now = self.clock.now();
331-
332-
let mut state = self
333-
.state
334-
.lock()
335-
.map_err(|_| EventError::LifecycleViolation("broker lock poisoned".to_owned()))?;
336-
337-
prune_expired(
338-
&mut state,
339-
&event.event_type,
340-
self.config.retention_window,
341-
now,
342-
);
343-
344-
let seen = state
345-
.seen_event_ids
346-
.entry(event.event_type.clone())
347-
.or_default();
348-
if seen.contains(&event.id) {
349-
// Duplicate emissions are silently discarded.
350-
return Ok(());
351-
}
352-
seen.insert(event.id.clone());
353-
354-
let next = state
355-
.next_cursor
356-
.entry(event.event_type.clone())
357-
.or_insert(0);
358-
*next = next.saturating_add(1);
359-
let cursor = *next;
360-
361-
let buffered = BufferedEvent {
362-
cursor,
363-
published_at: now,
364-
event: event.clone(),
365-
};
366-
367-
state
368-
.buffers
369-
.entry(event.event_type.clone())
370-
.or_default()
371-
.push_back(buffered.clone());
372-
373-
for sub in state.subscriptions.values_mut() {
374-
if sub.event_type != event.event_type {
375-
continue;
376-
}
377-
if sub
378-
.subject_id
379-
.as_deref()
380-
.is_some_and(|subject_id| event.subject_id.as_deref() != Some(subject_id))
381-
{
382-
continue;
383-
}
384-
enqueue_with_drop_oldest(&mut sub.queue, self.config.max_queue_len, buffered.clone());
385-
}
420+
self.publish_internal(&event, None)
421+
}
386422

387-
Ok(())
423+
/// Publish `event`, adopting `cursor` as this broker's own cursor for it
424+
/// instead of self-assigning the next per-type sequence value. Used by
425+
/// [`DurableBroker`](super::durable::DurableBroker) so live-delivery
426+
/// cursors stay numerically consistent with the durable journal's cursor
427+
/// space (spec 066 FR-007): a cursor obtained during live polling
428+
/// remains valid when a later `poll` falls back to durable replay.
429+
///
430+
/// # Errors
431+
///
432+
/// Returns [`EventError::InvalidCursor`] when `cursor` is not a base-10
433+
/// unsigned integer, plus the same errors as [`Self::publish`].
434+
fn publish_with_cursor(&self, event: TraverseEvent, cursor: &str) -> Result<(), EventError> {
435+
let assigned = parse_cursor(cursor)?;
436+
self.publish_internal(&event, Some(assigned))
388437
}
389438

390439
/// Create a subscription for `event_type` starting from `from_cursor`.
@@ -571,6 +620,105 @@ mod tests {
571620
assert!(matches!(err, EventError::InvalidRetentionWindow(_)));
572621
}
573622

623+
#[test]
624+
fn publish_with_cursor_adopts_the_given_cursor_instead_of_self_assigning() {
625+
let event_type = "dev.traverse.injected-cursor";
626+
let catalog = make_catalog(event_type, LifecycleStatus::Active);
627+
let broker = InProcessBroker::new(catalog).expect("broker must be created");
628+
629+
broker
630+
.publish_with_cursor(sample_event(event_type, "evt-1"), "42")
631+
.expect("publish_with_cursor must succeed");
632+
633+
let subscription = broker
634+
.subscribe(event_type, "0")
635+
.expect("subscribe must succeed");
636+
let poll = broker
637+
.poll(&subscription.subscription_id, 10)
638+
.expect("poll must succeed");
639+
assert_eq!(poll.events.len(), 1);
640+
assert_eq!(poll.events[0].cursor, "42");
641+
assert_eq!(poll.cursor, "42");
642+
}
643+
644+
#[test]
645+
fn publish_with_cursor_rejects_a_malformed_cursor() {
646+
let event_type = "dev.traverse.injected-cursor-invalid";
647+
let catalog = make_catalog(event_type, LifecycleStatus::Active);
648+
let broker = InProcessBroker::new(catalog).expect("broker must be created");
649+
650+
let err = broker
651+
.publish_with_cursor(sample_event(event_type, "evt-1"), "not-a-cursor")
652+
.expect_err("malformed cursor must be rejected");
653+
assert!(matches!(err, EventError::InvalidCursor(_)));
654+
}
655+
656+
#[test]
657+
fn default_publish_with_cursor_ignores_the_cursor_and_self_assigns() {
658+
struct SelfAssigningOnlyBroker(InProcessBroker);
659+
660+
impl EventBroker for SelfAssigningOnlyBroker {
661+
fn publish(&self, event: TraverseEvent) -> Result<(), EventError> {
662+
self.0.publish(event)
663+
}
664+
fn subscribe(
665+
&self,
666+
event_type: &str,
667+
from_cursor: &str,
668+
) -> Result<Subscription, EventError> {
669+
self.0.subscribe(event_type, from_cursor)
670+
}
671+
fn subscribe_for_subject(
672+
&self,
673+
event_type: &str,
674+
from_cursor: &str,
675+
subject_id: Option<&str>,
676+
) -> Result<Subscription, EventError> {
677+
self.0
678+
.subscribe_for_subject(event_type, from_cursor, subject_id)
679+
}
680+
fn poll(
681+
&self,
682+
subscription_id: &str,
683+
max_events: usize,
684+
) -> Result<SubscriptionPoll, EventError> {
685+
self.0.poll(subscription_id, max_events)
686+
}
687+
fn cancel(&self, subscription_id: &str) -> Result<(), EventError> {
688+
self.0.cancel(subscription_id)
689+
}
690+
}
691+
692+
let event_type = "dev.traverse.default-publish-with-cursor";
693+
let catalog = make_catalog(event_type, LifecycleStatus::Active);
694+
let broker =
695+
SelfAssigningOnlyBroker(InProcessBroker::new(catalog).expect("broker must be created"));
696+
697+
// The default `publish_with_cursor` ignores the supplied cursor and
698+
// behaves exactly like `publish` (self-assigning cursor "1").
699+
broker
700+
.publish_with_cursor(sample_event(event_type, "evt-1"), "999")
701+
.expect("default publish_with_cursor must succeed");
702+
703+
let subscription = broker
704+
.subscribe(event_type, "0")
705+
.expect("subscribe must succeed");
706+
let poll = broker
707+
.poll(&subscription.subscription_id, 10)
708+
.expect("poll must succeed");
709+
assert_eq!(poll.events[0].cursor, "1");
710+
711+
// The default `seed_restart_floor` is a no-op; exercise it alongside
712+
// this wrapper's other pass-through trait methods.
713+
broker.seed_restart_floor(999);
714+
let subject_subscription = broker
715+
.subscribe_for_subject(event_type, "0", None)
716+
.expect("subscribe_for_subject must succeed");
717+
broker
718+
.cancel(&subject_subscription.subscription_id)
719+
.expect("cancel must succeed");
720+
}
721+
574722
#[test]
575723
fn invalid_cursor_is_rejected() {
576724
let catalog = make_catalog("dev.traverse.cursor", LifecycleStatus::Active);

0 commit comments

Comments
 (0)