Skip to content

Support multi-pulse ev44 messages #708

@SimonHeybrock

Description

@SimonHeybrock

Problem

_require_single_pulse in src/ess/livedata/handlers/to_nxevent_data.py raises NotImplementedError when an ev44 message contains events from multiple pulses. The ev44 schema supports packing events from several pulses into a single message via reference_time (array of pulse timestamps) and reference_time_index (array of starting indices for each pulse's events within the flat event arrays).

This appears to be a leftover simplification from early development, when all producers sent single-pulse messages. It will break if an upstream source starts batching pulses.

Inconsistency with fast-path adapter

KafkaToMonitorEventsAdapter (the optimized direct-from-flatbuffer path) already silently ignores multi-pulse structure: it takes the flat TimeOfFlightAsNumpy() array and uses reference_time[-1] as the message timestamp, discarding per-pulse boundaries entirely. So the slow path (via MonitorEvents.from_ev44 / DetectorEvents.from_ev44) raises an error, while the fast path silently drops pulse information. Both behaviors are wrong.

What needs to change

The ev44 multi-pulse layout is:

  • reference_time[i] — timestamp for pulse i
  • reference_time_index[i] — starting index in the event arrays for pulse i
  • Events for pulse i span indices reference_time_index[i] to reference_time_index[i+1] (or end-of-array for the last pulse)

Each pulse should produce a separate events object with its own reference_time used as the event_time_zero timestamp in the downstream accumulator.

Implementation sketch

  • from_ev44 returns a list of events objects (one per pulse), slicing the flat arrays using reference_time_index as boundaries
  • The MessageAdapter protocol and AdaptingMessageSource need to support 1:N message expansion (one ev44 → N messages), or the expansion can be handled at the adapter level
  • KafkaToMonitorEventsAdapter needs the same treatment
  • The change is mechanically straightforward (NumPy index slicing), but touches several layers: dataclasses, adapters, and the adapting source

Alternatives

1->N message translation might not be what we want. Explore changing the internal detector/monitor message schema, adding reference_time (etc.) fields to them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions