Skip to content

Add split_when conditional parameter to split_event processor #6992

Description

@alamzeeshan

Is your feature request related to a problem? Please describe.
Yes. In multi-tenant logging pipelines, a single pipeline often ingests events from many different sources. Some sources send multi-line log bodies concatenated with \n while others send single-line bodies that should not be split.

Currently, split_event runs unconditionally on every event that passes through the processor. There is no way to selectively apply the split based on event content or metadata. This means:

  • Events that don't contain the delimiter still incur processing overhead.
  • There is no way to conditionally skip splitting for specific tenants, log formats, or event shapes without duplicating pipelines or adding complex routing.

Describe the solution you'd like

Add a split_when parameter to the split_event processor, consistent with the conditional expression pattern already used by other processors (e.g., key_value_when, parse_when, add_when).

Proposed configuration

- split_event:
    field: "body"
    delimiter: "\\n"
    split_when: 'contains(/body, "\\n")'
Option Type Description
split_when String A conditional expression (using Data Prepper expression syntax) that determines whether the processor should be applied to the event. If the condition evaluates to false, the event passes through unchanged.

Describe alternatives you've considered (Optional)

  1. Separate pipelines per tenant/log format - This leads to pipeline sprawl and duplicated sink configurations. Not scalable when handling dozens of tenants.

  2. Placing split_event after routing via sub-pipelines - Adds complexity and latency. Requires maintaining additional pipeline definitions solely to gate a single processor.

  3. Relying on downstream processors to handle unsplit events - Works in some cases but shifts complexity downstream and can cause parsing failures or unexpected behavior when processors receive concatenated multi-line bodies.

Additional context
The conditional expression pattern (*_when) is already established across multiple Data Prepper processors:

Adding split_when to split_event would maintain consistency with this pattern and make the processor usable in shared multi-tenant pipelines without requiring complex routing logic.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Unplanned

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions