Is your feature request related to a problem? Please describe.
Yes. In multi-tenant logging pipelines, a single pipeline often ingests events from many different sources. Some sources send multi-line log bodies concatenated with \n while others send single-line bodies that should not be split.
Currently, split_event runs unconditionally on every event that passes through the processor. There is no way to selectively apply the split based on event content or metadata. This means:
- Events that don't contain the delimiter still incur processing overhead.
- There is no way to conditionally skip splitting for specific tenants, log formats, or event shapes without duplicating pipelines or adding complex routing.
Describe the solution you'd like
Add a split_when parameter to the split_event processor, consistent with the conditional expression pattern already used by other processors (e.g., key_value_when, parse_when, add_when).
Proposed configuration
- split_event:
field: "body"
delimiter: "\\n"
split_when: 'contains(/body, "\\n")'
| Option |
Type |
Description |
split_when |
String |
A conditional expression (using Data Prepper expression syntax) that determines whether the processor should be applied to the event. If the condition evaluates to false, the event passes through unchanged. |
Describe alternatives you've considered (Optional)
-
Separate pipelines per tenant/log format - This leads to pipeline sprawl and duplicated sink configurations. Not scalable when handling dozens of tenants.
-
Placing split_event after routing via sub-pipelines - Adds complexity and latency. Requires maintaining additional pipeline definitions solely to gate a single processor.
-
Relying on downstream processors to handle unsplit events - Works in some cases but shifts complexity downstream and can cause parsing failures or unexpected behavior when processors receive concatenated multi-line bodies.
Additional context
The conditional expression pattern (*_when) is already established across multiple Data Prepper processors:
Adding split_when to split_event would maintain consistency with this pattern and make the processor usable in shared multi-tenant pipelines without requiring complex routing logic.
Is your feature request related to a problem? Please describe.
Yes. In multi-tenant logging pipelines, a single pipeline often ingests events from many different sources. Some sources send multi-line log bodies concatenated with
\nwhile others send single-line bodies that should not be split.Currently,
split_eventruns unconditionally on every event that passes through the processor. There is no way to selectively apply the split based on event content or metadata. This means:Describe the solution you'd like
Add a
split_whenparameter to thesplit_eventprocessor, consistent with the conditional expression pattern already used by other processors (e.g.,key_value_when,parse_when,add_when).Proposed configuration
split_whenfalse, the event passes through unchanged.Describe alternatives you've considered (Optional)
Separate pipelines per tenant/log format - This leads to pipeline sprawl and duplicated sink configurations. Not scalable when handling dozens of tenants.
Placing
split_eventafter routing via sub-pipelines - Adds complexity and latency. Requires maintaining additional pipeline definitions solely to gate a single processor.Relying on downstream processors to handle unsplit events - Works in some cases but shifts complexity downstream and can cause parsing failures or unexpected behavior when processors receive concatenated multi-line bodies.
Additional context
The conditional expression pattern (
*_when) is already established across multiple Data Prepper processors:Adding
split_whentosplit_eventwould maintain consistency with this pattern and make the processor usable in shared multi-tenant pipelines without requiring complex routing logic.