Add filter_list processor to filter list elements by yavmanis · Pull Request #6659 · opensearch-project/data-prepper

yavmanis · 2026-03-20T09:45:16Z

filter_list Processor

The filter_list processor filters elements within an array field by evaluating a condition against each element and retaining only those that satisfy the condition.

It supports:

Arrays of objects
Arrays of primitives (strings, numbers, booleans)

Description

This processor iterates over a specified array field and evaluates an expression for each element.

Element-level filtering is controlled by keep_element_when
Processor execution can be conditionally controlled using filter_list_when

This behavior is aligned with existing processors like delete_entries for consistency.

Configuration Updates

To promote consistency with other processors that iterate over elements:

source → iterate_on
Better reflects that the processor iterates over a collection
keep_when → keep_element_when
Clearly indicates evaluation happens at the element level

Other configuration options remain unchanged.

Configuration Options

Option	Required	Type	Default	Description
iterate_on	Yes	String	—	Key of the array field to iterate and filter (supports nested paths like `outer/inner`)
target	No	String	iterate_on	Key to write filtered output (defaults to in-place filtering)
keep_element_when	Yes	String	—	Expression evaluated per element; elements where true are retained
filter_list_when	No	String	null	Expression evaluated at root event level; processor runs only if true
tags_on_failure	No	List	null	Tags added to event metadata on failure

Execution Semantics

keep_element_when → evaluated in element scope
filter_list_when → evaluated in root event scope
If filter_list_when is false → processor is skipped
If target is not provided → filtering is in-place
If target is provided → filtered result is written to target field

Failure Handling

If evaluation fails for one or more elements:

Event is marked using tags_on_failure
Metadata fields are added:
- filter_list_processor_failed_elements_count
- filter_list_processor_failed_elements

This avoids silent data loss and improves observability.

Example Pipeline Configuration

filter-list-pipeline:
  source:
    file:
      path: "pipelines/input.json"
      format: "json"
      record_type: "event"

  processor:
    - filter_list:
        iterate_on: "items"
        keep_element_when: '/status == "active"'
        filter_list_when: '/env == "production"'

    - filter_list:
        iterate_on: "tags"
        keep_element_when: '/value != ""'
        filter_list_when: '/type == "tagged"'

    - filter_list:
        iterate_on: "scores"
        target: "passing_scores"
        keep_element_when: '/value > 50'
        filter_list_when: '/type == "grades"'

    - filter_list:
        iterate_on: "items"
        target: "critical_items"
        keep_element_when: '/priority == "critical"'
        filter_list_when: '/type == "tickets"'

  sink:
    - stdout:

Input

{"items": [{"name": "item1", "status": "active"}, {"name": "item2", "status": "inactive"}, {"name": "item3", "status": "active"}], "env": "production", "type": "inventory"}
{"tags": ["important", "", "urgent", ""], "type": "tagged"}
{"scores": [95, 30, 75, 10, 88], "subject": "math", "type": "grades"}
{"items": [{"name": "bug1", "priority": "critical"}, {"name": "bug2", "priority": "low"}, {"name": "bug3", "priority": "critical"}], "type": "tickets"}
{"items": [{"name": "item1", "status": "active"}], "env": "staging", "type": "inventory"}

Output

{"items":[{"name":"item1","status":"active"},{"name":"item3","status":"active"}],"env":"production","type":"inventory"}
{"tags":["important","urgent"],"type":"tagged"}
{"scores":[95,30,75,10,88],"subject":"math","type":"grades","passing_scores":[95,75,88]}
{"items":[{"name":"bug1","priority":"critical"},{"name":"bug2","priority":"low"},{"name":"bug3","priority":"critical"}],"type":"tickets","critical_items":[{"name":"bug1","priority":"critical"},{"name":"bug3","priority":"critical"}]}
{"items":[{"name":"item1","status":"active"}],"env":"staging","type":"inventory"}

End-to-End Scenarios

#	Scenario	Result
1	Object array filtered in-place (`status == active`, env=production)	PASS
2	Primitive array removes empty values	PASS
3	Number array filtered into new target (`> 50`)	PASS
4	Object array filtered into new target (`priority == critical`)	PASS
5	Condition not met → no mutation	PASS

Summary

Introduces filter_list processor for array element filtering
Aligns naming with existing processors:
- iterate_on
- keep_element_when
Supports:
- Object and primitive arrays
- In-place and target-based filtering
Adds strong failure visibility with metadata

Signed-off-by: Manisha Yadav <yavmanis@amazon.com>

bagmarnikhil · 2026-03-26T21:30:19Z

+                final JacksonEvent.Builder contextBuilder = JacksonEvent.builder()
+                        .withEventType("event");
+
+                for (final Object element : sourceList) {


Each array element gets wrapped in a full JacksonEvent via builder.withData(contextMap).build(), which internally runs mapper.valueToTree(). For large arrays this creates
significant GC pressure and serialization overhead — the same concern @dlvenable raised in the issue discussion. This matches the pattern used by the translate processor, which has reportedly caused performance issues.

For now this is probably acceptable since the ExpressionEvaluator requires an Event, but worth noting as a known limitation.

For now, we have acknowledged the limitation and mark it as TODO.

There are two problems with the translate processor approach. One is the performance. But the other issue is how we handle relative paths. There is a good discussion in #6609 about this same issue.

Basically, we need a way to decide relative versus absolute paths. Some processors are all absolute and some have relative paths. I propose in #6609 that we should use / to state absolute. And lack of that can be relative. Your elementEvent below will be a relative path with an absolute marker. This is a larger problem because it will lock us into the current behavior until a breaking change.

Thanks for flagging this. We understand this could conflict with a future convention where / explicitly means absolute (as discussed in #6609). We'll keep this in mind and align with whatever convention is adopted at the framework level. For now, this is the only viable approach given the current ExpressionEvaluator API.

@yavmanis , We should avoid pushing this bug further into processors. Using / is clearly indicating a JSON pointer at a root object.

I think these are our options at this point:

Resolve this correctly through JacksonEvent.

Create a throw-away configuration like use_relative_pointer_on_keep_when. The only allowable value would be true and this is not a default value so users must provide it.

- filter_list: source: "items" keep_when: '/status == "active"' filter_list_when: '/env == "production"' use_relative_pointer_on_keep_when: true

We would eventually remove it.

This name helps clarify that filter_list_when isn't using the relative pointer.

@dlvenable Since we implemented this processor from the beginning, we have carefully handled all the relevant conditions, particularly those related to relative JSON pointers. Specifically, the filter_list_when condition resolves against the root-level fields, while the keep_when condition resolves against the element-level fields for each object in the list.

Based on our testing, both conditions are working as expected within their respective contexts. Therefore, we believe that introducing the flag use_relative_pointer_on_keep_when: true may not be necessary for resolving JSON pointers, as this behavior is already managed within the processor implementation.

Additionally, the behavior of both conditions is consistent with the delete_entries processor.

However, if you have a specific use case in mind that is not adequately handled by the current implementation and would require this flag, please let us know. We would be happy to review and address it.

@dlvenable if keep_when syntax stays the same regardless, I'm not sure if adding a flag like this use_relative_pointer_on_keep_when: true is correct yet. Right now, as long as we are consistently saying "keep_when uses local element scope", and "filter_list_when" uses scope from the root of the Event", I think that is pretty clear.

We had a discussion last week about renaming the fields to promote consistency. I suggested in another comment that we use iterate_on and keep_element_when to signal that these have similar behavior to the delete entries processor.

The add entries processor is unfortunately not handling this in a great way currently. But this PR would match delete entries with those name changes.

bagmarnikhil · 2026-03-26T21:32:17Z

+
+                    try {
+                        final Event elementEvent = contextBuilder
+                                .withData(contextMap)


When evaluateConditional throws for a single element, that element is silently dropped from the result. This is a data loss scenario with no signal to the user beyond WARN-level logs

Thanks for calling this out.
To improve visibility for this case, I updated the processor to mark the event as failed (tags_on_failure) whenever evaluateConditional throws for one or more elements, and to attach diagnostic metadata:

filter_list_processor_failed_elements_count

filter_list_processor_failed_elements

This keeps current filtering behavior consistent with similar processors, while ensuring downstream processors and users can clearly detect and inspect element-level evaluation failures instead of relying only on WARN logs.

bagmarnikhil · 2026-03-26T21:34:15Z

+                }
+
+                final List<Object> filteredList = new ArrayList<>();
+                final JacksonEvent.Builder contextBuilder = JacksonEvent.builder()


The JacksonEvent.Builder is created once outside the loop and reused via .withData() on each iteration. This works today because build() consumes the current data reference, but it's fragile — if the builder ever accumulates state across calls, this breaks silently. Safer to create the builder inside the loop

bagmarnikhil

Nice work — clean implementation with thorough test coverage. Two items to address:

when evaluateConditional throws for a single element in the loop (FilterListProcessor.java:103-110), that element is silently dropped from the output — this is a data loss risk, consider failing open or tagging the event;
JacksonEvent.Builder at line 93 is reused across loop iterations which is fragile — safer to create it inside the loop. Rest looks good.

kkondaka · 2026-03-29T00:34:10Z

Looks good too. But it looks like we are creating new processors each of the "list" operations. Maybe we should have a list processor that can filter and that manipulate list items and one of the manipulations could be converting simple list items to map. That way the other PR (#6665) can be functionally merged into one processor that operates on lists?

…behaviour and addressed comments - Add processor-scoped metadata: - filter_list_processor_failed_elements_count - filter_list_processor_failed_elements - Keep per-element filtering behavior unchanged - Add TODO for future optimization of evaluation path overhead Signed-off-by: nishantKadivar <nimahesx@amazon.com>

dlvenable

Thank you @yavmanis for this contribution!

dlvenable · 2026-03-30T13:28:51Z

+        this.expressionEvaluator = expressionEvaluator;
+        this.target = config.getTarget() != null ? config.getTarget() : config.getSource();
+
+        if (config.getFilterListWhen() != null


Put validations in the Config classes themselves.

You can use an @AssertTrue from the Jakarta package. Other configs follow this pattern.

Target defaulting has been moved into FilterListProcessorConfig.getTarget(), which now returns source when target is not set.
Regarding @AssertTrue — the existing usages in this module are all for pure cross-field validations that don't require external dependencies. Our getTarget() never rejects anything — it always succeeds. It's a getter with a fallback default, not a validation. So @AssertTrue doesn't apply here.
Expression validation requires ExpressionEvaluator, which is a runtime dependency not available to config POJOs during Jakarta validation. No config class in the project uses @AssertTrue for expression validation.

dlvenable · 2026-03-30T13:34:40Z

+                final JacksonEvent.Builder contextBuilder = JacksonEvent.builder()
+                        .withEventType("event");
+
+                for (final Object element : sourceList) {


There are two problems with the translate processor approach. One is the performance. But the other issue is how we handle relative paths. There is a good discussion in #6609 about this same issue.

Basically, we need a way to decide relative versus absolute paths. Some processors are all absolute and some have relative paths. I propose in #6609 that we should use / to state absolute. And lack of that can be relative. Your elementEvent below will be a relative path with an absolute marker. This is a larger problem because it will lock us into the current behavior until a breaking change.

…ests Signed-off-by: Manisha Yadav <yavmanis@amazon.com>

github-actions · 2026-04-08T20:49:34Z

✅ License Header Check Passed

All newly added files have proper license headers. Great work! 🎉

yavmanis · 2026-04-09T10:00:32Z

⚠️ License Header Violations Found

The following newly added files are missing required license headers:

data-prepper-plugins/mutate-event-processors/src/test/resources/org/opensearch/dataprepper/plugins/processor/mutateevent/filter_objects_by_status.yaml

data-prepper-plugins/mutate-event-processors/src/test/resources/org/opensearch/dataprepper/plugins/processor/mutateevent/filter_objects_to_target.yaml

data-prepper-plugins/mutate-event-processors/src/test/resources/org/opensearch/dataprepper/plugins/processor/mutateevent/filter_primitives_with_condition.yaml

Please add the appropriate license header to each file and push your changes.

See the license header requirements: https://github.com/opensearch-project/data-prepper/blob/main/CONTRIBUTING.md#license-headers

We do not have headers specifically defined in these YAML files, as we observed that they were not included in any of the existing processor files. This was done to maintain consistency across processors.

Please let us know if it is mandatory to add headers in these files as well.

dlvenable · 2026-04-09T18:17:27Z

@yavmanis , Be sure to also correct the headers.

The last thing I want to do before merging this is to resolve the issue with the relative pointers.

Signed-off-by: Manisha Yadav <yavmanis@amazon.com>

yavmanis · 2026-04-10T09:41:14Z

@yavmanis , Be sure to also correct the headers.

The last thing I want to do before merging this is to resolve the issue with the relative pointers.

I have added the license headers in the yaml files

yavmanis · 2026-04-13T06:40:17Z

As discussed on the call, David will revisit the filter_list processor requirements and the current implementation, and will advise on how he would like to handle the processor’s behavior for condition evaluation at the root versus element level, as well as how this processor should align with the other processors.

We will make the necessary changes in the PR once we receive David’s final guidance on this.

cc: @nishantKadivar @bagmarnikhil

graytaylor0 · 2026-04-15T20:40:23Z

+                    @SuppressWarnings("unchecked")
+                    final Map<String, Object> contextMap = element instanceof Map
+                            ? (Map<String, Object>) element
+                            : Collections.singletonMap("value", element);


From this and the examples in the description it looks like we are using "/value" in the keep_when expression as a special name for primitive lists. Are we sure this is the right approach? At first I was worried this would conflict with actual fields named "value", but this checking for Map condition does handle that conflict. We just need to be very clear in the documentation for keep_when that "/value" is a special keyword for primitive lists.

graytaylor0 · 2026-04-15T20:41:39Z

+                try {
+                    sourceList = recordEvent.get(config.getSource(), List.class);
+                } catch (final Exception e) {
+                    LOG.warn(EVENT, "Given source path [{}] is not valid on record [{}]",


Can you SENSITIVE marker to this log? This is needed if we are logging the entire recordEvent. Also, may be useful having a metric on this error case as well, but not required.

graytaylor0 · 2026-04-15T20:42:12Z

+                recordEvent.put(config.getTarget(), filteredList);
+
+            } catch (final Exception e) {
+                LOG.atError()


Same here logging the full event requires SENSITIVE marker

Added SENSITIVE marker to warning logs that output full event context:

Line 66: LOG.warn(EVENT, SENSITIVE, ...) — when source path is invalid

Line 101: LOG.warn(EVENT, SENSITIVE, ...) — when keep_element_when evaluation fails

graytaylor0 · 2026-04-15T20:42:55Z

+    @JsonPropertyDescription("An expression evaluated per element. Elements where this expression evaluates to true are kept. " +
+            "The expression is evaluated against each element of the array as if it were a standalone event.")
+    @ExampleValues({
+            @Example(value = "/status == \"active\"", description = "Keeps only elements where 'status' equals 'active'."),
+            @Example(value = "/score > 50", description = "Keeps only elements where 'score' is greater than 50.")
+    })


Let's add examples for primitive lists where "/value" is used as keyword, and explain that in the description.

Updated FilterListProcessorConfig @JsonPropertyDescription and @ExampleValues for keep_element_when:

Added explanation: "For primitive elements (strings, numbers, booleans), the element value is accessible via the key /value"

Added examples:

/value > 50 (numeric primitives)

/value != "" (string primitives)

dlvenable · 2026-04-16T09:03:01Z

To promote consistency with other processors that iterate over elements, we can rename source to iterate_on. Data Prepper already has some precedent for using element_when to evaluate against the element and not the whole event. So we could also rename keep_when to keep_element_when. The other configuration options can stay the same.

…-processor # Conflicts: # data-prepper-plugins/mutate-event-processors/README.md

- Rename source → iterate_on, keep_when → keep_element_when - Update FilterListProcessorConfig fields/getters/validation - Update FilterListProcessor to use new getters - Add SENSITIVE marker to warning logs - Add TODO(opensearch-project#6609) reference in processor - Document /value for primitive elements in keep_element_when - Update README examples and configuration docs - Update all tests and YAML fixtures to new key names - Verified: :data-prepper-plugins:mutate-event-processors:test (PASS) Signed-off-by: nishantKadivar <nimahesx@amazon.com>

…ssor

nishantKadivar · 2026-04-17T06:20:10Z

To promote consistency with other processors that iterate over elements, we can rename source to iterate_on. Data Prepper already has some precedent for using element_when to evaluate against the element and not the whole event. So we could also rename keep_when to keep_element_when. The other configuration options can stay the same.

Updated as suggested — renamed source to iterate_on and keep_when to keep_element_when to align with existing Data Prepper conventions. Other configuration options remain unchanged.

dlvenable

Thank you @yavmanis and @nishantKadivar !

bagmarnikhil

Thank you for the change!

yavmanis marked this pull request as draft March 20, 2026 09:45

Add filter_list processor to filter list elements

6460e31

Signed-off-by: Manisha Yadav <yavmanis@amazon.com>

yavmanis force-pushed the feature/filter-list-processor branch from d9bee48 to 6460e31 Compare March 23, 2026 15:23

yavmanis marked this pull request as ready for review March 25, 2026 19:26

yavmanis requested review from KarstenSchnitter, dinujoh, dlvenable, engechas, graytaylor0, kkondaka, oeyh, san81, sb2k16 and srikanthjg as code owners March 25, 2026 19:26

bagmarnikhil reviewed Mar 26, 2026

View reviewed changes

dlvenable requested changes Mar 30, 2026

View reviewed changes

Address review comments: move validation to config, add integration t…

76c0411

…ests Signed-off-by: Manisha Yadav <yavmanis@amazon.com>

yavmanis requested a review from dlvenable April 7, 2026 09:11

Add license headers in the yaml files

324a610

Signed-off-by: Manisha Yadav <yavmanis@amazon.com>

graytaylor0 reviewed Apr 15, 2026

View reviewed changes

yavmanis and others added 3 commits April 16, 2026 10:27

Merge remote-tracking branch 'upstream/main' into feature/filter-list…

1487cc1

…-processor # Conflicts: # data-prepper-plugins/mutate-event-processors/README.md

Merge branch 'opensearch-project:main' into feature/filter-list-proce…

7bc5a03

…ssor

dlvenable approved these changes Apr 17, 2026

View reviewed changes

bagmarnikhil approved these changes Apr 17, 2026

View reviewed changes

dlvenable merged commit 74aa3cf into opensearch-project:main Apr 20, 2026
71 of 72 checks passed

dlvenable mentioned this pull request Apr 20, 2026

[FEATURE] Add filter_entries processor for array element filtering #6610

Closed

Conversation

yavmanis commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

filter_list Processor

Description

Configuration Updates

Configuration Options

Execution Semantics

Failure Handling

Example Pipeline Configuration

Input

Output

End-to-End Scenarios

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yavmanis Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bagmarnikhil left a comment

Choose a reason for hiding this comment

Uh oh!

kkondaka commented Mar 29, 2026

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yavmanis Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ License Header Check Passed

Uh oh!

yavmanis commented Apr 9, 2026

⚠️ License Header Violations Found

Uh oh!

dlvenable commented Apr 9, 2026

Uh oh!

yavmanis commented Apr 10, 2026

Uh oh!

yavmanis commented Apr 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yavmanis commented Mar 20, 2026 •

edited

Loading

yavmanis Apr 13, 2026 •

edited

Loading

yavmanis Mar 31, 2026 •

edited

Loading

github-actions Bot commented Apr 8, 2026 •

edited

Loading