Each array element gets wrapped in a full JacksonEvent via builder.withData(contextMap).build(), which internally runs mapper.valueToTree(). For large arrays this creates
significant GC pressure and serialization overhead — the same concern @dlvenable raised in the issue discussion. This matches the pattern used by the translate processor, which has reportedly caused performance issues.

For now this is probably acceptable since the ExpressionEvaluator requires an Event, but worth noting as a known limitation.

For now, we have acknowledged the limitation and mark it as TODO.

There are two problems with the translate processor approach. One is the performance. But the other issue is how we handle relative paths. There is a good discussion in #6609 about this same issue.

Basically, we need a way to decide relative versus absolute paths. Some processors are all absolute and some have relative paths. I propose in #6609 that we should use / to state absolute. And lack of that can be relative. Your elementEvent below will be a relative path with an absolute marker. This is a larger problem because it will lock us into the current behavior until a breaking change.

Thanks for flagging this. We understand this could conflict with a future convention where / explicitly means absolute (as discussed in #6609). We'll keep this in mind and align with whatever convention is adopted at the framework level. For now, this is the only viable approach given the current ExpressionEvaluator API.

@yavmanis , We should avoid pushing this bug further into processors. Using / is clearly indicating a JSON pointer at a root object.

I think these are our options at this point:

1. Resolve this correctly through JacksonEvent.
2. Create a throw-away configuration like use_relative_pointer_on_keep_when. The only allowable value would be true and this is not a default value so users must provide it.

- filter_list:
        source: "items"
        keep_when: '/status == "active"'
        filter_list_when: '/env == "production"'
        use_relative_pointer_on_keep_when: true

We would eventually remove it.

This name helps clarify that filter_list_when isn't using the relative pointer.

@dlvenable Since we implemented this processor from the beginning, we have carefully handled all the relevant conditions, particularly those related to relative JSON pointers. Specifically, the filter_list_when condition resolves against the root-level fields, while the keep_when condition resolves against the element-level fields for each object in the list.

Based on our testing, both conditions are working as expected within their respective contexts. Therefore, we believe that introducing the flag use_relative_pointer_on_keep_when: true may not be necessary for resolving JSON pointers, as this behavior is already managed within the processor implementation.

Additionally, the behavior of both conditions is consistent with the delete_entries processor.

However, if you have a specific use case in mind that is not adequately handled by the current implementation and would require this flag, please let us know. We would be happy to review and address it.

@dlvenable if keep_when syntax stays the same regardless, I'm not sure if adding a flag like this use_relative_pointer_on_keep_when: true is correct yet. Right now, as long as we are consistently saying "keep_when uses local element scope", and "filter_list_when" uses scope from the root of the Event", I think that is pretty clear.

We had a discussion last week about renaming the fields to promote consistency. I suggested in another comment that we use iterate_on and keep_element_when to signal that these have similar behavior to the delete entries processor.

The add entries processor is unfortunately not handling this in a great way currently. But this PR would match delete entries with those name changes.

When evaluateConditional throws for a single element, that element is silently dropped from the result. This is a data loss scenario with no signal to the user beyond WARN-level logs

Thanks for calling this out.
To improve visibility for this case, I updated the processor to mark the event as failed (tags_on_failure) whenever evaluateConditional throws for one or more elements, and to attach diagnostic metadata:

• filter_list_processor_failed_elements_count

• filter_list_processor_failed_elements

This keeps current filtering behavior consistent with similar processors, while ensuring downstream processors and users can clearly detect and inspect element-level evaluation failures instead of relying only on WARN logs.

The JacksonEvent.Builder is created once outside the loop and reused via .withData() on each iteration. This works today because build() consumes the current data reference, but it's fragile — if the builder ever accumulates state across calls, this breaks silently. Safer to create the builder inside the loop

Put validations in the Config classes themselves.

You can use an @AssertTrue from the Jakarta package. Other configs follow this pattern.

Target defaulting has been moved into FilterListProcessorConfig.getTarget(), which now returns source when target is not set.
Regarding @AssertTrue — the existing usages in this module are all for pure cross-field validations that don't require external dependencies. Our getTarget() never rejects anything — it always succeeds. It's a getter with a fallback default, not a validation. So @AssertTrue doesn't apply here.
Expression validation requires ExpressionEvaluator, which is a runtime dependency not available to config POJOs during Jakarta validation. No config class in the project uses @AssertTrue for expression validation.

There are two problems with the translate processor approach. One is the performance. But the other issue is how we handle relative paths. There is a good discussion in #6609 about this same issue.

Basically, we need a way to decide relative versus absolute paths. Some processors are all absolute and some have relative paths. I propose in #6609 that we should use / to state absolute. And lack of that can be relative. Your elementEvent below will be a relative path with an absolute marker. This is a larger problem because it will lock us into the current behavior until a breaking change.

From this and the examples in the description it looks like we are using "/value" in the keep_when expression as a special name for primitive lists. Are we sure this is the right approach? At first I was worried this would conflict with actual fields named "value", but this checking for Map condition does handle that conflict. We just need to be very clear in the documentation for keep_when that "/value" is a special keyword for primitive lists.

Can you SENSITIVE marker to this log? This is needed if we are logging the entire recordEvent. Also, may be useful having a metric on this error case as well, but not required.

Same here logging the full event requires SENSITIVE marker

Added SENSITIVE marker to warning logs that output full event context:

• Line 66: LOG.warn(EVENT, SENSITIVE, ...) — when source path is invalid
• Line 101: LOG.warn(EVENT, SENSITIVE, ...) — when keep_element_when evaluation fails

Let's add examples for primitive lists where "/value" is used as keyword, and explain that in the description.

Updated FilterListProcessorConfig @JsonPropertyDescription and @ExampleValues for keep_element_when:

• Added explanation: "For primitive elements (strings, numbers, booleans), the element value is accessible via the key /value"
• Added examples:
  • /value > 50 (numeric primitives)
  • /value != "" (string primitives)