Add support for including keys based on regex patterns for select entries processor

[graytaylor0]

Description

This change adds new configuration parameters to the select entries processor

include_keys_regex -> A list of regex patterns to filter out keys to include from the event. Can be used alongside include_keys, and by default will only check keys at the root of the event

NOTE: This is not made configurable at this time but it is implemented for within the processor and tested
include_keys_regex_pointer -> A key pointing to a nested map that will lead to keys being included only from within this poointer. This will result in all other keys to be deleted if they are not in the include_keys_regex_pointer or in the include_keys.

include_keys was previously required in this processor, and now at least one of include_keys or include_keys_regex is required.

Issues Resolved

Related to #6087

Check List

• New functionality includes testing.
• New functionality has a documentation issue. Please link to it in this PR.
  • New functionality has javadoc added
• Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dlvenable]

Thanks @graytaylor0 !

[dlvenable]

Should this be implemented to default to / instead of having logic with null?

[graytaylor0]

Currently that does break the tests because this returns false

if (includeKeysRegexPointer != null) {
            if (!event.containsKey(includeKeysRegexPointer)) {
                return Collections.emptyMap();
            }

I can change it to this if you think it's preferred as a default

if (includeKeysRegexPointer != null && !includeKeysRegexPointer.equals("/")) {
            if (!event.containsKey(includeKeysRegexPointer)) {
                return Collections.emptyMap();
            }

[dlvenable]

Doing validation and setting in one method is not an ideal method structure. It would be better to implement the setter for includeKeysRegex and not even have a field for this.

I believe this would work:

@JsonProperty("include_keys_regex")
@JsonPropertyDescription("A list of regex patterns to match keys be selected from an event.")
private List<String> setIncludeKeysRegex() {
    includeKeysRegexPatterns = includeKeysRegex.stream().map(Pattern::compile).collect(Collectors.toList());
}

[dlvenable]

Maybe for a future PR - we could update the parser framework to support reading Pattern natively, similar to how we read Java Duration. We are doing this in a lot of plugins now and it would be nice to just have a single:

@JsonProperty("include_keys_regex")
@JsonPropertyDescription("A list of regex patterns to match keys be selected from an event.")
private List<Pattern> includeKeysRegex

[graytaylor0]

I tried this and it did not get run so it never got set.

I do think having the parser framework handle Pattern is ideal here, but for now this has been the best way to provide a clear error message to customers (you could call it in the constructor but that would mean the error would not be found until the config validations pass.

[graytaylor0]

I could run the validation in the method and then initialize the patterns in the constructor with a setter if that is preferred. It compiles the patterns twice but that shouldn't matter

[dlvenable]

I tried this and it did not get run so it never got set.

Maybe Jackson is bypassing the setter in favor of using the private fields always.

I could run the validation in the method and then initialize the patterns in the constructor with a setter if that is preferred. It compiles the patterns twice but that shouldn't matter

Maybe have the validator call the setter method for clarity?

[graytaylor0]

I'll make that change

[dlvenable]

This code would be clearer if both took the same approach of having its own private method.

Also, you could use a single Map and pass it as an argument to the private methods.

[graytaylor0]

I will move it into a private method, but I think it may be easier to read when map is returned from these methods rather than manipulating the final output map from within the method.

[dlvenable]

That is fine as well. Create two maps and combine them.