Delete entries processor enhancements using regular expressions

[alamzeeshan]

Is your feature request related to a problem? Please describe.
Right now delete process entries processor provides a basic functionality of deleting fields listed under with_keys

Describe the solution you'd like
I would like to allow regular expression in with_keys which will delete any field matching the regex. Also

• a pointer option to specify the root field from the JSON data under which fields would be matched for deletion.
• a whitelist_names option to skip these fields for deletion.
• a delete_when option to conditionally apply this processor.

This can e.g. be useful if you have a parse_json or key_value processor that creates a number of fields with names that you don’t necessarily know the names of beforehand, and you only want to keep a subset of them.

Something in lines of logstash prune filter.

Describe alternatives you've considered (Optional)
NA

Additional context
Example config:

processor:
    - delete_entries:
        with_keys: ["message", "[^._\w]"]
        whitelist_names: [ "body-123" ]
        pointer: "/logs"
        delete_when: '/resource/attributes/k8s_label.sourcetype == "kv_pairs_generic"'

[graytaylor0]

Thanks for creating this issue! One question about this feature in regards to nested objects. How should the regex be matched for these?

For example, given the following event

{
   "logs": {
      "nested-key": {
          "one-key": "value",
          "two-key": "value"
      }
  }
}

and the following configuration

processor:
    - delete_entries:
        with_keys: ["[^one.*]"]
        pointer: "/logs"

Is the expectation that the one-key field is deleted, or would it only be deleted if the pointer points to nested-key

processor:
    - delete_entries:
        with_keys: ["[^one.*]"]
        pointer: "/nested-key"

[dlvenable]

Regarding the syntax, I think this should be a different value and not used by with_keys. One option is to have with_keys_regex.

Would we want to support using pointer and with_keys? If not, it would be confusing to have pointer here.

processor:
  - delete_entries:
       with_keys: ["a", "b"]
       pointer: "/nested-key"

Also, the whitelist_names could be skip_keys. Again, this doesn't make much sense combined with with_keys.

Might this make sense as a different processor?

processor:
  - delete_entries:
      regex:
        matches: ["[^._\w]"]
        pointer: /nested_key
        skip_keys: ["body-123"]

Or maybe even its own processor:

processor:
  - delete_entries_regex:
       matches: ["[^._\w]"]
       pointer: /nested_key
       skip_keys: ["body-123"]

[dlvenable]

Closed by #6094 and #6146.