Filter Steps Reference

Filter Steps Reference

This page provides a detailed reference for all 12 transformation steps available in the EventProcessor parsing pipeline.

1. json

Parses a field containing a JSON string and maps its keys to fields within the log.* namespace.

• Required: source (The field containing the JSON string, e.g., raw).
• Example:

  - json: 
      source: raw
      where: exists("raw")

2. rename

Maps existing fields to new names. Supports renaming multiple fields to a single target or vice versa, though typically used for normalization.

• Required: from (Array of source fields), to (Target field name).
• Example:

  - rename:
      from: [log.src_ip, log.source_address]
      to: origin.ip

3. cast

Converts field types to ensure correct indexing and correlation.

• Required: fields (Array), to (Target type).
• Supported Types: int, float, string, bool, []string.
• Example:

  - cast:
      fields: [origin.port, target.port]
      to: int

4. delete

Removes fields from the log to optimize storage and cleanup temporary processing data.

• Required: fields (Array).
• Example:

  - delete: 
      fields: [raw, temporary_meta]
      where: equals("actionResult", "success")

5. grok

Uses Go-template-based pattern matching for unstructured text. Since patterns are now globally integrated into the SDK, you can use standard aliases like {{.ipv4}} here and in other regex-capable steps.

• Required: source (Defaults to raw), patterns (List of { fieldName, pattern }).
• Standard Patterns: {{.ipv4}}, {{.time}}, {{.word}}, {{.greedy}}, {{.int}}.
• Example:

  - grok:
      source: log.message
      patterns:
        - fieldName: origin.ip
          pattern: '{{.ipv4}}'

6. kv (Key-Value)

Extracts key-value pairs from a string field.

• Required: source, fieldSplit (Separator between pairs), valueSplit (Separator between key and value).
• Example:

  - kv: 
      source: raw
      fieldSplit: " " 
      valueSplit: "="
      where: contains("raw", "=")

7. trim

Cleans strings by removing prefixes, suffixes, or matching patterns. With the latest SDK, the regex function supports global standard patterns (e.g., {{.int}}).

• Required: fields (Array), function (prefix, suffix, substring, regex).
• Optional: substring (The string or regex to remove).
• Example:

  - trim: 
      function: regex
      substring: "ID: {{.int}}"
      fields: [log.message]

8. add

Injects a new fixed string field into the log.

• Required: function: 'string', params: { key: "field_name", value: "fixed_value" }.
• Example:

  - add: 
      function: string
      params: 
        key: category
        value: security

9. reformat

Converts field formats, primarily used for standardizing timestamps.

• Required: fields (Array), function: time, fromFormat, toFormat.
• Example:

  - reformat:
      fields: [deviceTime]
      function: time
      fromFormat: 'Jan 02 15:04:05'
      toFormat: '2006-01-02T15:04:05Z'

10. csv

Parses comma-separated values from a string field.

• Required: source, separator, headers (Array of target field names).
• Example:

  - csv:
      source: raw
      separator: ","
      headers: [id, user, action, result]

11. dynamic

Calls an external gRPC plugin for enrichment or complex processing.

• Required: plugin (The plugin name), params (Map of key-value parameters).
• Example:

  - dynamic:
      plugin: com.utmstack.geolocation
      params: 
        source: origin.ip
        destination: origin.geolocation
      where: exists("origin.ip")

12. drop

Discards the log immediately, halting any further processing in the current or subsequent pipelines.

• Required: where (A CEL condition that must be met to drop the log).
• Example:

  - drop:
      where: equals("origin.ip", "127.0.0.1")