Skip to content

content-modifier.md

Latest commit

 

History

History
288 lines (212 loc) · 8.27 KB

content-modifier.md

File metadata and controls

288 lines (212 loc) · 8.27 KB

Content modifier

The content modifier processor lets you manipulate the content, metadata, and attributes of logs and traces.

Similar to how filters work, this processor uses a unified mechanism to perform operations for data manipulation. The most significant difference is that processors perform better than filters, and when chaining them, there are no encoding/decoding performance penalties.

{% hint style="info" %}

Only YAML configuration files support processors.

{% endhint %}

Contexts

The content modifier relies on context, meaning the place where the content modification will happen. Fluent Bit provides different contexts to manipulate the desired information.

The following contexts are available:

Name Telemetry type Description
attributes Logs Modifies the attributes or metadata of a log record.
body Logs Modifies the content of a log record.
span_name Traces Modifies the name of a span.
span_kind Traces Modifies the kind of a span.
span_status Traces Modifies the status of a span.
span_attributes Traces Modifies the attributes of a span.

OpenTelemetry contexts

Additionally, Fluent Bit provides specific contexts for modifying data that follows the OpenTelemetry log schema.

The following contexts are available:

Name Telemetry type Description
otel_resource_attributes Logs Modifies the attributes of the log resource. Operates on shared data across a group of records.
otel_scope_name Logs Modifies the name of a log scope. Operates on shared data across a group of records.
otel_scope_version Logs Modifies the version of a log scope. Operates on shared data across a group of records.
otel_scope_attributes Logs Modifies the attributes of a log scope. Operates on shared data across a group of records.
otel_log_attributes Logs Modifies the attributes of an individual OpenTelemetry log record. Operates per record.

{% hint style="info" %}

If your data doesn't follow the OpenTelemetry log schema, but your log destination expects to be in that format, you can use the OpenTelemetry envelope processor to transform your data. You can then pass that transformed data through the content modifier filter and use OpenTelemetry contexts accordingly.

{% endhint %}

Configuration parameters

The following configuration parameters are available:

Key Description
action Specifies the type of action to perform on the content.
context Specifies the context where the modifications will happen.
converted_type Defines the data type to perform the conversion. Possible values: string, boolean, int, and double.
key Specifies the name of the key that will be used to apply the modification.
pattern Defines a regular expression pattern. This property is only used by the extract action.
value The role of this parameter changes based on the action type.

Actions

The actions specify the type of operation to run on top of a specific key or content from a log or a trace. The following actions are available:

Action Description
convert Converts the data type of a key value. The key and converted_type parameters are required.
delete Deletes a key from the target context. The key parameter is required.
extract Extracts the value of a single key as a list of key/value pairs. This action needs the configuration of a regular expression in the pattern property. The key and pattern parameters are required.
hash Replaces the key value with a hash generated by the SHA-256 algorithm, the binary value generated is finally set as a hex string representation. The key parameter is required.
insert Inserts a new key with a value into the target context. The key and value parameters are required.
rename Changes the name of a key. The value set in the configuration will represent the new name. The key and value parameters are required.
upsert Given a specific key with a value, the upsert operation will try to update the value of the key. If the key doesn't exist, a new key will be created. The key and value parameters are required.

Insert example

The following example appends the key color with the value blue to the log stream.

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4"}'

      processors:
        logs:
          - name: content_modifier
            action: insert
            key: "color"
            value: "blue"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

{% endtab %} {% endtabs %}

Upsert example

Update the value of key1 and insert key2:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4"}'

      processors:
        logs:
          - name: content_modifier
            action: upsert
            key: "key1"
            value: "5678"

          - name: content_modifier
            action: upsert
            key: "key2"
            value: "example"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

{% endtab %} {% endtabs %}

Delete example

Delete key2 from the stream:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "key2": "example"}'

      processors:
        logs:
          - name: content_modifier
            action: delete
            key: "key2"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

{% endtab %} {% endtabs %}

Rename example

Change the name of key2 to test:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "key2": "example"}'

      processors:
        logs:
          - name: content_modifier
            action: rename
            key: "key2"
            value: "test"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

{% endtab %} {% endtabs %}

Hash example

Apply the SHA-256 algorithm for the value of the key password:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: dummy
      dummy: '{"username": "bob", "password": "12345"}'

      processors:
        logs:
          - name: content_modifier
            action: hash
            key: "password"

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

{% endtab %} {% endtabs %}

Extract example

By using a domain address, perform an extraction of the components of it as a list of key value pairs:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: dummy
      dummy: '{"http.url": "https://fluentbit.io/docs?q=example"}'

      processors:
        logs:
          - name: content_modifier
            action: extract
            key: "http.url"
            pattern: ^(?<http_protocol>https?):\/\/(?<http_domain>[^\/\?]+)(?<http_path>\/[^?]*)?(?:\?(?<http_query_params>.*))?

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

{% endtab %} {% endtabs %}

Convert example

Both keys in the example are strings. Convert the key1 to a double/float type and key2 to a boolean:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "key2": "true"}'

      processors:
        logs:
          - name: content_modifier
            action: convert
            key: key1
            converted_type: int

          - name: content_modifier
            action: convert
            key: key2
            converted_type: boolean

  outputs:
    - name : stdout
      match: '*'
      format: json_lines

{% endtab %} {% endtabs %}