Skip to content

oci-logging-analytics.md

Latest commit

 

History

History
332 lines (268 loc) · 9.64 KB

oci-logging-analytics.md

File metadata and controls

332 lines (268 loc) · 9.64 KB
description Send logs to Oracle Cloud Infrastructure Log Analytics Service

Oracle Cloud Infrastructure Logging Analytics

{% hint style="info" %} Supported event types: logs {% endhint %}

The Oracle Cloud Infrastructure Logging Analytics output plugin lets you ingest your log records into the Oracle Cloud Infrastructure (OCI) Log Analytics service.

Configuration parameters

This plugin uses the following configuration parameters:

Key Description Default
config_file_location The location of the configuration file that contains OCI authentication details. none
namespace The OCI tenancy namespace to upload log data to. none
oci_config_in_record If set to true, the following oci_la_* parameters will be read from the record itself instead of the output plugin configuration. false
oci_la_entity_id The OCID of the Log Analytics entity. none
oci_la_entity_type The entity type of the Log Analytics entity. none
oci_la_global_metadata Specifies additional global metadata along with original log content to Log Analytics. The format is key_name value. This option can be set multiple times. none
oci_la_log_group_id Required. The Oracle Cloud Identifier (OCID) of the Log Analytics log group where you want to store logs. none
oci_la_log_path Specifies the original location of the log files. none
oci_la_log_set_id The OCID of the Log Analytics log set. none
oci_la_log_source_name Required. The Log Analytics Source to use for processing log records. none
oci_la_metadata Specifies additional metadata for a log event along with original log content to Log Analytics. The format is key_name value. This option can be set multiple times. none
profile_name The OCI configuration profile name to be used from the configuration file. DEFAULT
proxy The proxy name, in http://host:port format. Only supports HTTP protocol. none
uri The URI for the OCI Log Analytics REST API request. none
workers The number of workers to perform flush operations for this output. 1

TLS/SSL

The OCI Logging Analytics output plugin supports TLS/SSL. For more details about the properties available and general configuration, see TLS/SSL.

Get started

Prerequisites

  • You must onboard with the OCI Log Analytics service for the minimum required policies in the OCI region where you want to monitor. Refer to Log Analytics Quick Start for details.

  • You must create one or more OCI Log Analytics log groups. Refer to Create Log Group for details.

Run the output plugin

To insert records into the OCI Log Analytics service, you can run the plugin from the command line or through a configuration file.

Command line

The OCI Logging Analytics plugin can read the parameters from the command line in two ways, through the -p (property) argument. For example:

fluent-bit -i dummy -t dummy -o oci_logan -p config_file_location=<location> -p namespace=<namespace> \
  -p oci_la_log_group_id=<lg_id> -p oci_la_log_source_name=<ls_name> -p tls=on -p tls.verify=off -m '*'

Configuration file

In your main configuration file append the following:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: dummy
      tag: dummy

  outputs:
    - name: oracle_log_analytics
      match: '*'
      namespace: <namespace>
      config_file_location: <location>
      profile_name: ADMIN
      oci_la_log_source_name: <log-source-name>
      oci_la_log_group_id: <log-group-ocid>
      tls: on
      tls.verify: off

{% endtab %} {% tab title="fluent-bit.conf" %}

[INPUT]
  Name dummy
  Tag dummy

[OUTPUT]
  Name oracle_log_analytics
  Match *
  Namespace <namespace>
  Config_File_Location <location>
  Profile_Name ADMIN
  Oci_La_Log_Source_Name <log-source-name>
  Oci_La_Log_Group_Id <log-group-ocid>
  tls On
  tls.verify Off

{% endtab %} {% endtabs %}

Insert oci_la configurations in the record

In case of multiple inputs, where oci_la_* properties can differ, you can add the properties in the record itself and instruct the plugin to read these properties from the record. The option oci_config_in_record, when set to true in the output config, will make the plugin read the mandatory and optional oci_la properties from the incoming record. The user must ensure that the necessary configurations have been inserted using relevant filters, otherwise the respective chunk will be dropped. The following example inserts oci_la_log_source_name and oci_la_log_group_id in the record:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: dummy
      tag: dummy

  filters:
    - name: modify
      match: '*'
      add:
        - oci_la_log_source_name <LOG_SOURCE_NAME>
        - oci_la_log_group_id <LOG_GROUP_OCID>

  outputs:
    - name: oracle_log_analytics
      match: '*'
      config_file_location: <oci_file_path>
      profile_name: ADMIN
      oci_config_in_record: true
      tls: on
      tls.verify: off

{% endtab %} {% tab title="fluent-bit.conf" %}

[INPUT]
  Name dummy
  Tag dummy

[FILTER]
  Name modify
  Match *
  Add oci_la_log_source_name <LOG_SOURCE_NAME>
  Add oci_la_log_group_id <LOG_GROUP_OCID>

[OUTPUT]
  Name oracle_log_analytics
  Match *
  Config_File_Location <oci_file_path>
  Profile_Name ADMIN
  Oci_Config_In_Record true
  tls On
  tls.verify Off

{% endtab %} {% endtabs %}

Add optional metadata

You can attach certain metadata to the log events collected from various inputs.

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: dummy
      tag: dummy

  outputs:
    - name: oracle_log_analytics
      match: '*'
      namespace: example_namespace
      config_file_location: /Users/example_file_location
      profile_name: ADMIN
      oci_la_log_source_name: example_log_source
      oci_la_log_group_id: ocid.xxxxxx
      oci_la_global_metadata:
        - glob_key1 value1
        - glob_key2 value2
      oci_la_metadata:
        - key1 value1
        - key2 value2
      tls: on
      tls.verify: off

{% endtab %} {% tab title="fluent-bit.conf" %}

[INPUT]
  Name dummy
  Tag dummy

[OUTPUT]
  Name oracle_log_analytics
  Match *
  Namespace example_namespace
  Config_File_Location /Users/example_file_location
  Profile_Name ADMIN
  Oci_La_Log_Source_Name example_log_source
  Oci_La_Log_Group_Id ocid.xxxxxx
  Oci_La_Global_Metadata glob_key1 value1
  Oci_La_Global_Metadata glob_key2 value2
  Oci_La_Metadata key1 value1
  Oci_La_Metadata key2 value2
  tls On
  tls.verify Off

{% endtab %} {% endtabs %}

The previous configuration will generate a payload that resembles the following:

{
  "metadata": {
    "glob_key1": "value1",
    "glob_key2": "value2"
  },
  "logEvents": [
    {
      "metadata": {
        "key1": "value1",
        "key2": "value2"
      },
      "logSourceName": "example_log_source",
      "logRecords": [
        "dummy"
      ]
    }
  ]
}

The multiple oci_la_global_metadata and oci_la_metadata options are turned into a JSON object of key value pairs, nested under the key metadata.

With oci_config_in_record option set to true, the metadata key/value pairs will need to be injected in the record as an object of key/value pairs nested under the respective metadata field. The following example shows one such configuration:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: dummy
      tag: dummy

  filters:
    - name: modify
      match: '*'
      add:
        - olgm.key1 val1
        - olgm.key2 val2

    - name: nest
      match: '*'
      operation: olgm.*
      wildcard: olgm.*
      nest_under: oci_la_global_metadata
      remove_prefix: olgm.

    - name: modify
      match: '*'
      add:
        - oci_la_log_source_name <LOG_SOURCE_NAME>
        - oci_la_log_group_id <LOG_GROUP_OCID>

  outputs:
    - name: oracle_log_analytics
      match: '*'
      config_file_location: <oci_file_path>
      namespace: <oci_tenancy_namespace>
      profile_name: ADMIN
      oci_config_in_record: true
      tls: on
      tls.verify: off

{% endtab %} {% tab title="fluent-bit.conf" %}

[INPUT]
  Name dummy
  Tag dummy

[FILTER]
  Name modify
  Match *
  Add olgm.key1 val1
  Add olgm.key2 val2

[FILTER]
  Name nest
  Match *
  Operation nest
  Wildcard olgm.*
  Nest_Under oci_la_global_metadata
  Remove_Prefix olgm.

[FILTER]
  Name modify
  Match *
  Add oci_la_log_source_name <LOG_SOURCE_NAME>
  Add oci_la_log_group_id <LOG_GROUP_OCID>

[OUTPUT]
  Name oracle_log_analytics
  Match *
  Config_File_Location <oci_file_path>
  Namespace <oci_tenancy_namespace>
  Profile_Name ADMIN
  Oci_Config_In_Record true
  tls On
  tls.verify Off

{% endtab %} {% endtabs %}

The previous configuration first injects the necessary metadata keys and values in the record directly, with a prefix olgm. attached to the keys, which segregates the metadata keys from rest of the record keys. Then, using a nest filter, only the metadata keys are selected by the filter and nested under the oci_la_global_metadata key in the record, and the prefix olgm. is removed from the metadata keys.