Skip to content

elasticsearch.md

Latest commit

 

History

History
164 lines (119 loc) · 7.35 KB

elasticsearch.md

File metadata and controls

164 lines (119 loc) · 7.35 KB

Elasticsearch

{% hint style="info" %} Supported event types: logs {% endhint %}

The Elasticsearch input plugin handles both Elasticsearch and OpenSearch Bulk API requests.

Configuration parameters

The plugin supports the following configuration parameters:

The table below includes both:

  • settings specific to the Elasticsearch input plugin
  • shared http_server.* listener settings that are used by several HTTP-based inputs

For a cross-plugin explanation of the shared listener settings, see Shared HTTP listener settings for inputs.

Key Description Default value
buffer_chunk_size Set the buffer chunk size. Compatibility alias for http_server.buffer_chunk_size. 512K
buffer_max_size Set the maximum size of buffer. Compatibility alias for http_server.buffer_max_size. 4M
hostname Specify hostname or fully qualified domain name. This parameter can be used for "sniffing" (auto-discovery of) cluster node information. localhost
http2 Enable HTTP/2 support. Compatibility alias for http_server.http2. true
http_server.max_connections Maximum number of concurrent active HTTP connections. 0 means unlimited. 0
http_server.workers Number of HTTP listener worker threads. 1
http_server.ingress_queue_event_limit Maximum number of deferred ingress queue entries. Applies only when http_server.workers is greater than 1. 8192
http_server.ingress_queue_byte_limit Maximum size of the deferred ingress queue. Applies only when http_server.workers is greater than 1. 256M
listen The address to listen on. 0.0.0.0
meta_key Specify a key name for meta information. @meta
port The port for Fluent Bit to listen on. 9200
tag_key Specify a key name for extracting as a tag. NULL
threaded Indicates whether to run this input in its own thread. false
version Specify the Elasticsearch version that Fluent Bit reports to clients during sniffing and API requests. 8.0.0

The http_server.ingress_queue_event_limit and http_server.ingress_queue_byte_limit settings matter only when http_server.workers is greater than 1.

TLS / SSL

The Elasticsearch input plugin supports TLS/SSL for receiving data from Beats agents or other clients over encrypted connections. For more details about the properties available and general configuration, refer to Transport Security.

When configuring TLS for Elasticsearch ingestion, common options include:

  • tls.verify: Enable or disable certificate validation for incoming connections.
  • tls.ca_file: Specify a CA certificate to validate client certificates when using mutual TLS (mTLS).
  • tls.crt_file and tls.key_file: Provide the server certificate and private key.

Sniffing

Elasticsearch clients use a process called "sniffing" to automatically discover cluster nodes. When a client connects, it can query the cluster to retrieve a list of available nodes and their addresses. This allows the client to distribute requests across the cluster and adapt when nodes join or leave.

The hostname parameter specifies the hostname or fully qualified domain name that Fluent Bit returns during sniffing requests. Clients use this information to build their connection list. Set this value to match how clients should reach this Fluent Bit instance (for example, an external IP or load balancer address rather than localhost in production environments).

Get started

In order to start performing the checks, you can run the plugin from the command line or through the configuration file:

Command line

From the command line you can configure Fluent Bit to handle Bulk API requests with the following options:

fluent-bit -i elasticsearch -p port=9200 -o stdout

Configuration file

In your configuration file append the following:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: elasticsearch
      listen: 0.0.0.0
      port: 9200

  outputs:
    - name: stdout
      match: '*'

{% endtab %} {% tab title="fluent-bit.conf" %}

[INPUT]
  name   elasticsearch
  listen 0.0.0.0
  port   9200

[OUTPUT]
  name  stdout
  match *

{% endtab %} {% endtabs %}

As described previously, the plugin will handle ingested Bulk API requests. For large bulk ingestion, you might have to increase buffer size using the buffer_max_size and buffer_chunk_size parameters:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: elasticsearch
      listen: 0.0.0.0
      port: 9200
      buffer_max_size: 20M
      buffer_chunk_size: 5M

  outputs:
    - name: stdout
      match: '*'

{% endtab %} {% tab title="fluent-bit.conf" %}

[INPUT]
  name              elasticsearch
  listen            0.0.0.0
  port              9200
  buffer_max_size   20M
  buffer_chunk_size 5M

[OUTPUT]
  name  stdout
  match *

{% endtab %} {% endtabs %}

Ingesting from beats series

Ingesting from beats series agents is also supported. For example, Filebeats, Metricbeat, and Winlogbeat are able to ingest their collected data through this plugin.

The Fluent Bit node information is returning as Elasticsearch 8.0.0.

Users must specify the following configurations on their beats configurations:

output.elasticsearch:
  allow_older_versions: true
  ilm: false

For large log ingestion on these beat plugins, users might have to configure rate limiting on those beats plugins when Fluent Bit indicates that the application is exceeding the size limit for HTTP requests:

processors:
  - rate_limit:
      limit: "200/s"