diff --git a/administration/configuring-fluent-bit/yaml/pipeline-section.md b/administration/configuring-fluent-bit/yaml/pipeline-section.md index ce2c7a0b9..f7168c62d 100644 --- a/administration/configuring-fluent-bit/yaml/pipeline-section.md +++ b/administration/configuring-fluent-bit/yaml/pipeline-section.md @@ -81,6 +81,14 @@ The `name` parameter is required and defines for Fluent Bit which input plugin s Some HTTP-based input plugins share the same listener implementation and support the following common settings in addition to their plugin-specific parameters: +These settings are shared by HTTP-based inputs such as `http`, `splunk`, +`elasticsearch`, `opentelemetry`, and `prometheus_remote_write`. +Use these keys the same way across those plugins. + +If a plugin page shows one of the `http_server.*` keys in its configuration +table, it is documenting one of these shared listener settings, not a +plugin-specific behavior. + | Key | Description | Default | | --- | ----------- | ------- | | `http_server.http2` | Enable HTTP/2 support for the input listener. | `true` | @@ -88,6 +96,12 @@ Some HTTP-based input plugins share the same listener implementation and support | `http_server.buffer_chunk_size` | Set the allocation chunk size used for the HTTP request buffer. | `512K` | | `http_server.max_connections` | Set the maximum number of concurrent active HTTP connections. `0` means unlimited. | `0` | | `http_server.workers` | Set the number of HTTP listener worker threads. | `1` | +| `http_server.ingress_queue_event_limit` | Set the maximum number of deferred ingress queue entries. This setting applies only when `http_server.workers` is greater than `1`. | `8192` | +| `http_server.ingress_queue_byte_limit` | Set the maximum size of the deferred ingress queue. This setting applies only when `http_server.workers` is greater than `1`. | `256M` | + +When `http_server.workers` is `1`, Fluent Bit does not use the deferred +ingress queue, so the two `http_server.ingress_queue_*` settings have no +effect. For backward compatibility, some plugins also accept the legacy aliases `http2`, `buffer_max_size`, `buffer_chunk_size`, `max_connections`, and `workers`. diff --git a/administration/configuring-fluent-bit/yaml/service-section.md b/administration/configuring-fluent-bit/yaml/service-section.md index 2a1e6f557..41ab4ef88 100644 --- a/administration/configuring-fluent-bit/yaml/service-section.md +++ b/administration/configuring-fluent-bit/yaml/service-section.md @@ -36,7 +36,7 @@ The `service` section of YAML configuration files defines global properties of t | `streams_file` | Path for the [stream processor](../../../stream-processing/overview.md) configuration file. This file defines the rules and operations for stream processing in Fluent Bit. Stream processor configurations can also be defined directly in the `streams` section of YAML configuration files. | _none_ | | `windows.maxstdio` | If specified, adjusts the limit of `stdio`. Only provided for Windows. Values from `512` to `2048` are allowed. | `512` | -The `service` section only controls the built-in monitoring and control HTTP server. Plugin-specific HTTP listener settings such as `http_server.http2`, `http_server.buffer_max_size`, `http_server.buffer_chunk_size`, `http_server.max_connections`, and `http_server.workers` are configured on the relevant input plugin in the [`pipeline.inputs`](../yaml/pipeline-section.md#shared-http-listener-settings-for-inputs) section. +The `service` section only controls the built-in monitoring and control HTTP server. Plugin-specific HTTP listener settings such as `http_server.http2`, `http_server.buffer_max_size`, `http_server.buffer_chunk_size`, `http_server.max_connections`, `http_server.workers`, `http_server.ingress_queue_event_limit`, and `http_server.ingress_queue_byte_limit` are configured on the relevant input plugin in the [`pipeline.inputs`](../yaml/pipeline-section.md#shared-http-listener-settings-for-inputs) section. ## Storage configuration diff --git a/administration/monitoring.md b/administration/monitoring.md index 9f2de57a1..167ff341c 100644 --- a/administration/monitoring.md +++ b/administration/monitoring.md @@ -189,6 +189,10 @@ The following terms are key to understanding how Fluent Bit processes metrics: The Fluent Bit engine attempts to fit records into chunks of at most `2 MB`, but the size can vary at runtime. Chunks are then sent to an output. An output plugin instance can successfully send the full chunk to the destination and mark it as successful. If an unrecoverable error is encountered, the chunk fails entirely. Otherwise, the output can request a retry. +Some metrics are available only for specific plugins or runtime modes. For +example, the `fluentbit_input_http_server_ingress_queue_*` metrics are exposed +only for HTTP-based inputs that use `http_server.workers` greater than `1`. + | Metric Name | Labels | Description | Type | Unit | | ----------- | ------ | ----------- | ---- | ---- | | `fluentbit_build_info` | hostname: the hostname, version: the version of Fluent Bit, os: OS type | Build version information. The value is the Unix epoch timestamp of the configuration context initialization. | gauge | seconds | @@ -202,6 +206,9 @@ The following terms are key to understanding how Fluent Bit processes metrics: | `fluentbit_input_files_opened_total` | name: the name or alias for the input instance | The total number of opened files. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | files | | `fluentbit_input_files_rotated_total` | name: the name or alias for the input instance | The total number of rotated files. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | files | | `fluentbit_input_ingestion_paused` | name: the name or alias for the input instance | Indicates whether the input instance ingestion is currently paused (1) or not (0). | gauge | boolean | +| `fluentbit_input_http_server_ingress_queue_busy_total` | name: the name or alias for the input instance | The total number of times an HTTP worker input hit a full deferred ingress queue and applied backpressure. Only available for HTTP-based inputs using `http_server.workers` greater than `1`. | counter | events | +| `fluentbit_input_http_server_ingress_queue_pending_bytes` | name: the name or alias for the input instance | The current size of the deferred ingress queue in bytes. Only available for HTTP-based inputs using `http_server.workers` greater than `1`. | gauge | bytes | +| `fluentbit_input_http_server_ingress_queue_pending_events` | name: the name or alias for the input instance | The current number of deferred ingress queue entries. Only available for HTTP-based inputs using `http_server.workers` greater than `1`. | gauge | events | | `fluentbit_input_long_line_skipped_total` | name: the name or alias for the input instance | The total number of skipped occurrences for long lines. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin when `skip_long_lines` is enabled. | counter | occurrences | | `fluentbit_input_long_line_truncated_total` | name: the name or alias for the input instance | The total number of truncated occurrences for long lines. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin when `truncate_long_lines` is enabled. | counter | occurrences | | `fluentbit_input_memrb_dropped_bytes` | name: the name or alias for the input instance | The number of bytes dropped by the memory ring buffer (`memrb`) storage type when the buffer is full. Only available for input plugins with `storage.type` set to `memrb`. | counter | bytes | diff --git a/installation/upgrade-notes.md b/installation/upgrade-notes.md index c866c80b1..3c8c1ffde 100644 --- a/installation/upgrade-notes.md +++ b/installation/upgrade-notes.md @@ -25,6 +25,8 @@ The HTTP-based input plugins now use a shared HTTP listener configuration model. - `http_server.buffer_max_size` - `http_server.max_connections` - `http_server.workers` +- `http_server.ingress_queue_event_limit` +- `http_server.ingress_queue_byte_limit` Legacy per-plugin names such as `http2`, `buffer_chunk_size`, and `buffer_max_size` are still accepted as compatibility aliases, but new configurations should use the `http_server.*` names. diff --git a/installation/whats-new-in-fluent-bit-v5.0.md b/installation/whats-new-in-fluent-bit-v5.0.md index 63eb30666..e05696167 100644 --- a/installation/whats-new-in-fluent-bit-v5.0.md +++ b/installation/whats-new-in-fluent-bit-v5.0.md @@ -44,6 +44,8 @@ HTTP-based inputs now use a shared listener configuration model. The preferred s - `http_server.buffer_max_size` - `http_server.max_connections` - `http_server.workers` +- `http_server.ingress_queue_event_limit` +- `http_server.ingress_queue_byte_limit` Legacy aliases such as `http2`, `buffer_chunk_size`, and `buffer_max_size` still work, but new configurations should use the `http_server.*` names. diff --git a/pipeline/inputs/elasticsearch.md b/pipeline/inputs/elasticsearch.md index b11e7462f..8f222247c 100644 --- a/pipeline/inputs/elasticsearch.md +++ b/pipeline/inputs/elasticsearch.md @@ -10,6 +10,14 @@ The _Elasticsearch_ input plugin handles both Elasticsearch and OpenSearch Bulk The plugin supports the following configuration parameters: +The table below includes both: + +- settings specific to the Elasticsearch input plugin +- shared `http_server.*` listener settings that are used by several HTTP-based inputs + +For a cross-plugin explanation of the shared listener settings, see +[Shared HTTP listener settings for inputs](../../administration/configuring-fluent-bit/yaml/pipeline-section.md#shared-http-listener-settings-for-inputs). + | Key | Description | Default value | |:--------------------|:-----------------------------------------------------------------------------------------------------------------------------------------|:--------------| | `buffer_chunk_size` | Set the buffer chunk size. Compatibility alias for `http_server.buffer_chunk_size`. | `512K` | @@ -18,6 +26,8 @@ The plugin supports the following configuration parameters: | `http2` | Enable HTTP/2 support. Compatibility alias for `http_server.http2`. | `true` | | `http_server.max_connections` | Maximum number of concurrent active HTTP connections. `0` means unlimited. | `0` | | `http_server.workers` | Number of HTTP listener worker threads. | `1` | +| `http_server.ingress_queue_event_limit` | Maximum number of deferred ingress queue entries. Applies only when `http_server.workers` is greater than `1`. | `8192` | +| `http_server.ingress_queue_byte_limit` | Maximum size of the deferred ingress queue. Applies only when `http_server.workers` is greater than `1`. | `256M` | | `listen` | The address to listen on. | `0.0.0.0` | | `meta_key` | Specify a key name for meta information. | `@meta` | | `port` | The port for Fluent Bit to listen on. | `9200` | @@ -25,6 +35,10 @@ The plugin supports the following configuration parameters: | `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | | `version` | Specify the Elasticsearch version that Fluent Bit reports to clients during sniffing and API requests. | `8.0.0` | +The `http_server.ingress_queue_event_limit` and +`http_server.ingress_queue_byte_limit` settings matter only when +`http_server.workers` is greater than `1`. + ### TLS / SSL The Elasticsearch input plugin supports TLS/SSL for receiving data from Beats agents or other clients over encrypted connections. For more details about the properties available and general configuration, refer to [Transport Security](../../administration/transport-security.md). diff --git a/pipeline/inputs/http.md b/pipeline/inputs/http.md index c7edfc735..2d883c65a 100644 --- a/pipeline/inputs/http.md +++ b/pipeline/inputs/http.md @@ -9,6 +9,14 @@ The _HTTP_ input plugin lets Fluent Bit open an HTTP port that you can then rout ## Configuration parameters +The table below includes both: + +- settings specific to the HTTP input plugin +- shared `http_server.*` listener settings that are used by several HTTP-based inputs + +For a cross-plugin explanation of the shared listener settings, see +[Shared HTTP listener settings for inputs](../../administration/configuring-fluent-bit/yaml/pipeline-section.md#shared-http-listener-settings-for-inputs). + | Key | Description | Default | |----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|-----------| | `add_remote_addr` | Adds a `REMOTE_ADDR` field to the record. The value of `REMOTE_ADDR` is the client's address, which is extracted from the `X-Forwarded-For` header. | `false` | @@ -17,6 +25,8 @@ The _HTTP_ input plugin lets Fluent Bit open an HTTP port that you can then rout | `http2` | Enable HTTP/2 support. Compatibility alias for `http_server.http2`. | `true` | | `http_server.max_connections` | Maximum number of concurrent active HTTP connections. `0` means unlimited. | `0` | | `http_server.workers` | Number of HTTP listener worker threads. | `1` | +| `http_server.ingress_queue_event_limit` | Maximum number of deferred ingress queue entries. Applies only when `http_server.workers` is greater than `1`. | `8192` | +| `http_server.ingress_queue_byte_limit` | Maximum size of the deferred ingress queue. Applies only when `http_server.workers` is greater than `1`. | `256M` | | `listen` | The address to listen on. | `0.0.0.0` | | `oauth2.allowed_audience` | Audience claim to enforce when validating incoming `OAuth 2.0` `JWT` tokens. | _none_ | | `oauth2.allowed_clients` | Authorized `client_id` or `azp` claim values. Can be specified multiple times. | _none_ | @@ -31,6 +41,10 @@ The _HTTP_ input plugin lets Fluent Bit open an HTTP port that you can then rout | `tag_key` | Specify the key name to overwrite a tag. If set, the tag will be overwritten by a value of the key. | _none_ | | `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | +The `http_server.ingress_queue_event_limit` and +`http_server.ingress_queue_byte_limit` settings matter only when +`http_server.workers` is greater than `1`. + ### TLS / SSL HTTP input plugin supports TLS/SSL. For more details about the properties available and general configuration, refer to [Transport Security](../../administration/transport-security.md). diff --git a/pipeline/inputs/opentelemetry.md b/pipeline/inputs/opentelemetry.md index cdc108287..d3f643941 100644 --- a/pipeline/inputs/opentelemetry.md +++ b/pipeline/inputs/opentelemetry.md @@ -14,6 +14,14 @@ Fluent Bit has a compliant implementation which fully supports `OTLP/HTTP` and ` ## Configuration +The table below includes both: + +- settings specific to the OpenTelemetry input plugin +- shared `http_server.*` listener settings that are used by several HTTP-based inputs + +For a cross-plugin explanation of the shared listener settings, see +[Shared HTTP listener settings for inputs](../../administration/configuring-fluent-bit/yaml/pipeline-section.md#shared-http-listener-settings-for-inputs). + | Key | Description | Default | |-------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------| | `alias` | Sets an alias for multiple instances of the same input plugin. If no alias is specified, a default name will be assigned using the plugin name followed by a dot and a sequence number. | _none_ | @@ -24,6 +32,8 @@ Fluent Bit has a compliant implementation which fully supports `OTLP/HTTP` and ` | `http2` | Enable HTTP/2 protocol support for the OpenTelemetry receiver. | `true` | | `http_server.max_connections` | Maximum number of concurrent active HTTP connections. `0` means unlimited. | `0` | | `http_server.workers` | Number of HTTP listener worker threads. | `1` | +| `http_server.ingress_queue_event_limit` | Maximum number of deferred ingress queue entries. Applies only when `http_server.workers` is greater than `1`. | `8192` | +| `http_server.ingress_queue_byte_limit` | Maximum size of the deferred ingress queue. Applies only when `http_server.workers` is greater than `1`. | `256M` | | `listen` | The network address to listen on. | `0.0.0.0` | | `log_level` | Specifies the log level for this plugin. If not set here, the plugin uses the global log level specified in the `service` section of your configuration file. | `info` | | `log_suppress_interval` | Suppresses log messages from this plugin that appear similar within a specified time interval. `0` no suppression. | `0` | @@ -69,6 +79,10 @@ Fluent Bit has a compliant implementation which fully supports `OTLP/HTTP` and ` | `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | | `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | +The `http_server.ingress_queue_event_limit` and +`http_server.ingress_queue_byte_limit` settings matter only when +`http_server.workers` is greater than `1`. + When `raw_traces` is set to `false` (default), the traces endpoint (`/v1/traces`) processes incoming trace data using the unified JSON parser with strict validation. The endpoint accepts both `protobuf` and `JSON` encoded payloads. When `raw_traces` is set to `true`, any data forwarded to the traces endpoint will be packed and forwarded as a log message without processing, validation, or conversion to the Fluent Bit internal trace format. ### OpenTelemetry transport protocol endpoints diff --git a/pipeline/inputs/prometheus-remote-write.md b/pipeline/inputs/prometheus-remote-write.md index 5e7109c78..1532379fd 100644 --- a/pipeline/inputs/prometheus-remote-write.md +++ b/pipeline/inputs/prometheus-remote-write.md @@ -12,6 +12,14 @@ The _Prometheus remote write_ input plugin lets you ingest a payload in the Prom ## Configuration parameters +The table below includes both: + +- settings specific to the Prometheus remote write input plugin +- shared `http_server.*` listener settings that are used by several HTTP-based inputs + +For a cross-plugin explanation of the shared listener settings, see +[Shared HTTP listener settings for inputs](../../administration/configuring-fluent-bit/yaml/pipeline-section.md#shared-http-listener-settings-for-inputs). + | Key | Description | Default | |----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| | `buffer_chunk_size` | Sets the chunk size for incoming data. These chunks are then stored and managed in the space specified by `buffer_max_size`. Compatibility alias for `http_server.buffer_chunk_size`. | `512K` | @@ -19,6 +27,8 @@ The _Prometheus remote write_ input plugin lets you ingest a payload in the Prom | `http2` | Enable HTTP/2 support. Compatibility alias for `http_server.http2`. | `true` | | `http_server.max_connections` | Maximum number of concurrent active HTTP connections. `0` means unlimited. | `0` | | `http_server.workers` | Number of HTTP listener worker threads. | `1` | +| `http_server.ingress_queue_event_limit` | Maximum number of deferred ingress queue entries. Applies only when `http_server.workers` is greater than `1`. | `8192` | +| `http_server.ingress_queue_byte_limit` | Maximum size of the deferred ingress queue. Applies only when `http_server.workers` is greater than `1`. | `256M` | | `listen` | The address to listen on. | `0.0.0.0` | | `port` | The port to listen on. | `8080` | | `successful_response_code` | Specifies the success response code. Supported values are `200`, `201`, and `204`. | `201` | @@ -26,6 +36,10 @@ The _Prometheus remote write_ input plugin lets you ingest a payload in the Prom | `threaded` | Specifies whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | | `uri` | Specifies an optional HTTP URI for the target web server listening for Prometheus remote write payloads (for example, `/api/prom/push`). | _none_ | +The `http_server.ingress_queue_event_limit` and +`http_server.ingress_queue_byte_limit` settings matter only when +`http_server.workers` is greater than `1`. + ## Configuration file The following examples are sample configuration files for this input plugin: diff --git a/pipeline/inputs/splunk.md b/pipeline/inputs/splunk.md index 82c4943b9..7820e67cc 100644 --- a/pipeline/inputs/splunk.md +++ b/pipeline/inputs/splunk.md @@ -10,6 +10,14 @@ The _Splunk_ input plugin handles [Splunk HTTP HEC](https://docs.splunk.com/Docu This plugin uses the following configuration parameters: +The table below includes both: + +- settings specific to the Splunk input plugin +- shared `http_server.*` listener settings that are used by several HTTP-based inputs + +For a cross-plugin explanation of the shared listener settings, see +[Shared HTTP listener settings for inputs](../../administration/configuring-fluent-bit/yaml/pipeline-section.md#shared-http-listener-settings-for-inputs). + | Key | Description | Default | |---------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------| | `add_remote_addr` | Inject a remote address field into the record, using the `X-Forwarded-For` header or the connection address as the value. | `false` | @@ -18,6 +26,8 @@ This plugin uses the following configuration parameters: | `http2` | Enable HTTP/2 support. Compatibility alias for `http_server.http2`. | `true` | | `http_server.max_connections` | Maximum number of concurrent active HTTP connections. `0` means unlimited. | `0` | | `http_server.workers` | Number of HTTP listener worker threads. | `1` | +| `http_server.ingress_queue_event_limit` | Maximum number of deferred ingress queue entries. Applies only when `http_server.workers` is greater than `1`. | `8192` | +| `http_server.ingress_queue_byte_limit` | Maximum size of the deferred ingress queue. Applies only when `http_server.workers` is greater than `1`. | `256M` | | `listen` | The address to listen on. | `0.0.0.0` | | `port` | The port for Fluent Bit to listen on. | `8088` | | `remote_addr_key` | Record key name used to store the remote address when `add_remote_addr` is enabled. | `remote_addr` | @@ -28,6 +38,10 @@ This plugin uses the following configuration parameters: | `tag_key` | Specify the key name to overwrite a tag. If set, the tag will be overwritten by a value of the key. | _none_ | | `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | +The `http_server.ingress_queue_event_limit` and +`http_server.ingress_queue_byte_limit` settings matter only when +`http_server.workers` is greater than `1`. + ## Get started To get started, you can run the plugin from the command line or through the configuration file.