Merge remote-tracking branch 'origin/master' into azure-blob-mi-docs

Author: temporaer

.gitignore

@@ -6,3 +6,4 @@ old/*
 .idea/
 .cursor
 CLAUDE.md
+.claude*

administration/backpressure.md

@@ -58,19 +58,23 @@ In a few seconds, if the scheduler was able to flush the initial 700 KB of
 
 If one or more active input plugins use [filesystem buffering](../pipeline/buffering.md#filesystem-buffering-hybrid), use the following settings to manage backpressure.
 
-### Set `storage.max_chunks_up` and `storage.backlog.mem_limit` in global settings
+### Set `storage.max_chunks_up` in global settings
 
-In the [`service` section](../administration/configuring-fluent-bit/yaml/service-section.md) of your Fluent Bit configuration file, you can configure the `storage.max_chunks_up` and `storage.backlog.mem_limit` settings. Both settings dictate how much data can be buffered to memory by input plugins that use filesystem buffering, and are combined limits shared by all applicable input plugins.
+In the [`service` section](../administration/configuring-fluent-bit/yaml/service-section.md) of your Fluent Bit configuration file, you can configure the `storage.max_chunks_up` setting. This setting is a global cap on how many filesystem-backed chunks can be `up` (mapped into memory) at any time, and is shared across all input plugins that use filesystem buffering.
 
 {% hint style="info" %}
-These settings don't affect how much data can be buffered to memory by plugins that use memory-only buffering.
+This setting doesn't affect how much data can be buffered to memory by plugins that use memory-only buffering.
 {% endhint %}
 
-When either the specified `storage.max_chunks_up` or `storage.backlog.mem_limit` capacity is reached, all input plugins that use filesystem buffering will stop buffering data to memory until more memory becomes available. Whether these input plugins continue buffering data to the filesystem depends on each plugin's specified `storage.pause_on_chunks_overlimit` value.
+When the `storage.max_chunks_up` capacity is reached, input plugins that use filesystem buffering stop bringing new chunks up into memory. Whether these input plugins continue buffering data to the filesystem depends on each plugin's specified `storage.pause_on_chunks_overlimit` value.
+
+### Tune `storage.backlog.mem_limit` for the backlog plugin
+
+`storage.backlog.mem_limit` only governs the built-in `storage_backlog` input plugin, which promotes filesystem chunks left over from a previous Fluent Bit run back into memory so output plugins can flush them. While the up chunks owned by `storage_backlog` consume less memory than this limit, Fluent Bit continues to promote additional backlog chunks. Reaching this limit doesn't pause regular input plugins, and the limit doesn't cap memory used by inputs other than `storage_backlog`.
 
 ### Set `storage.pause_on_chunks_overlimit` for input plugins
 
-For input plugins that use filesystem buffering, you can configure the `storage.pause_on_chunks_overlimit` setting to specify how each plugin should behave after the global `storage.max_chunks_up` or `storage.backlog.mem_limit` capacity is reached.
+For input plugins that use filesystem buffering, you can configure the `storage.pause_on_chunks_overlimit` setting to specify how each plugin should behave after the global `storage.max_chunks_up` capacity is reached.
 
 If `storage.pause_on_chunks_overlimit` is set to `off` for an input plugin, the input plugin will stop buffering data to memory but continue buffering data to the filesystem.
 

administration/configuring-fluent-bit/classic-mode/configuration-file.md

@@ -62,6 +62,8 @@ The `INPUT` section defines a source (related to an input plugin). Each [input p
 
 `Name` is mandatory and tells Fluent Bit which input plugin to load. `Tag` is mandatory for all plugins except for the `input forward` plugin, which provides dynamic tags.
 
+There is no hard-coded limit on the number of `INPUT` sections. The practical maximum depends on available system resources such as memory and file descriptors.
+
 ### Example
 
 The following is an example of an `INPUT` section:
@@ -85,6 +87,8 @@ The `FILTER` section defines a filter (related to an filter plugin). Each filter
 
 `Name` is mandatory and lets Fluent Bit know which filter plugin should be loaded. `Match` or `Match_Regex` is mandatory for all plugins. If both are specified, `Match_Regex` takes precedence.
 
+There is no hard-coded limit on the number of `FILTER` sections. The practical maximum depends on available system resources such as memory.
+
 ### Filter example
 
 The following is an example of a `FILTER` section:
@@ -98,7 +102,7 @@ The following is an example of a `FILTER` section:
 
 ## Config output
 
-The `OUTPUT` section specifies a destination that certain records should go to after a `Tag` match. Fluent Bit can route up to 256 `OUTPUT` plugins. The configuration supports the following keys:
+The `OUTPUT` section specifies a destination that certain records should go to after a `Tag` match. The configuration supports the following keys:
 
 | Key         | Description    |
 | ----------- | -------------- |
@@ -107,6 +111,8 @@ The `OUTPUT` section specifies a destination that certain records should go to a
 | `Match_Regex` | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regular expression syntax. |
 | `Log_Level` | Set the plugin's logging verbosity level. Allowed values are: `off`, `error`, `warn`, `info`, `debug`, and `trace`. Defaults to the `SERVICE` section's `Log_Level`. |
 
+There is no hard-coded limit on the number of `OUTPUT` sections. The routing `bitmask` is dynamically sized at startup based on the number of configured output plugins. The practical maximum depends on available system resources such as memory and file descriptors.
+
 ### Output example
 
 The following is an example of an `OUTPUT` section:

administration/configuring-fluent-bit/yaml/environment-variables-section.md

@@ -1,8 +1,12 @@
 # Environment variables
 
-The `env` section of YAML configuration files lets you define environment variables. These variables can then be used to dynamically replace values throughout your configuration using the `${VARIABLE_NAME}` syntax.
+The `env` section of YAML configuration files lets you define environment variables. These variables can then be used to dynamically replace values throughout your configuration using the `${VARIABLE_NAME}` syntax. The `env` section supports two forms: a map form for static key-value pairs and an extended list form for advanced use cases like file-backed secrets with automatic refresh.
 
-Values set in the `env` section are case-sensitive. However, as a best practice, Fluent Bit recommends using uppercase names for environment variables. The following example defines two variables, `FLUSH_INTERVAL` and `STDOUT_FMT`, which can be accessed in the configuration using `${FLUSH_INTERVAL}` and `${STDOUT_FMT}`:
+Values set in the `env` section are case-sensitive. However, as a best practice, Fluent Bit recommends using uppercase names for environment variables.
+
+## Map form
+
+The map form defines variables as key-value pairs. The following example defines two variables, `FLUSH_INTERVAL` and `STDOUT_FMT`, which can be accessed in the configuration using `${FLUSH_INTERVAL}` and `${STDOUT_FMT}`:
 
 ```yaml
 env:
@@ -23,6 +27,44 @@ pipeline:
       format: ${STDOUT_FMT}
 ```
 
+## Extended list form
+
+The `env` section also accepts a YAML list of entries. Each entry is a mapping with the following properties:
+
+| Property | Required | Description |
+| :------- | :------- | :---------- |
+| `name` | Yes | The name of the environment variable. |
+| `value` | No | An explicit value for the variable. Mutually exclusive with `uri`. |
+| `uri` | No | A URI pointing to the variable's value source. Currently supports `file://` URIs (for example, `file:///run/secrets/my_secret`). The file contents are read and used as the variable's value. |
+| `refresh_interval` | No | How often (in seconds) to re-read the value from the `uri` source. Only meaningful when `uri` is set. Default is `0` (read once at startup). |
+
+Either `value` or `uri` must be provided for each entry.
+
+When `uri` and `refresh_interval` are both set, Fluent Bit re-reads the file at the specified interval. This enables dynamic configuration values, such as rotating secrets, without restarting Fluent Bit. If `refresh_interval` is `0` or omitted, the file is read only once at startup.
+
+```yaml
+env:
+  - name: DB_PASSWORD
+    uri: file:///run/secrets/db_password
+    refresh_interval: 60
+  - name: REGION
+    value: us-east-1
+
+service:
+  flush: 1
+  log_level: info
+
+pipeline:
+  inputs:
+    - name: random
+
+  outputs:
+    - name: stdout
+      match: '*'
+```
+
+In this example, `${DB_PASSWORD}` is read from the file `/run/secrets/db_password` and refreshed every 60 seconds. `${REGION}` is set to the static value `us-east-1`.
+
 ## Predefined variables
 
 Fluent Bit supports the following predefined environment variables. You can reference these variables in configuration files without defining them in the `env` section.

administration/configuring-fluent-bit/yaml/pipeline-section.md

@@ -77,6 +77,8 @@ The `inputs` section defines one or more [input plugins](../../../pipeline/input
 
 The `name` parameter is required and defines for Fluent Bit which input plugin should be loaded. The `tag` parameter is required for all plugins except for the `forward` plugin, which provides dynamic tags.
 
+There is no hard-coded limit on the number of input plugins. The practical maximum depends on available system resources such as memory and file descriptors.
+
 ### Shared HTTP listener settings for inputs
 
 Some HTTP-based input plugins share the same listener implementation and support the following common settings in addition to their plugin-specific parameters:
@@ -144,6 +146,8 @@ The `filters` section defines one or more [filters](../../../pipeline/filters.md
 
 The `name` parameter is required and lets Fluent Bit know which filter should be loaded. One of either the `match` or `match_regex` parameters is required. If both are specified, `match_regex` takes precedence.
 
+There is no hard-coded limit on the number of filter plugins. The practical maximum depends on available system resources such as memory.
+
 ### Example filter configuration
 
 The following is an example of a `filters` section that contains a `grep` plugin:
@@ -168,7 +172,7 @@ The `outputs` section defines one or more [output plugins](../../../pipeline/out
 | `match_regex` | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regular expression syntax. |
 | `log_level` | Set the plugin's logging verbosity level. Allowed values are: `off`, `error`, `warn`, `info`, `debug`, and `trace`. The output log level defaults to the `service` section's `log_level`. |
 
-Fluent Bit can route up to 256 output plugins.
+Fluent Bit has no hard-coded limit on the number of output plugins. The routing `bitmask` is dynamically sized at startup based on the number of configured output plugins. The practical maximum depends on available system resources such as memory and file descriptors.
 
 ### Outgoing `OAuth 2.0` client credentials settings
 

administration/configuring-fluent-bit/yaml/service-section.md

@@ -45,7 +45,7 @@ The following storage-related keys can be set as children to the `storage` key:
 | Key | Description | Default Value |
 | --- | ----------- | ------------- |
 | `storage.backlog.flush_on_shutdown` | If enabled, Fluent Bit attempts to flush all backlog filesystem chunks to their destination during the shutdown process. This can help ensure data delivery before Fluent Bit stops, but can also increase shutdown time. Possible values: `off` or `on`. | `off` |
-| `storage.backlog.mem_limit` | Sets the memory allocated for storing buffered data for input plugins that use filesystem storage. | `5M` |
+| `storage.backlog.mem_limit` | Sets the memory limit used by the `storage_backlog` input plugin when promoting backlog chunks (filesystem chunks left over from a previous Fluent Bit run) back into memory so they can be flushed by output plugins. While the up chunks owned by `storage_backlog` consume less memory than this limit, Fluent Bit continues to promote additional backlog chunks. This setting doesn't cap memory use for other input plugins that use filesystem storage. | `5M` |
 | `storage.checksum` | Enables data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm. Possible values: `off` or `on`. | `off` |
 | `storage.delete_irrecoverable_chunks` | If enabled, deletes irrecoverable chunks during runtime and at startup. Possible values: `off` or `on`. | `off` |
 | `storage.inherit` | If enabled, input plugins that don't explicitly set `storage.type` will inherit the global `storage.type` value. Possible values: `off` or `on`. | `off` |