Skip to content

output: allow metrics chunks with 0 records. Skip them before flushing#10314

Draft
lecaros wants to merge 1 commit intomasterfrom
lecaros-out-splunk-skip-empty-payloads
Draft

output: allow metrics chunks with 0 records. Skip them before flushing#10314
lecaros wants to merge 1 commit intomasterfrom
lecaros-out-splunk-skip-empty-payloads

Conversation

@lecaros
Copy link
Copy Markdown
Contributor

@lecaros lecaros commented May 8, 2025 •
edited
Loading

Addresses #10313

Skips the flush if the payload is empty.
Empty payload can be the result of filtering the data. See #10313 for details.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Relevant message:

[2025/05/29 11:49:52] [debug] [output] skipping flush for event chunk with zero records.

fluent-bit -c drop.yml            
Fluent Bit v4.0.3
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____ 
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/ 


[2025/05/29 11:49:49] [ info] Configuration:
[2025/05/29 11:49:49] [ info]  flush time     | 1.000000 seconds
[2025/05/29 11:49:49] [ info]  grace          | 5 seconds
[2025/05/29 11:49:49] [ info]  daemon         | 0
[2025/05/29 11:49:49] [ info] ___________
[2025/05/29 11:49:49] [ info]  inputs:
[2025/05/29 11:49:49] [ info]      event_type
[2025/05/29 11:49:49] [ info] ___________
[2025/05/29 11:49:49] [ info]  filters:
[2025/05/29 11:49:49] [ info] ___________
[2025/05/29 11:49:49] [ info]  outputs:
[2025/05/29 11:49:49] [ info]      splunk.0
[2025/05/29 11:49:49] [ info] ___________
[2025/05/29 11:49:49] [ info]  collectors:
[2025/05/29 11:49:49] [ info] [fluent bit] version=4.0.3, commit=b491682b6d, pid=46548
[2025/05/29 11:49:49] [debug] [engine] coroutine stack size: 36864 bytes (36.0K)
[2025/05/29 11:49:49] [ info] [storage] ver=1.1.6, type=memory+filesystem, sync=normal, checksum=off, max_chunks_up=128
[2025/05/29 11:49:49] [ info] [storage] backlog input plugin: storage_backlog.1
[2025/05/29 11:49:49] [ info] [simd    ] disabled
[2025/05/29 11:49:49] [ info] [cmetrics] version=1.0.3
[2025/05/29 11:49:49] [ info] [ctraces ] version=0.6.6
[2025/05/29 11:49:49] [ info] [input:event_type:event_type.0] initializing
[2025/05/29 11:49:49] [ info] [input:event_type:event_type.0] storage_strategy='filesystem' (memory + filesystem)
[2025/05/29 11:49:49] [debug] [input:event_type:event_type.0] [thread init] initialization OK
[2025/05/29 11:49:49] [ info] [input:event_type:event_type.0] thread instance initialized
[2025/05/29 11:49:49] [debug] [event_type:event_type.0] created event channels: read=39 write=40
[2025/05/29 11:49:49] [ info] [input:storage_backlog:storage_backlog.1] initializing
[2025/05/29 11:49:49] [ info] [input:storage_backlog:storage_backlog.1] storage_strategy='memory' (memory only)
[2025/05/29 11:49:49] [debug] [storage_backlog:storage_backlog.1] created event channels: read=43 write=44
[2025/05/29 11:49:49] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 95.4M
[2025/05/29 11:49:49] [debug] [splunk:splunk.0] created event channels: read=45 write=46
[2025/05/29 11:49:49] [ info] [output:splunk:splunk.0] worker #0 started
[2025/05/29 11:49:49] [debug] [processor:metrics_selector:metrics_selector.0] action type EXCLUDE
[2025/05/29 11:49:49] [debug] [router] match rule event_type.0:splunk.0
[2025/05/29 11:49:49] [debug] [router] match rule storage_backlog.1:splunk.0
[2025/05/29 11:49:49] [ info] [sp] stream processor started
[2025/05/29 11:49:49] [ info] [input:storage_backlog:storage_backlog.1] register event_type.0/46419-1748533759.95882000.flb
[2025/05/29 11:49:50] [ info] [input:storage_backlog:storage_backlog.1] queueing event_type.0:46419-1748533759.95882000.flb
[2025/05/29 11:49:51] [debug] [input:event_type:event_type.0] metrics, ret=0
[2025/05/29 11:49:51] [ info] [input:event_type:event_type.0] [OK] collector_time
[2025/05/29 11:49:51] [debug] [task] created task=0x600000968000 id=0 OK
[2025/05/29 11:49:51] [debug] [input coro] destroy coro_id=0
[2025/05/29 11:49:51] [debug] [output:splunk:splunk.0] task_id=0 assigned to thread #0
[2025/05/29 11:49:51] [debug] [output] skipping flush for event chunk with zero records.
[2025/05/29 11:49:51] [debug] [out flush] cb_destroy coro_id=0
[2025/05/29 11:49:51] [ info] [engine] flush backlog chunk '46419-1748533759.95882000.flb' succeeded: task_id=0, input=storage_backlog.1 > output=splunk.0 (out_id=0)
[2025/05/29 11:49:51] [debug] [task] destroy task=0x600000968000 (task_id=0)
[2025/05/29 11:49:52] [debug] [task] created task=0x60000096c000 id=0 OK
[2025/05/29 11:49:52] [debug] [output:splunk:splunk.0] task_id=0 assigned to thread #0
[2025/05/29 11:49:52] [debug] [output] skipping flush for event chunk with zero records.
[2025/05/29 11:49:52] [debug] [out flush] cb_destroy coro_id=1
[2025/05/29 11:49:52] [debug] [task] destroy task=0x60000096c000 (task_id=0)
[2025/05/29 11:49:53] [debug] [input:event_type:event_type.0] metrics, ret=0
[2025/05/29 11:49:53] [ info] [input:event_type:event_type.0] [OK] collector_time
[2025/05/29 11:49:53] [debug] [input coro] destroy coro_id=1
[2025/05/29 11:49:54] [debug] [task] created task=0x600000970000 id=0 OK
[2025/05/29 11:49:54] [debug] [output:splunk:splunk.0] task_id=0 assigned to thread #0
[2025/05/29 11:49:54] [debug] [output] skipping flush for event chunk with zero records.
[2025/05/29 11:49:54] [debug] [out flush] cb_destroy coro_id=2
[2025/05/29 11:49:54] [debug] [task] destroy task=0x600000970000 (task_id=0)

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@edsiper
Copy link
Copy Markdown
Member

edsiper commented May 8, 2025

as discussed internally, if a metrics processor can potentially ship a context without metric series, it should be dropped before being ingested into the pipeline.

@lecaros
Copy link
Copy Markdown
Contributor Author

lecaros commented May 9, 2025

moving this to draft while a better solution is designed and implemented

@lecaros lecaros marked this pull request as draft May 9, 2025 22:27
@lecaros
output: allow metrics chunks with 0 records. Skip them before flushing
22bd46e 
Signed-off-by: lecaros <lecaros@calyptia.com>
@lecaros lecaros force-pushed the lecaros-out-splunk-skip-empty-payloads branch from e02803b to 22bd46e Compare May 29, 2025 15:50
@lecaros lecaros marked this pull request as ready for review May 29, 2025 15:53
@lecaros lecaros added this to the Fluent bit v4.0.3 milestone May 29, 2025
@lecaros lecaros changed the title out_splunk: skip the flush if payload is empty output: allow metrics chunks with 0 records. Skip them before flushing May 29, 2025
@lecaros lecaros marked this pull request as draft May 29, 2025 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich

@fujimotos fujimotos Awaiting requested review from fujimotos

@koleini koleini Awaiting requested review from koleini

@cosmo0920 cosmo0920 Awaiting requested review from cosmo0920 cosmo0920 will be requested when the pull request is marked ready for review cosmo0920 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

docs-required

Projects

None yet

Milestone

Fluent Bit Next

Development

Successfully merging this pull request may close these issues.

2 participants

@lecaros @edsiper