Skip to content

Emit buffered logs normally on timeout instead of throwing error#148

Closed
Watson1978 wants to merge 1 commit into
masterfrom
timeout
Closed

Emit buffered logs normally on timeout instead of throwing error#148
Watson1978 wants to merge 1 commit into
masterfrom
timeout

Conversation

@Watson1978

Copy link
Copy Markdown
Contributor

Previously, when a timeout occurred while waiting for the next multiline log, the plugin raised a TimeoutError via emit_error_event.
This caused the remaining buffered logs—most notably the last line of log outputs—to be dropped or incorrectly routed.

While the plugin provides a timeout_label option to route logs on timeout, users often use it without realizing they must explicitly configure a <label> block to catch and handle the rerouted logs. Without this awareness, it simply results in silent data loss.

This commit changes the behavior to log an info message and emit the buffered record normally using router.emit when no timeout_label is set, ensuring no logs are lost by default.

Fixes #109
Fixes #124
Fixes #94
Fixes #87
Fixes #84
Fixes #63

Previously, when a timeout occurred while waiting for the next multiline log, the plugin raised a `TimeoutError` via `emit_error_event`.
This caused the remaining buffered logs—most notably the last line of log outputs—to be dropped or incorrectly routed.

While the plugin provides a `timeout_label` option to route logs on timeout, users often use it without realizing they must explicitly configure a
`<label>` block to catch and handle the rerouted logs.
Without this awareness, it simply results in silent data loss.

This commit changes the behavior to log an info message and emit the buffered record normally using `router.emit` when no `timeout_label` is set,
ensuring no logs are lost by default.

Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
@Watson1978 Watson1978 marked this pull request as ready for review June 5, 2026 01:17
@Watson1978 Watson1978 requested a review from kenhys June 5, 2026 01:17
@Watson1978 Watson1978 marked this pull request as draft June 5, 2026 01:28
@Watson1978

Copy link
Copy Markdown
Contributor Author

Investigation: Why we cannot simply change emit_error_event to router.emit

After deeply investigating this issue and attempting to fix it by replacing emit_error_event with a standard router.emit on timeout, I realized that the current behavior is not a bug, but a necessary design choice due to Fluentd's architectural limitations.

Here is the breakdown of why the last log line disappears and why we cannot simply "fix" it by emitting it normally:

1. The Async Thread Re-injection Loop

The timeout flush is triggered by an asynchronous timer thread (on_timer). In Fluentd's Filter plugin architecture, calling router.emit(tag, time, record) from an asynchronous thread does not pass the record to the "next" plugin. Instead, it re-injects the record at the very beginning of the routing pipeline.

If we use router.emit:

  • The emitted log goes back to the top of the pipeline.
  • It passes through upstream filters again (causing duplicate processing).
  • It hits the concat filter again, matches the multiline_start_regexp, and starts buffering all over again, causing an infinite loop.

2. The Original Author's Intention

To break this infinite loop, the plugin must route the timeout record to a different path. This is exactly why the plugin uses emit_error_event (which goes to the @ERROR label by default) or requires a timeout_label. It is the only safe way to flush the buffer from an async thread without breaking the pipeline.

3. The Real Issue: Documentation & UX

The core problem causing #109, #124, #94, and #84 is not a code bug, but a silent failure caused by missing documentation.
Many users do not realize that they must configure a timeout_label and handle it explicitly. Without it, the last log line is silently sent to the @ERROR label, which looks like data loss to the user.


I will close this PR for now and consider submitting a new PR focused purely on adding the warning log and updating the documentation to save future users from this trap.

@Watson1978 Watson1978 closed this Jun 5, 2026
@Watson1978 Watson1978 deleted the timeout branch June 5, 2026 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant