Skip to content

[logstash-input] Handle connection errors correctly#536

Open
ash-darin wants to merge 5 commits into
logstash-plugins:mainfrom
ash-darin:patch-2
Open

[logstash-input] Handle connection errors correctly#536
ash-darin wants to merge 5 commits into
logstash-plugins:mainfrom
ash-darin:patch-2

Conversation

@ash-darin

@ash-darin ash-darin commented Apr 14, 2026

Copy link
Copy Markdown

Current handling of connection errors results in a log message that the case is handled, followed by an error and stack trace because it is not handled.

This change handles this by correcting the logging to indicate when an Error is handled and when it isn't.

Here is an example of connection resets that are not handled because the are not as the expected ones, observed in the wild:

[2026-04-14T10:01:18,143][INFO ][org.logstash.beats.BeatsHandler][pipeline-name][0091cc855ef1087f43a697aa15a65059dde76fb10db72eac9d82c3a650a9376e] [local: 192.168.0.2:5001, remote: 192.168.0.11:36316] Handling exception: java.net.SocketException: Connection reset (caused by: java.net.SocketException: Connection reset)
[2026-04-14T10:01:18,143][WARN ][io.netty.channel.DefaultChannelPipeline][pipeline-name][0091cc855ef1087f43a697aa15a65059dde76fb10db72eac9d82c3a650a9376e] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.net.SocketException: Connection reset
[..]

Added 07.2026: The above error is caused by winlogbeat 8.17.10. It is unclear why as I have found no reference to it anywhere and goes away with an update to > 8.18.0 Might be an artifact of virtualized machines. Removed original handling of this error from PR as it's fixed by a client upgrade.

P.S. Please backport to 8.x

Comment thread src/main/java/org/logstash/beats/BeatsHandler.java
Comment thread src/main/java/org/logstash/beats/BeatsHandler.java
Comment thread src/main/java/org/logstash/beats/BeatsHandler.java
Comment thread src/main/java/org/logstash/beats/BeatsHandler.java Outdated
Comment thread src/main/java/org/logstash/beats/BeatsHandler.java Outdated
@ash-darin ash-darin marked this pull request as ready for review April 14, 2026 10:56
@robbavey robbavey requested a review from Copilot April 16, 2026 20:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Beats input’s Netty exception handling to better classify/handle connection reset scenarios and to log handled vs. unhandled exceptions more clearly, avoiding misleading “handled” logs followed by Netty pipeline warnings.

Changes:

  • Adjusts exceptionCaught logging to log “Handling exception” only for handled RejectedExecutionExceptions, and “Unhandled exception” when forwarding to Netty.
  • Extends noisy-exception detection to treat SocketException: Connection reset as a connection reset case.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/main/java/org/logstash/beats/BeatsHandler.java Outdated
Comment thread src/main/java/org/logstash/beats/BeatsHandler.java Outdated
Comment thread src/main/java/org/logstash/beats/BeatsHandler.java Outdated
@robbavey robbavey self-assigned this May 1, 2026
@ash-darin

Copy link
Copy Markdown
Author

Hi @robbavey ,

am I missing anything to move this forward? I do not know which other files changed or prerequisites should be fulfilled for this. Thank you.

The observed "socket excetions" are/were a bug of winlogbeat 8.17.10 at least. No mention of it was discovered anywhere and the problems went away with 8.18.8 or newer versions. As it is hard to replicate (might be an artifact of virtualization) remove the special handling and only keep correcting the else if issue.
@ash-darin ash-darin changed the title [logstash-input] Handle connection resets correctly [logstash-input] Handle connection errors correctly Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants