When fact's gRPC client disconnects, it will accumulate events in the intermediate channels between tasks, once those channels fill up each task will wait until the following one reads an event before being able to process the next event in the chain. This in turn can lead to the BPF worker not being able to retrieve events from its ringbuffer, therefore dropping events.
We should come up with a strategy for when a disconnect occurs, in which we can still accumulate events and replay them at a later time when the connection is reestablished. Bonus points if the approach is able to replay events across fact restarts (i.e: dump pending events to a file to be read on the next startup). What this looks like is still not clear, maybe we want to have some WAL-like file in which we can drop events and have the outputs work through it at their own pace instead of using a regular in-memory channel.
The (still under-development) replay feature (#1010) might be useful for this, though ma immediate concern with this is that the current RateLimiter implementation could potential drop some of the events we are intending to replay, so we probably want to look into this as well and push the rate limiting further down the line to the output components themselves, (e.g: since we alread include governor for the current rate limit implementation and because BroadcastStream already implements Stream, it gets access to StreamRateLimitedExt, allowing us to rate limit directly at the output).
When
fact's gRPC client disconnects, it will accumulate events in the intermediate channels between tasks, once those channels fill up each task will wait until the following one reads an event before being able to process the next event in the chain. This in turn can lead to the BPF worker not being able to retrieve events from its ringbuffer, therefore dropping events.We should come up with a strategy for when a disconnect occurs, in which we can still accumulate events and replay them at a later time when the connection is reestablished. Bonus points if the approach is able to replay events across fact restarts (i.e: dump pending events to a file to be read on the next startup). What this looks like is still not clear, maybe we want to have some WAL-like file in which we can drop events and have the outputs work through it at their own pace instead of using a regular in-memory channel.
The (still under-development) replay feature (#1010) might be useful for this, though ma immediate concern with this is that the current
RateLimiterimplementation could potential drop some of the events we are intending to replay, so we probably want to look into this as well and push the rate limiting further down the line to the output components themselves, (e.g: since we alread includegovernorfor the current rate limit implementation and becauseBroadcastStreamalready implementsStream, it gets access to StreamRateLimitedExt, allowing us to rate limit directly at the output).