Skip to content

Temporary solution to handle lagged stats listeners#1451

Merged
josecelano merged 2 commits into
torrust:developfrom
josecelano:1449-overhaul-stats-handle-lagging-in-event-listeners-channels
Apr 14, 2025
Merged

Temporary solution to handle lagged stats listeners#1451
josecelano merged 2 commits into
torrust:developfrom
josecelano:1449-overhaul-stats-handle-lagging-in-event-listeners-channels

Conversation

@josecelano
Copy link
Copy Markdown
Member

Relates to: #1449

When I switched the channels for statistics from tokio mpsc to tokio broadcast, I introduced a bug that was only revealed after deploying the demo tracker.

The problem is that MPSC provides backpressure, so all events are processed. However, the broadcast channel has a limit, and listeners are informed with a Lagged error when the limit has been reached and events have been deleted.

We need to decide if:

  1. We go back to mpsc channel
  2. Or we implement another solution
  3. Or we accept imprecise metrics on a high load

In the meantime, I have increased the channel capacity to be able to collect all events at the current demo peak load. That should avoid the problem of loosing events in practice.

I have also fixed the listener to handle the Lagged error. In this case, we don't have to break the loop but continue processing events to update metrics. The only difference is metrics will not be precise after this error happens because some events were lost.

for events channel to avoid lagged listeners.

This does not fix the problem. Stats listener migth loose events and
keep imprecise metrics.
If the stats listener is lagged we continue processing new events even
if metrics become imprecise after it.

This is a temporary solution. See
torrust#1449.
@josecelano
Copy link
Copy Markdown
Member Author

ACK 6fdbc47

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 14, 2025

Codecov Report

Attention: Patch coverage is 31.57895% with 13 lines in your changes missing coverage. Please review.

Project coverage is 84.51%. Comparing base (30684b5) to head (6fdbc47).
Report is 3 commits behind head on develop.

Files with missing lines Patch % Lines
...http-tracker-core/src/statistics/event/listener.rs 33.33% 3 Missing and 1 partial ⚠️
.../udp-tracker-core/src/statistics/event/listener.rs 33.33% 3 Missing and 1 partial ⚠️
...dp-tracker-server/src/statistics/event/listener.rs 33.33% 3 Missing and 1 partial ⚠️
src/bootstrap/jobs/torrent_cleanup.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1451      +/-   ##
===========================================
- Coverage    84.52%   84.51%   -0.01%     
===========================================
  Files          254      254              
  Lines        19157    19160       +3     
  Branches     19157    19160       +3     
===========================================
+ Hits         16192    16194       +2     
- Misses        2694     2695       +1     
  Partials       271      271              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@josecelano josecelano merged commit 6ba9699 into torrust:develop Apr 14, 2025
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant