Overhaul stats: Allow removing peerless torrents without affecting global torrent downloads metric

[josecelano]

Context

In certain sections of the application, we display the total number of downloads for all torrents.

• It's the number of peer downloads for all torrents.
• It includes all tracker instances (UDP and HTTP trackers).

In the current solution, we only count downloads for the torrents that are active (loaded in memory). That forces us to keep all torrents in memory forever, even if they don't have peers. The counter should not be related to which torrents are currently active.

Where this metric is exposed

API REST

stats endpoint:

curl -s "http://localhost:1212/api/v1/stats?token=MyAccessToken&format=prometheus" | grep "completed"

Output:

completed 0

metrics endpoint:

http://localhost:1212/api/v1/metrics?token=MyAccessToken&format=prometheus    

Output:

torrent_repository_torrents_downloads_total{} 3
tracker_core_persistent_torrents_downloads_total{} 3

Problems

• We have to keep all torrents in memory in order to count downloads, including inactive torrents (swarms without peers).

NOTICE: The field total_downloaded in AggregateSwarmMetadata only contains the sum of downloads for the current active swarms.

pub struct AggregateSwarmMetadata {
    pub total_downloaded: u64,
    pub total_complete: u64,
    pub total_incomplete: u64,
    pub total_torrents: u64,
}

If we allow removing peerless torrents, that value would not be equal to the total number of downloads ever for all torrents.

Solution 1

Add an in-memory cache value to the Swarms type to store the total number of downloads for all torrents, regardless of which torrents are currently in active swarms.

• Add the new total_downloaded field to Swarms.
• Increase the counter when a new download is completed.

This requires either:

1. Listen to peer events and update the counter in Swarms when a new peer completes.
2. Or return a boolean (again) in the announce method to know if the peer has completed.

Solution 2

• We keep AggregateSwarmMetadata as it is.
• We allow removing peerless torrents.
• In the API stats endpoint, we take the value from:
  • The tracker_core_persistent_torrents_downloads_total if persistent_torrent_completed_stat is enabled.
  • The torrent_repository_torrents_downloads_total if persistent_torrent_completed_stat is disabled. It will be the total since the tracker started and it will reset when the tracker is restarted.

NOTE: For scrape request, that also returns the number of downloads, it is not a problem because the number of downloads is per torrent:

cargo run -p torrust-tracker-client --bin http_tracker_client scrape http://127.0.0.1:7070 443c7602b4fde83d1154d6d9da48808418b181b6 | jq
    Finished `dev` profile [optimized + debuginfo] target(s) in 0.10s
     Running `target/debug/http_tracker_client scrape 'http://127.0.0.1:7070' 443c7602b4fde83d1154d6d9da48808418b181b6`
{
  "443c7602b4fde83d1154d6d9da48808418b181b6": {
    "complete": 2,
    "downloaded": 13, <- per torrent
    "incomplete": 0
  }
}

That number is loaded from the database if persistent_torrent_completed_stat is enabled.

NOTE:

• Rename AggregateSwarmMetadata to AggregateActiveSwarmMetadata to make it clear metadata is only for currently active swarms.

cc @da2ce7