Skip to content

add: collector transaction log retention and cleanup periodical#26496

Open
amaan14999 wants to merge 1 commit into
Graylog2:masterfrom
amaan14999:feature/collector-txn-log-cleanup
Open

add: collector transaction log retention and cleanup periodical#26496
amaan14999 wants to merge 1 commit into
Graylog2:masterfrom
amaan14999:feature/collector-txn-log-cleanup

Conversation

@amaan14999

Copy link
Copy Markdown

Description

Resolves #26267
Add a configurable retention threshold for collector fleet transaction log markers, with a daily periodical that purges expired entries. This prevents unbounded growth of the fleet_transaction_log collection.

Backend changes:

  • Add collectorTransactionLogRetentionThreshold field to CollectorsConfig (default: 30 days)
  • Add compound (created_at, _id) index in FleetTransactionLogService and new deleteOldMarkers() method
  • Add getMinLastProcessedTxnSeq() aggregation to CollectorInstanceService to find the safe purge boundary
  • Create PurgeCollectorFleetTransactionLogPeriodical (daily, leader-only) that deletes markers older than the retention threshold and below the minimum active sequence number
  • Add retention field to CollectorsConfigRequest and validation in CollectorsConfigResource (must be positive, must be >= collector expiration period)
  • Add telemetry entry in CollectorMetricsSupplier

Frontend changes:

  • Add collector_transaction_log_retention_threshold to config types
  • Add TimeUnitInput field in CollectorsSettings form with default of 30 DAYS

Tests:

  • FleetTransactionLogServiceTest: 3 tests for deleteOldMarkers
  • CollectorInstanceServiceTest: 4 tests for getMinLastProcessedTxnSeq
  • PurgeCollectorFleetTransactionLogPeriodicalTest: 3 mock-based tests
  • CollectorsConfigResourceTest: 3 validation tests (zero retention rejected, retention below expiration rejected, retention equal to expiration accepted)
  • CollectorsConfigTest: 2 serialization/default tests
  • CollectorsSettings.test.tsx: updated mock data and expected request with retention field

Motivation and Context

The fleet_transaction_log MongoDB collection grows unbounded as collector instances report status. Without cleanup, this collection accumulates stale markers indefinitely, consuming disk space and degrading query performance over time. This change introduces a configurable retention window so old markers are automatically purged.

Fixes #26267

How Has This Been Tested?

Unit tests (passing):

  • PurgeCollectorFleetTransactionLogPeriodicalTest: 3/3 passed
  • CollectorsConfigTest: 5/5 passed (including 2 new retention tests)
  • CollectorsConfigResourceTest: 21/21 passed (including 3 new validation tests)
  • CollectorsSettings.test.tsx: 7/7 passed

Integration tests (require Docker/TestContainers, not run locally):

  • FleetTransactionLogServiceTest: 3 deleteOldMarkers tests
  • CollectorInstanceServiceTest: 4 getMinLastProcessedTxnSeq tests

Type checking: tsc --noEmit passes with zero errors.
Compilation: ./mvnw compile passes with no errors from changed files.

Screenshots (if appropriate):

N/A - UI uses existing TimeUnitInput component, placed after the collector expiration field in Collectors Settings.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Refactoring (non-breaking change)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have requested a documentation update.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.

@CLAassistant

CLAassistant commented Jun 26, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add cleanup job for Collector transaction log

2 participants