add: collector transaction log retention and cleanup periodical#26496
Open
amaan14999 wants to merge 1 commit into
Open
add: collector transaction log retention and cleanup periodical#26496amaan14999 wants to merge 1 commit into
amaan14999 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Resolves #26267
Add a configurable retention threshold for collector fleet transaction log markers, with a daily periodical that purges expired entries. This prevents unbounded growth of the
fleet_transaction_logcollection.Backend changes:
collectorTransactionLogRetentionThresholdfield toCollectorsConfig(default: 30 days)(created_at, _id)index inFleetTransactionLogServiceand newdeleteOldMarkers()methodgetMinLastProcessedTxnSeq()aggregation toCollectorInstanceServiceto find the safe purge boundaryPurgeCollectorFleetTransactionLogPeriodical(daily, leader-only) that deletes markers older than the retention threshold and below the minimum active sequence numberCollectorsConfigRequestand validation inCollectorsConfigResource(must be positive, must be >= collector expiration period)CollectorMetricsSupplierFrontend changes:
collector_transaction_log_retention_thresholdto config typesTimeUnitInputfield inCollectorsSettingsform with default of 30 DAYSTests:
FleetTransactionLogServiceTest: 3 tests fordeleteOldMarkersCollectorInstanceServiceTest: 4 tests forgetMinLastProcessedTxnSeqPurgeCollectorFleetTransactionLogPeriodicalTest: 3 mock-based testsCollectorsConfigResourceTest: 3 validation tests (zero retention rejected, retention below expiration rejected, retention equal to expiration accepted)CollectorsConfigTest: 2 serialization/default testsCollectorsSettings.test.tsx: updated mock data and expected request with retention fieldMotivation and Context
The
fleet_transaction_logMongoDB collection grows unbounded as collector instances report status. Without cleanup, this collection accumulates stale markers indefinitely, consuming disk space and degrading query performance over time. This change introduces a configurable retention window so old markers are automatically purged.Fixes #26267
How Has This Been Tested?
Unit tests (passing):
PurgeCollectorFleetTransactionLogPeriodicalTest: 3/3 passedCollectorsConfigTest: 5/5 passed (including 2 new retention tests)CollectorsConfigResourceTest: 21/21 passed (including 3 new validation tests)CollectorsSettings.test.tsx: 7/7 passedIntegration tests (require Docker/TestContainers, not run locally):
FleetTransactionLogServiceTest: 3deleteOldMarkerstestsCollectorInstanceServiceTest: 4getMinLastProcessedTxnSeqtestsType checking:
tsc --noEmitpasses with zero errors.Compilation:
./mvnw compilepasses with no errors from changed files.Screenshots (if appropriate):
N/A - UI uses existing
TimeUnitInputcomponent, placed after the collector expiration field in Collectors Settings.Types of changes
Checklist: