Problem
Compaction's cleanup path (`Job.deleteOldFiles` in `internal/compaction/job.go`) iterates `j.compactedFiles` and calls `StorageBackend.Delete(ctx, fileKey)` per file. On large compaction cycles (hundreds of source files), this is one S3/Azure API call per file — wasteful and slow.
Both S3 and Azure expose batch-delete APIs:
- S3: `DeleteObjects` — up to 1000 keys per call
- Azure Blob: `BlobBatch` — up to 256 deletes per batch
Scope
Only the compaction cleanup path (`Job.deleteOldFiles`). The cluster Raft manifest has already committed the logical delete via `CommandDeleteFile`; this is pure physical cleanup.
Not affected:
- User-facing `DELETE` API (keeps all existing guardrails: enabled flag, row limits, confirmation, dry run)
- Retention policies (separate code path, different semantics)
- Local storage backend (just a loop — no batch equivalent needed)
Proposed Solution
- Add `DeleteBatch(ctx context.Context, keys []string) error` to `StorageBackend` interface.
- S3 backend: use `s3.DeleteObjects` with 1000-key chunks.
- Azure backend: use `BlobBatch` with 256-key chunks.
- Local backend: default implementation = loop over `Delete`.
- `Job.deleteOldFiles` calls `DeleteBatch(ctx, j.compactedFiles)`.
Impact
- Reduces S3 DELETE API calls by up to 1000× on large compactions
- Lowers latency of the compaction cleanup phase proportionally
- No behavior change from user's perspective — logical deletes still go through Raft
Problem
Compaction's cleanup path (`Job.deleteOldFiles` in `internal/compaction/job.go`) iterates `j.compactedFiles` and calls `StorageBackend.Delete(ctx, fileKey)` per file. On large compaction cycles (hundreds of source files), this is one S3/Azure API call per file — wasteful and slow.
Both S3 and Azure expose batch-delete APIs:
Scope
Only the compaction cleanup path (`Job.deleteOldFiles`). The cluster Raft manifest has already committed the logical delete via `CommandDeleteFile`; this is pure physical cleanup.
Not affected:
Proposed Solution
Impact