Skip to content

feat: log agent exporter batcher#3548

Open
jeffreylimnardy wants to merge 18 commits into
kyma-project:mainfrom
jeffreylimnardy:logagent-exporter-batcher
Open

feat: log agent exporter batcher#3548
jeffreylimnardy wants to merge 18 commits into
kyma-project:mainfrom
jeffreylimnardy:logagent-exporter-batcher

Conversation

@jeffreylimnardy
Copy link
Copy Markdown
Contributor

@jeffreylimnardy jeffreylimnardy commented May 20, 2026

Description

Changes proposed in this pull request (what was done and why):

  • Enable exporter helper batching in log agent using bytes sizer and values determined in the PoC.

Changes refer to particular issues, PRs or documents:

Traceability

  • The PR is linked to a GitHub issue.
  • The follow-up issues (if any) are linked in the Related Issues section.
  • If the change is user-facing, the documentation has been adjusted.
  • If a CRD is changed, the corresponding Busola ConfigMap has been adjusted.
  • The feature is unit-tested.
  • The feature is e2e-tested.

@jeffreylimnardy jeffreylimnardy requested a review from a team as a code owner May 20, 2026 13:18
@github-actions github-actions Bot added this to the 1.64.0 milestone May 20, 2026
@github-actions github-actions Bot added the kind/feature Categorizes issue or PR as related to a new feature. label May 20, 2026
@hyperspace-insights
Copy link
Copy Markdown
Contributor

hyperspace-insights Bot commented May 20, 2026

Control Panel

Hi, I'm an AI-powered Review Bot that helps you with summarizing and reviewing pull requests.
To interact with me, just use the following actions:

  • 📝 Summarize PR
  • 🔍 Review
  • 🗑️ Delete all bot comments and reviews

@hyperspace-insights
Copy link
Copy Markdown
Contributor

Summary

The following content is AI-generated and provides a summary of the pull request:


What Changed

Enables the OTel Collector exporter batcher (exporterhelper batching) in the log agent by configuring a bytes-based sending queue with explicit batch size and flush timeout constraints. The change also refactors the OTLPExporterConfigBuilder to accept a SendingQueue struct directly instead of a raw integer queue size, and adds factory helpers for constructing sending queue configurations.

Affected Signal Types

Logs

Key Changes
  • internal/otelcollector/config/common/types.go: Adds Sizer, Batch, SendingQueueOption types and NewSendingQueue/DisabledSendingQueue factory functions to support bytes-based exporter batcher configuration with functional options.
  • internal/otelcollector/config/common/otlpexporter_config_builder.go: Replaces the raw queueSize int parameter with a SendingQueue struct throughout the builder, removing the internal queue-construction logic.
  • internal/otelcollector/config/logagent/config_builder.go: Configures the log agent OTLP exporter with a bytes-sizer sending queue (200 MB max), a 2 MB minimum batch size, a 4 MB maximum batch size, and a 10-second flush timeout.
  • internal/otelcollector/config/metricagent/config_builder.go, internal/otelcollector/config/otlpgateway/config_builder_logs.go, internal/otelcollector/config/otlpgateway/config_builder_metrics.go, internal/otelcollector/config/otlpgateway/config_builder_traces.go: Updated call sites to use common.NewSendingQueue(queueSize) in place of the raw integer.

Notes for Reviewers

The bytes-based batcher constants (200 MB queue, 2–4 MB batch, 10 s flush) come from ADR-029 and the associated PoC. The issue notes that the exporter batcher may expose new metrics; those need to be reviewed for integration into self-monitor alerts and OPS dashboards before this is considered fully complete. E2E tests are not yet in place.

Release Notes Input

Recommended Action: Review whether the new exporter batcher metrics (exposed by the OTel Collector exporterhelper batching feature) need to be incorporated into your self-monitor alerts or OPS dashboards.

Logs: The log agent OTLP exporter now uses the exporterhelper exporter batcher with a bytes sizer. Batches are flushed when they reach 2–4 MB or after 10 seconds, replacing implicit batching from the filelog receiver's BatchingLogEmitter. The sending queue is capped at 200 MB.


  • 🔄 Regenerate and Update Summary
  • ✏️ Insert as PR Description (deletes this comment)
  • 🗑️ Delete comment
PR Bot Information

Version: 1.20.51

Copy link
Copy Markdown
Contributor

@hyperspace-insights hyperspace-insights Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR is well-structured overall — the refactoring from a raw queueSize int to a SendingQueue value type with functional options is a clean improvement, and the log agent batching configuration follows the PoC findings. Two issues were flagged: a typo in a comment, and a potential time.Duration YAML serialization bug where the FlushTimeout field may marshal as nanoseconds instead of the "10s" string the golden file and OTel Collector expect — this should be verified and likely changed to string to match the pattern used by RetryOnFailure.

PR Bot Information

Version: 1.20.51

Comment thread internal/otelcollector/config/logagent/config_builder.go
Comment thread internal/otelcollector/config/common/types.go
jeffreylimnardy and others added 2 commits May 22, 2026 09:33
Co-authored-by: hyperspace-insights[bot] <209611008+hyperspace-insights[bot]@users.noreply.github.com>
@jeffreylimnardy jeffreylimnardy modified the milestones: 1.64.0, 1.65.0 May 26, 2026
@jeffreylimnardy jeffreylimnardy linked an issue May 27, 2026 that may be closed by this pull request
2 tasks
@jeffreylimnardy jeffreylimnardy added the area/logs LogPipeline label Jun 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/logs LogPipeline kind/feature Categorizes issue or PR as related to a new feature.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable exporterbatcher in Log Agent

3 participants