Skip to content

[AWSINTS-3516] refactor(go-forwarder)#1115

Merged
ndakkoune merged 7 commits into
nabil.dakkoune/go-forwarderfrom
nabil.dakkoune/AWSINTS-3516
May 4, 2026
Merged

[AWSINTS-3516] refactor(go-forwarder)#1115
ndakkoune merged 7 commits into
nabil.dakkoune/go-forwarderfrom
nabil.dakkoune/AWSINTS-3516

Conversation

@ndakkoune

@ndakkoune ndakkoune commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Improve performance, refactor the pipeline and log entry structures.

Motivation

Testing Guidelines

Additional Notes

  • Introduce Handler interface + registry pattern
  • Handlers are registered at startup in main
  • Pipeline selects the handler from the registry map
  • Config.Load() no longer takes context.Context
  • Removed ScrubbingConfig / FilteringConfig nested structs. Config now holds *scrubbing.Scrubber and *filtering.Filter directly (compiled at load time, fail-fast on invalid regex)
  • Tag extraction runs once at init, stored as model.Tags on Config
  • filtering.NewFiltercompiles regexes upfront + ShouldExclude nil-safe
  • scrubbing.NewScrubber compiles rules upfront + Scrub nil-safe
  • Forwarder.Forward() → Forwarder.Start() + merged with batching
  • Batcher no longer generic
  • Storage constants (CloudwatchStorage, S3Storage)
  • pipeline.Run() replaced with pipeline.NewRun() + pipeline.Start() to pass useful information we get from the invocation source
  • Kinesis handler now calls decompressCloudwatchLogs() directly instead of base64-encoding
  • New testutil package with helpers

Types of changes

  • Bug fix
  • New feature
  • Breaking change
  • Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

  • This PR's description is comprehensive
  • This PR contains breaking changes that are documented in the description
  • This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
  • This PR impacts documentation, and it has been updated (or a ticket has been logged)
  • This PR's changes are covered by the automated tests
  • This PR collects user input/sensitive content into Datadog
  • This PR passes the integration tests (ask a Datadog member to run the tests)
  • This PR passes the unit tests
  • This PR passes the installation tests (ask a Datadog member to run the tests)

@ndakkoune ndakkoune marked this pull request as ready for review April 30, 2026 15:11
@ndakkoune ndakkoune requested a review from a team as a code owner April 30, 2026 15:11
@ndakkoune ndakkoune requested review from ViBiOh and ge0Aja April 30, 2026 15:11
entries := make([]model.LogEntry, maxItemsPerBatch+1)
for i := range entries {
entries[i] = makeTestLogEntry()
entries[i] = model.NewLogEntry()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥜 nitpick: ‏The added value of the NewLogEntry is to put aws as the sourcecategory. I don't think it's strictly necessary for the test.

We could avoid doing the loop on entries, the line 49 is already allocating the slice with non-pointer struct, so struct are already allocated (empty).

Comment on lines +22 to +37
if includeMatch != "" {
re, includeErr := regexp.Compile(includeMatch)
if includeErr != nil {
err = errors.Join(err, fmt.Errorf("compile custom include: %w", includeErr))
} else {
filter.includeRegex = re
}
}
if excludeMatch != "" {
re, excludeErr := regexp.Compile(excludeMatch)
if excludeErr != nil {
err = errors.Join(err, fmt.Errorf("compile custom exclude: %w", excludeErr))
} else {
filter.excludeRegex = re
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 suggestion: ‏ Can we factorize the "check empty, then compile and return error" to avoid repetition here?

Also, add blank line between "section" of the function to make it easier to read.


batches := make(chan []byte)
batcher := batching.NewBatcher()
g.Go(func() error {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 suggestion: ‏ Usually, the name for errgroup is eg or errgroup. Long-shot feedback is to avoid one letter variable name (i is tolerated).

}
}()

var d events.CloudwatchLogsData

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here regarding one letter variable name

@ndakkoune ndakkoune merged commit b14650c into nabil.dakkoune/go-forwarder May 4, 2026
10 checks passed
@ndakkoune ndakkoune deleted the nabil.dakkoune/AWSINTS-3516 branch May 4, 2026 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants