Skip to content

perf(forwarder): create scrubber, matcher, and batcher once in __init__#1081

Merged
LorisFriedel merged 1 commit into
masterfrom
loris.friedel/perf-forwarder-init-reuse
Mar 18, 2026
Merged

perf(forwarder): create scrubber, matcher, and batcher once in __init__#1081
LorisFriedel merged 1 commit into
masterfrom
loris.friedel/perf-forwarder-init-reuse

Conversation

@LorisFriedel
Copy link
Copy Markdown
Member

Summary

  • _forward_logs() created a new DatadogScrubber (compiles regex per scrubbing rule), DatadogMatcher (compiles include/exclude regex), and DatadogBatcher on every call
  • These are config-driven and stateless after construction
  • Move all three to Forwarder.__init__() as self._scrubber, self._matcher, and self._batcher
  • Since Forwarder is a container-global singleton, this also eliminates re-compilation across warm Lambda invocations
  • Also simplifies the log evaluation loop in _forward_logs and uses idiomatic emptiness checks

Test plan

  • Existing test_forwarder.py tests pass
  • Verify log forwarding with include/exclude rules works end-to-end via integration tests

🤖 Generated with Claude Code

@LorisFriedel LorisFriedel requested a review from a team as a code owner March 16, 2026 10:29
@github-actions github-actions Bot added the aws label Mar 16, 2026
@ViBiOh ViBiOh self-assigned this Mar 17, 2026
Comment thread aws/logs_monitoring/forwarder.py Outdated
Comment on lines +99 to +103
serialized = dump_event(log)
evaluated_log = log.get("message") or serialized
else:
serialized = dump_event(log)
evaluated_log = log
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 suggestion: ‏ We'll dump event to JSON even if we don't forward them with this change. Some event, especially cloudtrail, can be very large and that's why we wanted to dump only if required.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I've updated the code to defer json.dumps until after matching. For dicts with a message field (CloudTrail), it now matches on the message string directly and only serialize if the event passes the filter. This avoids wasted serialization for excluded events!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch on the serialization concern! That said, this is actually what the code did before your change — you've essentially circled back to the previous behavior, just with the matching logic split across three branches now. The original version intentionally serialized once and matched once in a single path, which I'd argue was easier to follow. 🤖

@LorisFriedel LorisFriedel requested a review from ViBiOh March 17, 2026 11:48
…der.__init__

Move DatadogScrubber, DatadogMatcher, and DatadogBatcher instantiation
from per-call in _forward_logs to __init__, avoiding repeated object
creation on every invocation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@LorisFriedel LorisFriedel force-pushed the loris.friedel/perf-forwarder-init-reuse branch from 0374eca to 1444f4c Compare March 18, 2026 08:33
@LorisFriedel LorisFriedel merged commit eae8664 into master Mar 18, 2026
10 checks passed
@LorisFriedel LorisFriedel deleted the loris.friedel/perf-forwarder-init-reuse branch March 18, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants