Skip to content

Fix: _send_log_batch fails permanently when log group is managed by IaC#689

Merged
viw-test1 merged 3 commits intoaws-observability:mainfrom
thenom:issue-688-fix
Apr 21, 2026
Merged

Fix: _send_log_batch fails permanently when log group is managed by IaC#689
viw-test1 merged 3 commits intoaws-observability:mainfrom
thenom:issue-688-fix

Conversation

@thenom
Copy link
Copy Markdown
Contributor

@thenom thenom commented Mar 26, 2026

Fixes #688

Problem

When PutLogEvents fails with ResourceNotFoundException, the recovery path in _send_log_batch unconditionally calls _create_log_group_if_needed() before _create_log_stream_if_needed(). If the IAM role lacks logs:CreateLogGroup (because the log group is pre-created by Terraform/IaC), the AccessDeniedException propagates and CreateLogStream is never attempted. The EMF metrics exporter then fails permanently, logging errors every ~60s.

Fix

Reorder the recovery path to try creating the log stream first. If that succeeds (log group already exists), log group creation is skipped entirely. Only if CreateLogStream itself fails with ResourceNotFoundException (meaning the group genuinely doesn't exist) does it fall back to creating both.

Changes

_cloudwatch_log_client.py: Reordered resource creation in _send_log_batch — stream-first, group only if needed.
test_cloudwatch_log_client.py: Updated existing tests to reflect stream-first ordering, added tests for the IaC-managed log group scenario and the both-missing scenario.

@thenom thenom requested a review from a team as a code owner March 26, 2026 15:16
@viw-test1 viw-test1 self-assigned this Apr 20, 2026
@viw-test1
Copy link
Copy Markdown
Contributor

Update:

  1. Thanks for the report and the PR. The analysis makes sense.
  2. The bug is when the log group is pre-created by IaC and the role lacks logs:CreateLogGroup, the AccessDeniedException from CreateLogGroup prevents CreateLogStream from ever being attempted.
  3. I'll review the suggested PR Fix: _send_log_batch fails permanently when log group is managed by IaC #689.

Copy link
Copy Markdown
Contributor

@viw-test1 viw-test1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@viw-test1 viw-test1 added skip changelog doesn't need a CHANGELOG entry labels Apr 21, 2026
@viw-test1 viw-test1 merged commit d9bbe80 into aws-observability:main Apr 21, 2026
38 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip changelog doesn't need a CHANGELOG entry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aws-opentelemetry-distro send_log_batch fails when log group is managed by IaC

4 participants