feat: add SQS queue support for failed events storage#38
Merged
LorisFriedel merged 2 commits intoMar 18, 2026
Merged
Conversation
The Datadog Forwarder Lambda (v5.3.0+, layer >= 97) supports SQS as an alternative to S3 for storing failed events. This wires the new DD_SQS_QUEUE_URL env var through the Terraform module via a new dd_sqs_queue_url variable. When set, SQS takes priority over S3 for retry storage and DD_STORE_FAILED_EVENTS is automatically enabled. The SQS queue must be user-managed (not created by the module). IAM permissions for sqs:SendMessage, sqs:ReceiveMessage, sqs:DeleteMessage, and sqs:ChangeMessageVisibility are granted on the derived queue ARN. Also fixes a pre-existing bug where dd_forwarder_buckets_access_logs_target could fail when no S3 bucket was created, and marks the dd_api_key_secret_arn output as sensitive. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove conflicting dd_api_key from sqs_with_existing_iam_role test that already uses dd_api_key_ssm_parameter_name, and fix terraform fmt alignment. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ktmq
approved these changes
Mar 18, 2026
LorisFriedel
deleted the
loris.friedel/add-sqs-queue-support-for-failed-events
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Datadog Forwarder Lambda (v5.3.0+, layer >= 97) supports SQS as an alternative to S3 for storing failed events via
DD_SQS_QUEUE_URL. This wires that capability through the Terraform module with a newdd_sqs_queue_urlvariable.When set, SQS takes priority over S3 for retry storage,
DD_STORE_FAILED_EVENTSis automatically enabled, and the S3 bucket is no longer created solely for failed events (still created if tag caching is needed). The SQS queue is user-managed — the module only configures IAM permissions and passes the URL to the Lambda.Changes
variables.tf— Newdd_sqs_queue_urlvariable with URL format validation and layer version >= 97 guarddata.tf— Updatedcreate_s3_bucketto skip S3 when SQS handles failed events; newsqs_queue_arn(derived from URL) andstore_failed_events_enabledlocalsmain.tf— Passsqs_queue_arnto IAM module; setDD_STORE_FAILED_EVENTSandDD_SQS_QUEUE_URLenv vars; update scheduled retry conditions to usestore_failed_events_enabled; fix pre-existing bug wheredd_forwarder_buckets_access_logs_targetcould fail when no S3 bucket existsmodules/iam/main.tf+variables.tf— Add SQS IAM policy statement (sqs:SendMessage,sqs:ReceiveMessage,sqs:DeleteMessage,sqs:ChangeMessageVisibility)outputs.tf— Markdd_api_key_secret_arnas sensitive (pre-existing fix, was blocking all mock_provider tests)tests/sqs_failed_events.tftest.hcl— 9 test scenarios: auto-enable, tag fetching + SQS, existing IAM role, scheduled retry, URL validation, layer version validation (old + new), S3 fallback, IAM permissionsREADME.md— Document new variable in inputs tableOBSPLTF-1040