Skip to content

Commit c80079e

Browse files
ViBiOhViBiOh
authored
Merge pull request #7 from DataDog/vibioh/forwarder-v5
feat(aws): Adding new env variable for forwarder v5
1 parent 9ed5413 commit c80079e

6 files changed

Lines changed: 227 additions & 105 deletions

File tree

‎README.md‎

Lines changed: 98 additions & 91 deletions
Original file line numberDiff line numberDiff line change
@@ -21,142 +21,147 @@ For complete usage examples demonstrating different configuration scenarios, see
2121

2222
## Requirements
2323

24-
| Name | Version |
25-
|------|---------|
26-
| terraform | >= 1.9 |
27-
| aws | >= 6.0 |
24+
| Name | Version |
25+
| --------- | ------- |
26+
| terraform | >= 1.9 |
27+
| aws | >= 6.0 |
2828

2929
## Providers
3030

3131
| Name | Version |
32-
|------|---------|
33-
| aws | >= 5.0 |
32+
| ---- | ------- |
33+
| aws | >= 5.0 |
3434

3535
## Inputs
3636

3737
### Required
3838

39-
| Name | Description | Type | Default |
40-
|------|-------------|------|---------|
39+
| Name | Description | Type | Default |
40+
| ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------- |
4141
| dd_site | Datadog site to send data to. Options: `datadoghq.com`, `datadoghq.eu`, `us3.datadoghq.com`, `us5.datadoghq.com`, `ap1.datadoghq.com`, `ap2.datadoghq.com`, `ddog-gov.com` | `string` | `"datadoghq.com"` |
4242

4343
**Note**: You must provide **one** of the following for the Datadog API key:
44+
4445
- `dd_api_key` - The API key directly (will be stored in Secrets Manager)
4546
- `dd_api_key_secret_arn` - ARN of existing Secrets Manager secret containing the API key
4647
- `dd_api_key_ssm_parameter_name` - Name of SSM Parameter containing the API key
4748

4849
### AWS Configuration
49-
| Name | Description | Type | Default |
50-
|------|-------------|------|---------|
51-
| region | AWS region to deploy the Datadog Forwarder to. If empty, the forwarder will be deployed to the region set by the provider. | `string` | `null` |
50+
51+
| Name | Description | Type | Default |
52+
| ------ | -------------------------------------------------------------------------------------------------------------------------- | -------- | ------- |
53+
| region | AWS region to deploy the Datadog Forwarder to. If empty, the forwarder will be deployed to the region set by the provider. | `string` | `null` |
5254

5355
### Lambda Configuration
5456

55-
| Name | Description | Type | Default |
56-
|------|-------------|------|---------|
57-
| function_name | Lambda function name | `string` | `"DatadogForwarder"` |
58-
| memory_size | Memory size (128-3008 MB) | `number` | `1024` |
59-
| timeout | Timeout in seconds | `number` | `120` |
60-
| reserved_concurrency | Reserved concurrency | `string` | `null` |
61-
| log_retention_in_days | CloudWatch log retention | `number` | `90` |
62-
| layer_version | Version of the Datadog Forwarder Lambda layer | `string` | `"latest"` |
63-
| layer_arn | Custom layer ARN (optional) | `string` | `null` |
64-
| existing_iam_role_arn | ARN of existing IAM role. **Requires** `dd_forwarder_existing_bucket_name` and either `dd_api_key_secret_arn` or `dd_api_key_ssm_parameter_name` to avoid cross-region conflicts. | `string` | `null` |
65-
| tags | Resource tags | `map(string)` | `{}` |
57+
| Name | Description | Type | Default |
58+
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | -------------------- |
59+
| function_name | Lambda function name | `string` | `"DatadogForwarder"` |
60+
| memory_size | Memory size (128-3008 MB) | `number` | `1024` |
61+
| timeout | Timeout in seconds | `number` | `120` |
62+
| reserved_concurrency | Reserved concurrency | `string` | `null` |
63+
| log_retention_in_days | CloudWatch log retention | `number` | `90` |
64+
| layer_version | Version of the Datadog Forwarder Lambda layer | `string` | `"latest"` |
65+
| layer_arn | Custom layer ARN (optional) | `string` | `null` |
66+
| existing_iam_role_arn | ARN of existing IAM role. **Requires** `dd_forwarder_existing_bucket_name` and either `dd_api_key_secret_arn` or `dd_api_key_ssm_parameter_name` to avoid cross-region conflicts. | `string` | `null` |
67+
| tags | Resource tags | `map(string)` | `{}` |
6668

6769
### Datadog Configuration
6870

69-
| Name | Description | Type | Default |
70-
|------|-------------|------|---------|
71-
| dd_api_key | Datadog API key | `string` | `null` |
72-
| dd_api_key_secret_arn | ARN of secret storing API key | `string` | `null` |
73-
| dd_api_key_ssm_parameter_name | SSM parameter name for API key | `string` | `null` |
74-
| dd_site | Datadog site | `string` | `"datadoghq.com"` |
75-
| dd_tags | Custom tags for forwarded logs | `string` | `null` |
76-
| dd_trace_enabled | Enable trace forwarding | `bool` | `true` |
77-
| dd_enhanced_metrics | Enable enhanced Lambda metrics | `bool` | `false` |
78-
79-
### Tag Fetching
80-
81-
| Name | Description | Type | Default |
82-
|------|-------------|------|---------|
83-
| dd_fetch_lambda_tags | Fetch Lambda tags | `bool` | `null` |
84-
| dd_fetch_log_group_tags | Fetch Log Group tags | `bool` | `null` |
85-
| dd_fetch_step_functions_tags | Fetch Step Functions tags | `bool` | `null` |
86-
| dd_fetch_s3_tags | Fetch S3 bucket tags | `bool` | `null` |
71+
| Name | Description | Type | Default |
72+
| ----------------------------- | ------------------------------ | -------- | ----------------- |
73+
| dd_api_key | Datadog API key | `string` | `null` |
74+
| dd_api_key_secret_arn | ARN of secret storing API key | `string` | `null` |
75+
| dd_api_key_ssm_parameter_name | SSM parameter name for API key | `string` | `null` |
76+
| dd_site | Datadog site | `string` | `"datadoghq.com"` |
77+
| dd_tags | Custom tags for forwarded logs | `string` | `null` |
78+
| dd_trace_enabled | Enable trace forwarding | `bool` | `true` |
79+
| dd_enhanced_metrics | Enable enhanced Lambda metrics | `bool` | `false` |
80+
81+
### Tag Enrichment & Fetching
82+
83+
| Name | Description | Type | Default |
84+
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ------- |
85+
| dd_enrich_s3_tags | Enrich logs from S3 with bucket tags via Datadog backend (requires Resource Collection enabled). Mutually exclusive with `dd_fetch_s3_tags` | `bool` | `null` |
86+
| dd_enrich_cloudwatch_tags | Enrich logs from CloudWatch with log group tags via Datadog backend (requires Resource Collection enabled). Mutually exclusive with `dd_fetch_log_group_tags` | `bool` | `null` |
87+
| dd_fetch_lambda_tags | Fetch Lambda tags | `bool` | `null` |
88+
| dd_fetch_log_group_tags | **(Deprecated in favor of dd_enrich_cloudwatch_tags)** Fetch Log Group tags | `bool` | `null` |
89+
| dd_fetch_step_functions_tags | Fetch Step Functions tags | `bool` | `null` |
90+
| dd_fetch_s3_tags | **(Deprecated in favor of dd_enrich_s3_tags)** Fetch S3 bucket tags | `bool` | `null` |
8791

8892
### Log Processing
8993

90-
| Name | Description | Type | Default |
91-
|------|-------------|------|---------|
92-
| dd_forward_log | Enable log forwarding | `bool` | `null` |
93-
| dd_step_functions_trace_enabled | Enable Step Functions tracing | `bool` | `null` |
94-
| dd_use_compression | Enable log compression | `bool` | `null` |
95-
| redact_ip | Redact IP addresses | `bool` | `null` |
96-
| redact_email | Redact email addresses | `bool` | `null` |
97-
| dd_scrubbing_rule | Regex pattern for log scrubbing | `string` | `null` |
98-
| dd_scrubbing_rule_replacement | Replacement text for scrubbing | `string` | `null` |
99-
| exclude_at_match | Regex to exclude logs | `string` | `null` |
100-
| include_at_match | Regex to include only matching logs | `string` | `null` |
101-
| dd_multiline_log_regex_pattern | Regex for multiline log detection | `string` | `null` |
94+
| Name | Description | Type | Default |
95+
| ------------------------------- | ----------------------------------- | -------- | ------- |
96+
| dd_forward_log | Enable log forwarding | `bool` | `null` |
97+
| dd_step_functions_trace_enabled | Enable Step Functions tracing | `bool` | `null` |
98+
| dd_use_compression | Enable log compression | `bool` | `null` |
99+
| redact_ip | Redact IP addresses | `bool` | `null` |
100+
| redact_email | Redact email addresses | `bool` | `null` |
101+
| dd_scrubbing_rule | Regex pattern for log scrubbing | `string` | `null` |
102+
| dd_scrubbing_rule_replacement | Replacement text for scrubbing | `string` | `null` |
103+
| exclude_at_match | Regex to exclude logs | `string` | `null` |
104+
| include_at_match | Regex to include only matching logs | `string` | `null` |
105+
| dd_multiline_log_regex_pattern | Regex for multiline log detection | `string` | `null` |
102106

103107
### Network Configuration
104108

105-
| Name | Description | Type | Default |
106-
|------|-------------|------|---------|
107-
| dd_use_vpc | Deploy in VPC | `bool` | `false` |
108-
| vpc_security_group_ids | VPC Security Group IDs | `list(string)` | `[]` |
109-
| vpc_subnet_ids | VPC Subnet IDs | `list(string)` | `[]` |
110-
| dd_http_proxy_url | List of url endpoints your proxy server exposes | `string` | `null` |
111-
| dd_no_proxy | List of domain names that should be excluded from the web proxy | `string` | `null` |
112-
| dd_no_ssl | Disable SSL | `string` | `null` |
113-
| dd_url | Custom endpoint URL | `string` | `null` |
114-
| dd_port | Custom endpoint port | `string` | `null` |
115-
| dd_skip_ssl_validation | Skip SSL validation | `bool` | `null` |
109+
| Name | Description | Type | Default |
110+
| ---------------------- | --------------------------------------------------------------- | -------------- | ------- |
111+
| dd_use_vpc | Deploy in VPC | `bool` | `false` |
112+
| vpc_security_group_ids | VPC Security Group IDs | `list(string)` | `[]` |
113+
| vpc_subnet_ids | VPC Subnet IDs | `list(string)` | `[]` |
114+
| dd_http_proxy_url | List of url endpoints your proxy server exposes | `string` | `null` |
115+
| dd_no_proxy | List of domain names that should be excluded from the web proxy | `string` | `null` |
116+
| dd_no_ssl | Disable SSL | `string` | `null` |
117+
| dd_url | Custom endpoint URL | `string` | `null` |
118+
| dd_port | Custom endpoint port | `string` | `null` |
119+
| dd_skip_ssl_validation | Skip SSL validation | `bool` | `null` |
116120

117121
### Advanced Configuration
118122

119-
| Name | Description | Type | Default |
120-
|------|-------------|------|---------|
121-
| dd_compression_level | Compression level (0-9) | `string` | `null` |
122-
| dd_max_workers | Max concurrent workers | `string` | `null` |
123-
| dd_log_level | Log level | `string` | `null` |
124-
| dd_store_failed_events | Store failed events in S3 | `bool` | `null` |
125-
| dd_forwarder_bucket_name | Custom S3 bucket name | `string` | `null` |
126-
| dd_forwarder_existing_bucket_name | Existing S3 bucket name | `string` | `null` |
127-
| dd_api_url | Custom API URL | `string` | `null` |
128-
| dd_trace_intake_url | Custom trace intake URL | `string` | `null` |
129-
| additional_target_lambda_arns | Additional Lambda ARNs to invoke | `string` | `null` |
123+
| Name | Description | Type | Default |
124+
| --------------------------------- | -------------------------------- | -------- | ------- |
125+
| dd_compression_level | Compression level (0-9) | `string` | `null` |
126+
| dd_max_workers | Max concurrent workers | `string` | `null` |
127+
| dd_log_level | Log level | `string` | `null` |
128+
| dd_store_failed_events | Store failed events in S3 | `bool` | `null` |
129+
| dd_forwarder_bucket_name | Custom S3 bucket name | `string` | `null` |
130+
| dd_forwarder_existing_bucket_name | Existing S3 bucket name | `string` | `null` |
131+
| dd_api_url | Custom API URL | `string` | `null` |
132+
| dd_trace_intake_url | Custom trace intake URL | `string` | `null` |
133+
| additional_target_lambda_arns | Additional Lambda ARNs to invoke | `string` | `null` |
130134

131135
### IAM Configuration
132136

133-
| Name | Description | Type | Default |
134-
|------|-------------|------|---------|
135-
| iam_role_path | IAM role path | `string` | `"/"` |
136-
| permissions_boundary_arn | Permissions boundary ARN | `string` | `null` |
137-
| tags_cache_ttl_seconds | Tags cache TTL in seconds | `number` | `300` |
138-
| dd_forwarder_buckets_access_logs_target | Access logs target bucket | `string` | `null` |
137+
| Name | Description | Type | Default |
138+
| --------------------------------------- | ------------------------- | -------- | ------- |
139+
| iam_role_path | IAM role path | `string` | `"/"` |
140+
| permissions_boundary_arn | Permissions boundary ARN | `string` | `null` |
141+
| tags_cache_ttl_seconds | Tags cache TTL in seconds | `number` | `300` |
142+
| dd_forwarder_buckets_access_logs_target | Access logs target bucket | `string` | `null` |
139143

140144
## Boolean Variable Behavior
141145

142146
For boolean variables with `null` defaults, three states are supported:
147+
143148
- `true` → Sets environment variable to `"true"`
144149
- `false` → Sets environment variable to `"false"`
145150
- `null` (unset) → Environment variable not set (uses forwarder defaults)
146151

147152
## Outputs
148153

149-
| Name | Description |
150-
|------|-------------|
151-
| datadog_forwarder_arn | Datadog Forwarder Lambda Function ARN |
152-
| datadog_forwarder_function_name | Datadog Forwarder Lambda Function Name |
153-
| datadog_forwarder_role_arn | Forwarder IAM Role ARN |
154-
| datadog_forwarder_role_name | Forwarder IAM Role Name |
155-
| dd_api_key_secret_arn | Secrets Manager secret ARN (if created) |
156-
| forwarder_bucket_name | S3 bucket name (if created or existing) |
157-
| forwarder_bucket_arn | S3 bucket ARN (if created) |
158-
| forwarder_log_group_name | CloudWatch Log Group name |
159-
| forwarder_log_group_arn | CloudWatch Log Group ARN |
154+
| Name | Description |
155+
| ------------------------------- | --------------------------------------- |
156+
| datadog_forwarder_arn | Datadog Forwarder Lambda Function ARN |
157+
| datadog_forwarder_function_name | Datadog Forwarder Lambda Function Name |
158+
| datadog_forwarder_role_arn | Forwarder IAM Role ARN |
159+
| datadog_forwarder_role_name | Forwarder IAM Role Name |
160+
| dd_api_key_secret_arn | Secrets Manager secret ARN (if created) |
161+
| forwarder_bucket_name | S3 bucket name (if created or existing) |
162+
| forwarder_bucket_arn | S3 bucket ARN (if created) |
163+
| forwarder_log_group_name | CloudWatch Log Group name |
164+
| forwarder_log_group_arn | CloudWatch Log Group ARN |
160165

161166
## Setting up Log Forwarding
162167

@@ -262,6 +267,7 @@ module "datadog_forwarder_us_west_2" {
262267
```
263268

264269
**Requirements when using `existing_iam_role_arn`:**
270+
265271
- Must specify `dd_forwarder_existing_bucket_name` (S3 bucket accessible from all regions)
266272
- Must specify either `dd_api_key_secret_arn` or `dd_api_key_ssm_parameter_name`
267273
- Your IAM role must have appropriate permissions for resources in each target region
@@ -282,6 +288,7 @@ Enable debug logging by setting `dd_log_level = "DEBUG"` in your module configur
282288
### Monitoring
283289

284290
Monitor the forwarder using:
291+
285292
- CloudWatch Logs: `/aws/lambda/{function_name}`
286293
- CloudWatch Metrics: Lambda function metrics
287294

‎examples/basic/main.tf‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,8 @@ module "datadog_forwarder" {
2727

2828
# Optional: Custom tags for all AWS resources created by the module
2929
tags = {
30-
environment = "production"
31-
terraform = "true"
32-
dd_forwarder_name = var.function_name
30+
environment = "production"
31+
terraform = "true"
3332
}
3433
}
3534

‎examples/multi-region/main.tf‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,8 @@ module "datadog_forwarder_us_east_1" {
2828

2929
# Optional: Custom tags for all AWS resources created by the module
3030
tags = {
31-
environment = "production"
32-
terraform = "true"
33-
dd_forwarder_name = var.function_name
31+
environment = "production"
32+
terraform = "true"
3433
}
3534
}
3635

@@ -51,8 +50,7 @@ module "datadog_forwarder_us_east_2" {
5150

5251
# Optional: Custom tags for all AWS resources created by the module
5352
tags = {
54-
environment = "production"
55-
terraform = "true"
56-
dd_forwarder_name = var.function_name
53+
environment = "production"
54+
terraform = "true"
5755
}
5856
}

‎main.tf‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,8 @@ resource "aws_lambda_function" "forwarder" {
201201
# Optional environment variables
202202
{
203203
DD_TAGS = var.dd_tags
204+
DD_ENRICH_S3_TAGS = var.dd_enrich_s3_tags != null ? tostring(var.dd_enrich_s3_tags) : null
205+
DD_ENRICH_CLOUDWATCH_TAGS = var.dd_enrich_cloudwatch_tags != null ? tostring(var.dd_enrich_cloudwatch_tags) : null
204206
DD_FETCH_LAMBDA_TAGS = var.dd_fetch_lambda_tags != null ? tostring(var.dd_fetch_lambda_tags) : null
205207
DD_FETCH_LOG_GROUP_TAGS = var.dd_fetch_log_group_tags != null ? tostring(var.dd_fetch_log_group_tags) : null
206208
DD_FETCH_S3_TAGS = var.dd_fetch_s3_tags != null ? tostring(var.dd_fetch_s3_tags) : null

0 commit comments

Comments
 (0)