Skip to content

Commit 89b5b01

Browse files
alistairjevansalistairjevans
committed
Initial content and readmes
1 parent da864a6 commit 89b5b01

6 files changed

Lines changed: 1713 additions & 2 deletions

File tree

‎README.md‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,18 @@
1-
# aws
2-
Contains all Better Stack CloudFormation stacks and Lambdas for AWS Integration
1+
# Better Stack AWS Integration
2+
3+
This repositoriy contains all Better Stack CloudFormation stacks and Lambdas for the Better Stack AWS Integration.
4+
5+
## Getting Started
6+
7+
To get started with Better Stack on AWS, (https://telemetry.betterstack.com/team/t0/sources/new?platform=aws)[create a new AWS Source in Better Stack Telemetry].
8+
9+
## Ingested Data
10+
11+
When you deploy our CloudFormation stack you get:
12+
13+
- Automatic ingestion of all CloudWatch metrics into Better Stack.
14+
- Support for RDS Enhanced Metrics (RDSOS)
15+
- Detection and per-log-group optional ingestion of all Cloudwatch log groups.
16+
- Automatic integration with AWS X-Ray, ingest traces into Better Stack Telemetry and view your traces.
17+
- Built-in dashboards for your AWS Services.
18+

‎cloudformation/full/README.md‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
# Better Stack CloudWatch Integration
2+
3+
Stream AWS CloudWatch metrics, logs, and optionally X-Ray traces and CloudTrail audit logs to Better Stack via Kinesis Data Firehose.
4+
5+
## Prerequisites
6+
7+
1. **Better Stack Account** - Sign up at [betterstack.com](https://betterstack.com)
8+
2. **Cluster ID** - Your Better Stack cluster identifier (e.g., `g1`, `s1234`, `acme`)
9+
3. **Source Token** - Authentication token from Better Stack
10+
4. **Source ID** - Source ID from Better Stack for resource matching
11+
5. **AWS CLI** - Configured with appropriate permissions
12+
13+
## Quick Start
14+
15+
### First Region Deployment
16+
17+
```bash
18+
aws cloudformation deploy \
19+
--template-file better-stack-full.yaml \
20+
--stack-name better-stack-full \
21+
--parameter-overrides \
22+
ClusterId=YOUR_CLUSTER_ID \
23+
SourceToken=YOUR_SOURCE_TOKEN \
24+
SourceId=YOUR_SOURCE_ID \
25+
--capabilities CAPABILITY_NAMED_IAM
26+
```
27+
28+
### Additional Regions
29+
30+
When deploying to additional regions, set `CreateGlobalResources=false` to reuse the IAM roles created in the first region:
31+
32+
```bash
33+
aws cloudformation deploy \
34+
--template-file better-stack-full.yaml \
35+
--stack-name better-stack-full \
36+
--region us-west-2 \
37+
--parameter-overrides \
38+
ClusterId=YOUR_CLUSTER_ID \
39+
SourceToken=YOUR_SOURCE_TOKEN \
40+
SourceId=YOUR_SOURCE_ID \
41+
CreateGlobalResources=false \
42+
--capabilities CAPABILITY_NAMED_IAM
43+
```
44+
45+
## Data Flows
46+
47+
| Data Type | Flow |
48+
|-----------|------|
49+
| Metrics | CloudWatch Metric Stream (JSON) -> Firehose -> Lambda (enrichment) -> Better Stack |
50+
| Logs | CloudWatch Logs -> Subscription Filter -> Firehose -> Lambda (enrichment) -> Better Stack |
51+
| Traces | X-Ray -> CloudWatch Logs (`aws/spans`) -> Subscription Filter -> Better Stack |
52+
| Audit | CloudTrail -> CloudWatch Logs -> Subscription Filter -> Better Stack |
53+
54+
## Parameters
55+
56+
### Required Parameters
57+
58+
| Parameter | Description |
59+
|-----------|-------------|
60+
| `ClusterId` | Your Better Stack cluster ID |
61+
| `SourceToken` | Better Stack source token for authentication |
62+
| `SourceId` | Better Stack source ID for resource matching |
63+
64+
### Deployment Options
65+
66+
| Parameter | Default | Description |
67+
|-----------|---------|-------------|
68+
| `CreateGlobalResources` | `true` | Create IAM roles. Set to `false` for secondary regions. |
69+
70+
### Feature Toggles
71+
72+
| Parameter | Default | Description |
73+
|-----------|---------|-------------|
74+
| `EnableTagEnrichment` | `true` | Enable Lambda-based tag enrichment for metrics and logs |
75+
| `EnableCloudTrail` | `true` | Create a new CloudTrail trail and forward to Better Stack |
76+
| `EnableXRayTransactionSearch` | `true` | Enable X-Ray Transaction Search and forward spans |
77+
78+
## CloudTrail Notes
79+
80+
Most AWS accounts already have CloudTrail enabled. Before enabling `EnableCloudTrail=true`, check for existing trails:
81+
82+
```bash
83+
aws cloudtrail describe-trails
84+
```
85+
86+
If a trail already exists, you may want to:
87+
1. Keep `EnableCloudTrail=false` and manually subscribe the existing CloudTrail log group
88+
2. Or enable it to create a dedicated trail for Better Stack
89+
90+
CloudTrail logs have a 1-day retention in CloudWatch (they are forwarded immediately to Better Stack).
91+
92+
## X-Ray Transaction Search
93+
94+
When `EnableXRayTransactionSearch=true`, the template automatically:
95+
1. Creates a CloudWatch Logs resource policy allowing X-Ray to write spans
96+
2. Enables Transaction Search via `AWS::XRay::TransactionSearchConfig`
97+
3. X-Ray sends trace segments to the `aws/spans` CloudWatch Logs group
98+
99+
No manual steps required - fully automated via CloudFormation.
100+
101+
## Better Stack Integration Role
102+
103+
The template creates a `better-stack-integration-role` IAM role that allows Better Stack to:
104+
- Read CloudWatch metrics and logs
105+
- Manage metric streams and log subscriptions
106+
- Discover AWS resources for context enrichment
107+
108+
After deployment, provide these values to Better Stack:
109+
- **Role ARN**: From stack output `IntegrationRoleArn`
110+
- **External ID**: From stack output `ExternalId`
111+
112+
## Tag Enrichment
113+
114+
When `EnableTagEnrichment=true`, Lambda functions enrich metrics and logs with AWS resource tags.
115+
116+
**Metrics enrichment** adds tags from:
117+
- EC2 instances
118+
- RDS databases
119+
- Lambda functions
120+
- DynamoDB tables
121+
- S3 buckets
122+
- ELB/ALB/NLB load balancers
123+
- SQS queues
124+
- SNS topics
125+
126+
**Logs enrichment** extracts resource info from log group/stream names:
127+
- `/aws/lambda/{function-name}` -> Lambda tags
128+
- `/aws/rds/instance/{db-instance}/{type}` -> RDS tags
129+
- `RDSOSMetrics` (Enhanced Monitoring) -> RDS tags from message body
130+
- `/ecs/{cluster}/...` -> ECS tags
131+
- `/aws/api-gateway/{api-id}` -> API Gateway tags
132+
133+
Lambda code is loaded from regional S3 buckets (`better-stack-lambda-{region}`).
134+
135+
## Stack Outputs
136+
137+
When `CreateGlobalResources=true`, the stack outputs:
138+
139+
| Output | Description |
140+
|--------|-------------|
141+
| `IntegrationRoleArn` | STS role ARN to provide to Better Stack |
142+
| `ExternalId` | External ID to provide to Better Stack |
143+
144+
Better Stack will discover all other resources (Firehose streams, Metric Streams, etc.) via the STS integration role.
145+
146+
## Troubleshooting
147+
148+
### Check Firehose Delivery Errors
149+
150+
```bash
151+
aws logs filter-log-events \
152+
--log-group-name /aws/kinesisfirehose/better-stack-metrics \
153+
--start-time $(date -d '1 hour ago' +%s000)
154+
```
155+
156+
### Check Lambda Errors
157+
158+
```bash
159+
aws logs filter-log-events \
160+
--log-group-name /aws/lambda/better-stack-metrics \
161+
--start-time $(date -d '1 hour ago' +%s000)
162+
```
163+
164+
### Verify Metric Stream Status
165+
166+
```bash
167+
aws cloudwatch get-metric-stream --name better-stack-metric-stream
168+
```
169+
170+
## Cleanup
171+
172+
To delete the stack:
173+
174+
```bash
175+
aws cloudformation delete-stack --stack-name better-stack-full
176+
```
177+
178+
**Note**: S3 buckets have `DeletionPolicy: Retain` to preserve any failed delivery data and CloudTrail logs. Delete manually if needed:
179+
180+
```bash
181+
# Delete Firehose backup bucket
182+
aws s3 rb s3://better-stack-firehose-ACCOUNT_ID-REGION --force
183+
184+
# Delete CloudTrail bucket (if CloudTrail was enabled)
185+
aws s3 rb s3://better-stack-cloudtrail-ACCOUNT_ID-REGION --force
186+
```
187+
188+
## Support
189+
190+
For issues or questions, please reach out at hello@betterstack.com

0 commit comments

Comments
 (0)