Streams userdata log to s3 via kinesis to improve delivery reliability#211
Open
lorengordon wants to merge 2 commits into
Open
Streams userdata log to s3 via kinesis to improve delivery reliability#211lorengordon wants to merge 2 commits into
lorengordon wants to merge 2 commits into
Conversation
lorengordon
force-pushed
the
feat/kinesis-log-streaming
branch
2 times, most recently
from
401384c to
fd63011
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Pull request overview
This PR adds Kinesis Firehose-based streaming for userdata logs to improve log delivery reliability during build instances, and updates CI cleanup to flush streams before destroying resources.
Changes:
- Adds Firehose streams, IAM permissions, and outputs for per-build userdata log streaming.
- Configures Linux and Windows userdata scripts to install/start Kinesis agents and send userdata logs to Firehose.
- Updates integration workflow cleanup and narrows the tested OS matrix.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
main.tf |
Adds Firehose/IAM resources and passes stream names into userdata templates. |
outputs.tf |
Exposes Firehose stream names for workflow cleanup. |
templates/lx_userdata.sh |
Configures Linux Kinesis agent and removes per-debug S3 uploads. |
templates/win_userdata.ps1 |
Configures Windows Kinesis Tap and removes heartbeat/debug S3 uploads. |
.github/actions/test/action.yml |
Attempts to reduce Firehose buffering before Terraform destroy. |
.github/workflows/test.yml |
Reduces source and standalone integration test matrices. |
README.md |
Updates generated Terraform docs for new data sources/output. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
lorengordon
force-pushed
the
feat/kinesis-log-streaming
branch
from
fd63011 to
cd55832
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lorengordon
force-pushed
the
feat/kinesis-log-streaming
branch
from
cd55832 to
f9eea80
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lorengordon
force-pushed
the
feat/kinesis-log-streaming
branch
from
f9eea80 to
18fb7ed
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lorengordon
force-pushed
the
feat/kinesis-log-streaming
branch
from
18fb7ed to
b5882ce
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.