Skip to content

AWS Secrets end to end test case failure fix #5771

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
san81 wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

AWS Secrets end to end test case failure fix #5771

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b80b12d
along with the profile, setting additional AWS credentials through en…
san81 Jun 11, 2025
cb91b35
Trying to pass the credentials in a different approach
san81 Jun 11, 2025
f1f6cfb
Trying again with build time environment variables
san81 Jun 11, 2025
5933c85
enabling debugging
san81 Jun 11, 2025
7ca08d3
keeping untouched
san81 Jun 11, 2025
3cfb0ed
Trying upload test artifacts to better understand the failure reason
san81 Jun 11, 2025
e6e2caa
printing the docker inspect to make sure the docker image got created…
san81 Jun 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion .github/workflows/data-prepper-aws-secrets-e2e-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
pull_request_target:
types: [ opened, synchronize, reopened ]
workflow_dispatch:
# permission can be added at job level or workflow level
# permission can be added at job level or workflow level
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout
Expand Down Expand Up @@ -51,3 +51,12 @@ jobs:
uses: actions/checkout@v2
- name: Run basic grok with AWS secrets end-to-end tests with Gradle
run: AWS_PROFILE=default ./gradlew -PendToEndJavaVersion=${{ matrix.java }} :e2e-test:log:${{ matrix.test }}
- name: Upload Test Reports
if: always()
uses: actions/upload-artifact@v3
with:
name: test-reports-${{ matrix.java }}
path: |
**/build/reports/tests/
**/build/test-results/
retention-days: 7
16 changes: 16 additions & 0 deletions e2e-test/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,13 @@ subprojects {
else if(targetJavaVersion == 'docker') {
dependsOn ':release:docker:docker'
imageId = "${project.rootProject.name}:${project.version}"
doLast {
println "\n=== Inspecting Docker Image: ${project.rootProject.name}:${project.version} ==="
exec {
commandLine 'docker', 'inspect', "${project.rootProject.name}:${project.version}"
}
println "=== End of Docker Inspect ==="
}
}
else {
dependsOn 'createDataPrepperDockerFile'
Expand Down Expand Up @@ -145,4 +152,13 @@ subprojects {
dependencies {
testImplementation testLibs.junit.vintage
}

tasks.register('inspectDataPrepperImage') {
dependsOn dataPrepperDockerImage
doLast {
exec {
commandLine 'docker', 'inspect', "${project.rootProject.name}:${project.version}"
}
}
}
}
8 changes: 6 additions & 2 deletions e2e-test/log/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,12 +79,16 @@ logTestConfigurations.each { testConfiguration ->
exposePorts('tcp', [2021, 4900])
hostConfig.portBindings = ['2021:2021', '4900:4900']
hostConfig.binds = [
"${System.getProperty('user.home')}/.aws" : '/root/.aws',
"${System.getProperty('user.home')}/.aws" : '/root/.aws',
(project.file("src/integrationTest/resources/${testConfiguration.pipelineConfiguration}").toString()) : '/usr/share/data-prepper/pipelines/log-pipeline.yaml',
(project.file("src/integrationTest/resources/${testConfiguration.dataPrepperConfiguration}").toString()): '/usr/share/data-prepper/config/data-prepper-config.yaml'
]
envVars = [
'AWS_PROFILE' : System.getenv('AWS_PROFILE') ?: ''
'AWS_PROFILE' : System.getenv('AWS_PROFILE') ?: '',
'AWS_ACCESS_KEY_ID' : System.getenv('AWS_ACCESS_KEY_ID') ?: '',
'AWS_SECRET_ACCESS_KEY': System.getenv('AWS_SECRET_ACCESS_KEY') ?: '',
'AWS_SESSION_TOKEN' : System.getenv('AWS_SESSION_TOKEN') ?: '',
'AWS_REGION' : System.getenv('AWS_REGION') ?: 'us-east-2'
]
hostConfig.network = createDataPrepperNetwork.getNetworkName()
targetImageId dataPrepperDockerImage.imageId
Expand Down
Loading