ci: upload RUM nginx module snapshots to S3#363
Conversation
Port snapshot upload jobs from httpd-datadog and build RUM-enabled modules for every supported nginx version on every commit so they can be published to s3://rum-auto-instrumentation/nginx/.
Upload raw .so files named ngx_http_datadog_module-${ARCH}-${VERSION}.so
to s3://rum-auto-instrumentation/nginx/latest/ to match the layout
referenced by the public RUM nginx setup docs.
There was a problem hiding this comment.
Pull request overview
This PR extends the GitLab CI pipeline to publish RUM-enabled nginx module snapshot artifacts to S3, aligning with the snapshot upload workflow used in httpd-datadog and ensuring snapshots exist across all supported nginx versions.
Changes:
- Expands
build-nginx-rum-fastto build RUM modules for nginx 1.24.0 → 1.30.0 (both amd64/arm64). - Updates the RUM build to also emit per-build zip artifacts into an
upload/directory and includes that directory in job artifacts. - Adds a new
awsstage with snapshot upload jobs (branch snapshots + default-branch snapshots/latest) and a job to configure a 30-day lifecycle rule for branch snapshots; removesneeds:fromssi-buildto avoid GitLab’s 50-needs limit.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| .gitlab/ssi-package.yml | Removes needs from ssi-build to avoid exceeding GitLab dependency limits. |
| .gitlab/common.yml | Produces zip artifacts for RUM builds and exports them via upload/ artifacts. |
| .gitlab/build-and-test-fast.yml | Expands RUM build matrix and adds S3 upload + lifecycle configuration jobs under a new aws stage. |
| .gitlab-ci.yml | Adds the new aws stage to the pipeline stage list. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Move S3 accessibility verification into the main script so a HEAD failure now fails the upload job (after_script failures don't). - Skip upload-rum-branch-snapshots on tag pipelines, where $CI_COMMIT_BRANCH is empty and the build artifacts don't exist. - Restrict test-nginx-rum-fast needs to its tested matrix subset to stay under GitLab's 50-need limit (build-nginx-rum-fast now has 62 matrix entries).
🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: d99009b | Docs | Datadog PR Page | Give us feedback! |
Revert the fast-pipeline matrix expansion and the related 50-need-limit workarounds. The S3 upload jobs introduced here will publish snapshots only for the 7 versions already built by the fast pipeline; full-coverage uploads will follow in a separate change that hooks into build-nginx-rum-all.
xlamorlette-datadog
left a comment
xlamorlette-datadog
left a comment
There was a problem hiding this comment.
Thanks, looks great! Please address the few comments.
| upload-rum-snapshots-latest: | ||
| extends: .upload-rum-snapshot-template | ||
| variables: | ||
| S3_PATH: s3://rum-auto-instrumentation/nginx/snapshots/latest |
There was a problem hiding this comment.
Could we rather do a server side copy (cheaper and faster)?:
aws s3 cp --recursive s3://rum-auto-instrumentation/nginx/snapshots/pipeline-${CI_PIPELINE_ID} s3://rum-auto-instrumentation/nginx/snapshots/latest
(to be tested)
| - DD_API_KEY=$(vault kv get -field key kv/k8s/gitlab-runner/nginx-datadog/datadoghq-api-key 2>/dev/null) make coverage | ||
|
|
||
| .upload-rum-snapshot-template: | ||
| stage: aws |
There was a problem hiding this comment.
I would prefer a bit more descriptive stage name. Suggestions: aws-upload, aws-s3-upload, aws-rum-upload, aws-s3-rum-upload.
| rules: | ||
| - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH | ||
|
|
||
| configure-rum-branch-snapshot-expiry: |
There was a problem hiding this comment.
Should we also define an expiry rule for nginx/snapshots (to avoid indefinite accumulation)?
(Maybe more than 30 days?)
| script: | ||
| - DD_API_KEY=$(vault kv get -field key kv/k8s/gitlab-runner/nginx-datadog/datadoghq-api-key 2>/dev/null) make coverage | ||
|
|
||
| .upload-rum-snapshot-template: |
There was a problem hiding this comment.
Please move the jobs of this new stage in a dedicated file.
Summary
upload-rum-branch-snapshots/-snapshots/-snapshots-latest+ 30-day branch-snapshot lifecycle), under a newawsstage.build-nginx-rum-fastto every supported nginx version (1.24.0 → 1.30.0) so snapshots cover all variants on every commit..build-nginx-rumnow also emitsupload/ngx_http_datadog_module-${VERSION}-${ARCH}.zip.needs:onssi-build(mirrors existingssi-build-allworkaround) since the 62-entry matrix exceeds GitLab's 50-need limit.Test plan
upload-rum-branch-snapshotspublishes zips tos3://rum-auto-instrumentation/nginx/branch-snapshots/pipeline-<id>/master:upload-rum-snapshotsandupload-rum-snapshots-latestpopulatesnapshots/pipeline-<id>/andsnapshots/latest/