Move self-monitoring layers and private images to serverless-testing (093468662994) (#1183)

## Overview

Move both self-monitoring **layers** and **container images** from the
sandbox account (`425362996713`) to the serverless-testing account
(`093468662994`), where the LOD/LMI self-monitoring runtimes live.
Eliminates cross-account ECR pulls during CDK Docker builds *and* the
cross-account Lambda layer query in the image build script.

After this PR, the self-monitoring test artifacts are entirely
self-contained in `093468662994`. The regular `publish layer sandbox`
and `publish layer prod` jobs are untouched.

### Changes
- **`environments.yaml`**: add `serverless_testing` environment for
`093468662994` (assumes role `layer-deployer`, externalId
`serverless-testing-publish-externalid`, mirrors
`automatically_bump_version: 1` / `add_layer_version_permissions: 0`
from sandbox)
- **`pipeline.yaml.tpl`** — two job changes:
- `publish private images`: switch from `sandbox` env →
`serverless_testing` env (push to new ECR)
- `publish layer [self-monitoring]`: switch from `sandbox` env →
`serverless_testing` env (publish Datadog-Extension layer to
`093468662994` in us-east-1 + us-west-2)
- **`build_private_image.sh`**:
- Push to
`093468662994.dkr.ecr.us-east-1.amazonaws.com/datadog-lambda-extension`
(parameterizable via `PRIVATE_IMAGE_ECR_ACCOUNT` /
`PRIVATE_IMAGE_ECR_REPO`)
- Drop the cross-account `arn:aws:lambda:us-east-1:425362996713:layer:…`
lookup. Query the same account we publish to — works because `publish
layer [self-monitoring]` now lives in that account too.

### Prerequisites
- ECR repo `datadog-lambda-extension` in `093468662994` — created by
[`serverless-self-monitoring#637`](DataDog/serverless-self-monitoring#637)
(LVU CDK), already deployed manually
- IAM role `layer-deployer` in `093468662994` with Lambda layer publish
+ ECR push perms — created by
[`cloud-inventory#59058`](DataDog/cloud-inventory#59058),
already merged
- Vault key `serverless-testing-publish-externalid` at
`kv/k8s/gitlab-runner/datadog-lambda-extension/secrets` — created
manually

### Knock-on for serverless-self-monitoring
Layer-version-updater (`latest-dev.json`) currently pins
`Datadog-Extension` to
`arn:aws:lambda:us-east-1:425362996713:layer:Datadog-Extension:…`. After
this PR's first run, the next "self-monitoring" extension layer is
published to `093468662994` instead — LVU will need to learn to query
`093468662994` for the Datadog-Extension dev layer. Tracked as a
follow-up; safe because the existing `425362996713` layers don't
disappear, they just stop receiving new versions from the
`[self-monitoring]` job.

## Testing
- [ ] Generated pipeline YAML has `serverless_testing` environment for
both `publish private images` and `publish layer [self-monitoring]`
- [ ] Trigger manual `publish layer [self-monitoring]` on a test
pipeline → confirm Datadog-Extension layer published in `093468662994`
(us-east-1, us-west-2)
- [ ] Trigger manual `publish private images` on the same pipeline →
confirm image pushed to
`093468662994/datadog-lambda-extension:<VERSION>` with version matching
the layer just published
- [ ] Verify LOD/LMI can pull from `093468662994` ECR during CDK deploy

Author: duncanista

.gitlab/datasources/environments.yaml

@@ -5,6 +5,15 @@ environments:
     account: 425362996713
     add_layer_version_permissions: 0
     automatically_bump_version: 1
+  serverless_testing:
+    external_id: serverless-testing-publish-externalid
+    role_to_assume: layer-deployer
+    # Quoted: account starts with leading zero. Without quotes the YAML/gomplate
+    # pipeline parses it as a float and substitutes "9.3468662994e+10", which
+    # then forms an invalid role ARN in get_secrets.sh.
+    account: "093468662994"
+    add_layer_version_permissions: 0
+    automatically_bump_version: 1
   prod:
     external_id: prod-publish-externalid
     role_to_assume: dd-serverless-layer-deployer-role

.gitlab/scripts/build_private_image.sh

@@ -7,17 +7,17 @@
 
 set -e
 
-DOCKER_TARGET_IMAGE="425362996713.dkr.ecr.us-east-1.amazonaws.com/self-monitoring-lambda-extension"
+# ECR target for private extension images, used by self-monitoring container runtimes.
+# Defaults to the serverless-testing account's datadog-lambda-extension repo.
+PRIVATE_IMAGE_ECR_ACCOUNT="${PRIVATE_IMAGE_ECR_ACCOUNT:-093468662994}"
+PRIVATE_IMAGE_ECR_REPO="${PRIVATE_IMAGE_ECR_REPO:-datadog-lambda-extension}"
+DOCKER_TARGET_IMAGE="${PRIVATE_IMAGE_ECR_ACCOUNT}.dkr.ecr.us-east-1.amazonaws.com/${PRIVATE_IMAGE_ECR_REPO}"
 EXTENSION_DIR=".layers"
 IMAGE_TAG="latest"
 
-printf "Authenticating Docker to ECR...\n"
-aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 425362996713.dkr.ecr.us-east-1.amazonaws.com
+printf "Authenticating Docker to ECR (%s)...\n" "$PRIVATE_IMAGE_ECR_ACCOUNT"
+aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin "${PRIVATE_IMAGE_ECR_ACCOUNT}.dkr.ecr.us-east-1.amazonaws.com"
 
-# NOTE: this probably does not work the way that we expect it to, especially
-# when suffixes are involved. This is a known bug but we don't really check
-# anything other than the basic `self-monitoring-lambda-extension:latest` image
-# in our self-monitoring, so it's not a thing we're going to fix right now.
 LAYER_NAME="Datadog-Extension"
 if [ -z "$PIPELINE_LAYER_SUFFIX" ]; then
     printf "Building container images tagged without suffix\n"
@@ -27,7 +27,7 @@ else
 fi
 
 # Increment last version
-latest_version=$(aws lambda list-layer-versions --region us-east-1 --layer-name $LAYER_NAME --query 'LayerVersions[0].Version || `0`')
+latest_version=$(aws lambda list-layer-versions --region us-east-1 --layer-name "$LAYER_NAME" --query 'LayerVersions[0].Version || `0`')
 VERSION=$(($latest_version + 1))
 printf "Tagging container image with version: $VERSION and latest\n"
 

.gitlab/scripts/get_secrets.sh

@@ -33,10 +33,13 @@ export DD_APP_KEY=$(vault kv get -field=dd-app-key kv/k8s/gitlab-runner/datadog-
 
 printf "Assuming role...\n"
 
+# Use --external-id= (with =) so the value is one argument and not parsed as
+# a separate option. Matters when the externalId starts with '-' (which the
+# layer-deployer role's externalId does).
 export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
     $(aws sts assume-role \
     --role-arn "arn:aws:iam::$AWS_ACCOUNT:role/$ROLE_TO_ASSUME"  \
     --role-session-name "ci.datadog-lambda-extension-$CI_JOB_ID-$CI_JOB_STAGE" \
     --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
-    --external-id $EXTERNAL_ID \
+    --external-id="$EXTERNAL_ID" \
     --output text))

.gitlab/templates/pipeline.yaml.tpl

@@ -198,6 +198,11 @@ publish layer {{ $environment_name }} ({{ $flavor.name }}):
 
 {{ end }} # end environments
 
+# Self-monitoring layer publishing — split by region/account inside one matrix:
+#   us-east-1 → serverless-testing account (093468662994), used by LOD/LMI
+#   us-west-2 → sandbox account (425362996713), used by E2E tests
+# Region-specific env values are injected per matrix row so the parallel jobs
+# still collapse into a single group in the GitLab UI.
 publish layer [self-monitoring] ({{ $flavor.name }}):
   stage: self-monitoring
   tags: ["arch:amd64"]
@@ -207,22 +212,32 @@ publish layer [self-monitoring] ({{ $flavor.name }}):
       allow_failure: true
   parallel:
     matrix:
-      - REGION: us-east-1 # Self Monitoring
-      - REGION: us-west-2 # E2E Testing
+      {{ with $environment := (ds "environments").environments.serverless_testing }}
+      - REGION: us-east-1
+        EXTERNAL_ID_NAME: {{ $environment.external_id }}
+        ROLE_TO_ASSUME: {{ $environment.role_to_assume }}
+        AWS_ACCOUNT: "{{ $environment.account }}"
+      {{ end }}
+      {{ with $environment := (ds "environments").environments.sandbox }}
+      - REGION: us-west-2
+        EXTERNAL_ID_NAME: {{ $environment.external_id }}
+        ROLE_TO_ASSUME: {{ $environment.role_to_assume }}
+        AWS_ACCOUNT: "{{ $environment.account }}"
+      {{ end }}
   needs:
     - layer ({{ $flavor.name }})
   dependencies:
     - layer ({{ $flavor.name }})
-  {{ with $environment := (ds "environments").environments.sandbox }}
   variables:
     LAYER_NAME_BASE_SUFFIX: {{ $flavor.layer_name_base_suffix }}
     ARCHITECTURE: {{ $flavor.arch }}
     LAYER_FILE: datadog_extension-{{ $flavor.suffix }}.zip
-    ADD_LAYER_VERSION_PERMISSIONS: {{ $environment.add_layer_version_permissions }}
-    AUTOMATICALLY_BUMP_VERSION: {{ $environment.automatically_bump_version }}
+    # Both target environments agree on these flags; if they ever diverge,
+    # move them into the matrix above alongside the account/role.
+    ADD_LAYER_VERSION_PERMISSIONS: 0
+    AUTOMATICALLY_BUMP_VERSION: 1
   before_script:
-    - EXTERNAL_ID_NAME={{ $environment.external_id }} ROLE_TO_ASSUME={{ $environment.role_to_assume }} AWS_ACCOUNT={{ $environment.account }} source .gitlab/scripts/get_secrets.sh
-  {{ end }}
+    - source .gitlab/scripts/get_secrets.sh
   script:
     - .gitlab/scripts/publish_layers.sh
 
@@ -322,7 +337,7 @@ publish private images ({{ $multi_arch_image_flavor.name }}):
   variables:
     SUFFIX: {{ $multi_arch_image_flavor.suffix }}
   before_script:
-    {{ with $environment := (ds "environments").environments.sandbox }}
+    {{ with $environment := (ds "environments").environments.serverless_testing }}
     - EXTERNAL_ID_NAME={{ $environment.external_id }} ROLE_TO_ASSUME={{ $environment.role_to_assume }} AWS_ACCOUNT={{ $environment.account }} source .gitlab/scripts/get_secrets.sh
     {{ end }}
   script: