Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
8100bcb
test(telemetry): add isolated comprehensive e2e telemetry suite
sarayev Jul 1, 2026
3b6cf80
ci: add Telemetry E2E job to PR checks workflow
sarayev Jul 2, 2026
bcf6bc7
fix(telemetry): inherit worker stderr when NODE_DEBUG=blocks-telemetr…
sarayev Jul 2, 2026
d07e9b0
refactor(telemetry-e2e): rewrite tests to invoke real scripts instead…
sarayev Jul 2, 2026
af16467
fix: update changeset to include @aws-blocks/core patch
sarayev Jul 2, 2026
e56328c
fix(create-blocks-app): remove CI telemetry suppression to match core…
sarayev Jul 2, 2026
6a88007
test(telemetry): add console.ts script and use it in e2e test
sarayev Jul 2, 2026
d640f31
fix(telemetry-e2e): fix stderr capture timing for delivery assertions
sarayev Jul 2, 2026
9246009
fix(telemetry-e2e): add AWS creds to CI job, increase timeouts
sarayev Jul 2, 2026
fd9bc0e
fix(ci): add environment: publish for OIDC role assumption in telemet…
sarayev Jul 2, 2026
b2d1237
fix(telemetry-e2e): add diagnostic output on failure, fix projectId s…
sarayev Jul 2, 2026
be87eb9
fix(telemetry-e2e): add index.cdk.ts and fix script paths for CDK com…
sarayev Jul 2, 2026
72b41a8
fix(telemetry-e2e): run create-blocks-app from APP_ROOT so npx resolv…
sarayev Jul 2, 2026
6ee51e8
fix(ci): increase telemetry e2e job timeout to 40min for CDK deploy t…
sarayev Jul 2, 2026
cb8074a
fix(create-blocks-app): port telemetry to detached subprocess worker
sarayev Jul 2, 2026
4625de7
fix(ci): add BLOCKS_STACK_SUFFIX and cleanup step for telemetry e2e
sarayev Jul 3, 2026
a3cafa3
fix(telemetry-e2e): add stderr to sandbox SUCCESS failure diagnostic
sarayev Jul 3, 2026
634a5ce
fix(telemetry-e2e): fix create-blocks-app resolution and stack conflicts
sarayev Jul 3, 2026
d0c0ebb
fix(telemetry-e2e): add AWS credential preflight check before deploy …
sarayev Jul 3, 2026
1cb0d15
fix(telemetry-e2e): remove HOME override — use real HOME with seeded …
sarayev Jul 3, 2026
c8a1f2b
fix(telemetry-e2e): add pre-cleanup + deployOnly to match comprehensi…
sarayev Jul 3, 2026
4a8de53
fix(telemetry-e2e): scope SSM param by stack id + run create-blocks-a…
sarayev Jul 3, 2026
455af4d
fix(telemetry-e2e): drop in-script pre-cleanup destroy; raise suite t…
sarayev Jul 3, 2026
dc96860
fix(console): cross-platform, headless-safe browser open
sarayev Jul 3, 2026
b28bea6
test(telemetry-e2e): surface console command output on failure
sarayev Jul 3, 2026
70633ad
test(telemetry): add missing test categories from reference suite
sarayev Jul 3, 2026
cc66b92
test(telemetry): add missing test categories + fix test correctness i…
sarayev Jul 3, 2026
10422db
fix(telemetry-e2e): run blocks-telemetry CLI from built dist via node
sarayev Jul 3, 2026
3d4dfc1
fix(telemetry-e2e): address review findings — docs, port reliability,…
sarayev Jul 3, 2026
c5b6739
fix(telemetry-e2e): fix getNextPort — use sequential counter instead …
sarayev Jul 6, 2026
f071e54
fix(telemetry-e2e): fix vendorize npx resolution + relax delivery ass…
sarayev Jul 6, 2026
0194f96
fix(telemetry-e2e): vendorize uses process.exit — assert no telemetry…
sarayev Jul 6, 2026
6a77900
fix(telemetry-e2e): stop killing process groups — use SIGTERM on chil…
sarayev Jul 7, 2026
34a8e5c
fix(telemetry-e2e): remove detached:true from spawnDevServer
sarayev Jul 7, 2026
b175c51
fix(telemetry-e2e): restore detached:true + process group kill
sarayev Jul 7, 2026
f102023
fix(telemetry-e2e): don't kill dev server — disconnect and let it die…
sarayev Jul 7, 2026
5d0abb1
fix(dev-server): never reclaim a port owned by self or an ancestor
sarayev Jul 7, 2026
4010bcc
revert(dev-server): undo self/ancestor reclaim guard (keep PR test-only)
sarayev Jul 8, 2026
92bed51
test(telemetry-e2e): dev FAIL via privileged-port bind (no in-runner …
sarayev Jul 8, 2026
bb2b03a
Merge branch 'main' into feat/telemetry-e2e-tests-clean
sarayev Jul 10, 2026
d9c1ffb
Merge remote-tracking branch 'aws-blocks/main' into feat/telemetry-e2…
sarayev Jul 13, 2026
eb1db0a
Merge branch 'main' into feat/telemetry-e2e-tests-clean
sarayev Jul 16, 2026
88d9373
Merge branch 'main' into feat/telemetry-e2e-tests-clean
sarayev Jul 20, 2026
47ef470
chore(telemetry): address reviewer NIT/MINOR feedback
sarayev Jul 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .changeset/telemetry-e2e-isolated-tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
"@aws-blocks/core": patch
"@aws-blocks/create-blocks-app": patch
---

fix(telemetry): inherit worker stderr in debug mode; enable telemetry in CI for create-blocks-app

- When `NODE_DEBUG=blocks-telemetry` is set, the telemetry worker subprocess
inherits the parent's stderr so delivery confirmation is observable. Silent
by default.
- Remove CI telemetry suppression from create-blocks-app to match core behavior.
Telemetry is now enabled in CI (same as all other CLI commands).
- Make `console` cross-platform and headless-safe: pick the OS browser opener
(`open`/`xdg-open`/`start`), resolve the region from the environment, and treat
a missing opener (CI / remote shells) as best-effort success instead of failing.
- Add an isolated E2E telemetry test suite (`test-apps/telemetry`) that verifies
payload structure, delivery to the real endpoint, disable mechanisms, and
per-command success/failure events.
54 changes: 52 additions & 2 deletions .github/workflows/pr-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -474,13 +474,62 @@ jobs:
cd ${{ matrix.app-dir }}
npx tsx test/sandbox-destroy.ts aws-blocks/index.cdk.ts || true

e2e-telemetry:
name: Telemetry E2E
needs: [build-and-test-local, detect-changes]
if: needs.detect-changes.outputs.source-changed == 'true' && github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
timeout-minutes: 60
environment: publish
permissions:
contents: read
id-token: write
env:
BLOCKS_STACK_SUFFIX: telemetry-${{ github.event.pull_request.number || github.run_id }}-${{ github.run_attempt }}
steps:
- uses: actions/checkout@v5
with:
ref: ${{ github.event.pull_request.head.sha }}
persist-credentials: false

- uses: actions/setup-node@v5
with:
node-version-file: '.nvmrc'
cache: npm

- run: npm ci

- run: npm run build

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v6
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
aws-region: us-east-1

- name: Seed fixed e2e telemetry installation ID
shell: bash
run: |
mkdir -p "$HOME/.blocks/telemetry"
printf '%s' '00000000-0000-0000-0000-000000000e2e' > "$HOME/.blocks/telemetry/installation-id"

- name: Telemetry E2E Tests
run: npm run test:telemetry

- name: Cleanup stacks
if: always()
run: |
cd test-apps/telemetry
npx tsx aws-blocks/scripts/sandbox-destroy.ts || true
npx tsx aws-blocks/scripts/destroy.ts || true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Both destroys end in || true, so a failed teardown is swallowed. Stacks are unique per run (BLOCKS_STACK_SUFFIX: telemetry-<pr>-<attempt> + a random sandbox id), so leaks are distinct and cumulative -> ongoing AWS cost. Fail the job (or emit a warning annotation) on cleanup error, and/or add a scheduled sweeper for stale bb-telemetry-e2e-* stacks (or tag blocks:purpose=telemetry-e2e).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The best-effort teardown here mirrors the existing convention: e2e-hosting's cleanup step uses the identical if: always() + || true pattern (line 475), and the other e2e jobs rely on in-script cleanup + the separate cleanup-stacks.yml sweeper. Stacks are uniquely named per run, so a genuinely stronger story (fail-on-cleanup-error and/or a scheduled sweeper for bb-telemetry-e2e-*) is a reasonable improvement — but to stay consistent with the repo's current approach I'd do that as a follow-up (ideally across all e2e jobs, not just this one). Tracking it.


# Gate job — matches the branch protection required check name.
# Skipped E2E jobs (no source changes) are treated as passing.
# Note: Amplify interop tests live in a separate workflow
# (e2e-amplify-interop.yml) so their failures don't show red here.
build-and-test:
name: Build and Test
needs: [blocks-integrity, build-and-test-local, e2e-templates, e2e-sandbox, e2e-sandbox-vpc, e2e-production, e2e-supabase, e2e-hosting]
needs: [blocks-integrity, build-and-test-local, e2e-templates, e2e-sandbox, e2e-sandbox-vpc, e2e-production, e2e-supabase, e2e-hosting, e2e-telemetry]
runs-on: ubuntu-latest
if: always()
steps:
Expand All @@ -502,7 +551,8 @@ jobs:
[[ "${{ needs.e2e-sandbox-vpc.result }}" != "success" && "${{ needs.e2e-sandbox-vpc.result }}" != "skipped" ]] ||
[[ "${{ needs.e2e-production.result }}" != "success" && "${{ needs.e2e-production.result }}" != "skipped" ]] ||
[[ "${{ needs.e2e-supabase.result }}" != "success" && "${{ needs.e2e-supabase.result }}" != "skipped" ]] ||
[[ "${{ needs.e2e-hosting.result }}" != "success" && "${{ needs.e2e-hosting.result }}" != "skipped" ]]; then
[[ "${{ needs.e2e-hosting.result }}" != "success" && "${{ needs.e2e-hosting.result }}" != "skipped" ]] ||
[[ "${{ needs.e2e-telemetry.result }}" != "success" && "${{ needs.e2e-telemetry.result }}" != "skipped" ]]; then
echo "One or more E2E jobs failed"
exit 1
fi
Loading
Loading