Commit e6a3b65
authored
Fix/e2e reliability (#3430)
#### Description of changes
Comprehensive E2E test reliability improvements to reduce flaky failures
across the test suite. Changes target six categories of transient
failures:
**1. Credential expiration (`ExpiredToken`) — 8 files**
Added `tryScheduleCredentialRefresh()` calls in `beforeAll` hooks for
CDK-based construct tests that run long enough for STS credentials to
expire. Affected tests: GSI projection type, log config, DDB IAM access,
data construct, utils, references migration, base migration, and deploy
velocity.
**2. Timeout increases for long-running CDK operations — 12 files**
- Increased migration test timeouts from 20 → 30 minutes (references,
base, many-to-many migrations).
- Increased `cdkDeploy` `noOutputTimeout` for `amplify-table-1` and
`amplify-table-5` tests from 15 → 30 minutes. These tests perform
DynamoDB GSI rename/billing mode changes and attribute type changes that
produce no stdout for >15 min, triggering the default timeout.
- Bumped deploy velocity thresholds: single-GSI tests from 10 → 20
minutes, 3-GSI tests from 30 → 45 minutes. Actual deploy durations were
exceeding the previous thresholds by small margins.
- Added `DURATION_15_MINUTES` constant to duration-constants.
**3. Memory (OOM) mitigation — `e2e_workflow.yml`**
Added `MAX_WORKERS: '3'` to all remaining build groups in the E2E
workflow to prevent OOM kills during parallel test execution.
**4. npm install transient failure resilience — `shared-scripts.sh`**
Added `retry_with_backoff` function (3 attempts with 15s/30s/45s
backoff) wrapping `npm install -g @aws-amplify/cli-internal` in
`_installCLIFromLocalRegistry`. Addresses occasional npm registry
connectivity failures during E2E setup.
**5. SQL test stabilization — 2 files**
- Changed `postDeployWaitMs` from `0` to `ONE_MINUTE` in
`sql-custom-ssl.test.ts` to allow deployed resources to stabilize before
running assertions.
- Increased sleep after RDS PostgreSQL schema generation from 30s to 45s
in `rds-v2-tests.ts` to account for schema propagation delays.
**6. RUN_SOLO regex fix — `split-e2e-tests.ts`**
Fixed the deploy-velocity `RUN_SOLO` regex to also match tests in the
`deploy-velocity-temporarily-disabled` directory.
#### CDK / CloudFormation Parameters Changed
N/A — no CDK or CloudFormation parameter changes. All changes are to
test infrastructure, timeouts, and E2E workflow configuration.
#### Issue #, if available
N/A — addresses multiple transient E2E test failures observed across CI
runs.
#### Description of how you validated changes
Validated with multiple full E2E suite runs (181 tests each). Final
validation achieved **back-to-back 181/181 clean runs** with zero
failures on first attempt:
Additional validation runs during iterative development (13 total
batches) confirmed each fix independently.
#### Checklist
- [x] PR description included
- [x] `yarn test` passes
- [x] [E2E test run
linked](https://us-east-1.console.aws.amazon.com/codesuite/codebuild/594813022831/projects/amplify-category-api-e2e-workflow/batch/amplify-category-api-e2e-workflow:238f44cb-c70a-4646-aefa-ef26e9162a48)
— 180/181 (1 unrelated `conversation` test failure)
- [x] Tests are changed or added
- [ ] Relevant documentation is changed or added (and target authors are
notified)
- [x] No new AWS SDK calls or CloudFormation actions added
- [x] No CDK or CloudFormation parameter changes
By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.25 files changed
Lines changed: 157 additions & 116 deletions
File tree
- codebuild_specs
- packages/amplify-graphql-api-construct-tests/src
- __tests__
- deploy-velocity-temporarily-disabled
- deploy-velocity
- migration
- sql-custom-ssl
- utils
- scripts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
769 | 769 | | |
770 | 770 | | |
771 | 771 | | |
| 772 | + | |
772 | 773 | | |
773 | 774 | | |
774 | 775 | | |
| |||
794 | 795 | | |
795 | 796 | | |
796 | 797 | | |
| 798 | + | |
797 | 799 | | |
798 | 800 | | |
799 | 801 | | |
| |||
806 | 808 | | |
807 | 809 | | |
808 | 810 | | |
| 811 | + | |
809 | 812 | | |
810 | 813 | | |
811 | 814 | | |
| |||
Lines changed: 53 additions & 49 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
41 | 43 | | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
53 | 55 | | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
63 | 65 | | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
78 | 82 | | |
Lines changed: 38 additions & 33 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | | - | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
38 | 41 | | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
59 | 64 | | |
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| |||
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
| 16 | + | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
| 16 | + | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
| 16 | + | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| |||
0 commit comments