Commit 8b878af
authored
refactor(test): restructure integration tests with tiered CI, parallel jobs, and Finch cross-platform smoke tests (#8666)
* test
* skip credential setup
* test
* Restructure integration tests: remove finch/container_runtime matrix, add test tiering, scope Docker cleanup
- Remove container_runtime matrix (docker/finch/no-container) in favor of explicit test_suite entries
- Split build tests into build-x86, build-arm64, build-x86-container, build-arm64-container
- Merge local-start1/local-start2 into local-start with -n 2 parallelism
- Add cloud-based-tests job using @pytest.mark.requires_credential marker
- Skip credential-requiring tests (SAR, layers, STS) in local-only jobs via SAM_CLI_NO_CREDENTIALS
- Scope Docker container/image cleanup to per-test-class snapshots to prevent cross-worker interference
- Extract credential setup/reset into setup_testing_resources.py and reset_testing_resources.py
- Update credential test runtimes: dotnet10, java25, python3.12, ruby3.4, nodejs22.x, provided.al2023 for Go
- Remove Free up disk space step
- Add uv availability check in Makefile init
* Fix test failures: credential markers, parallel isolation, runtime updates, schema registry lookup
- Add requires_credential marker to terraform S3 backend and layer tests
- Exclude requires_credential tests from terraform job, run in cloud-based-tests
- Remove AWS credentials from terraform job (no longer needed)
- Add Terraform install to cloud-based-tests job
- Fix java25 dir rename and pom.xml compiler version
- Add go.sum for Go STS credential test
- Remove global Docker image cleanup from WarmContainersRemoteLayers tests
- Add xdist_group markers for durable tests (port 9014) and rapid image tests
- Use --dist loadgroup for local-invoke and local-start jobs
- Fix EventBridge schema registry tests with dynamic position lookup
- Update CONTRIBUTING.md with integration test guidelines
* test
* Simplify credential handling, fix parallel isolation, split terraform jobs
- All jobs now get credentials unconditionally (remove conditional OIDC/ECR/reset)
- Use -m "not requires_credential" in build/local jobs instead of SAM_CLI_NO_CREDENTIALS env var
- Remove SKIP_CREDENTIAL_TESTS and skipIf patterns from test files
- Remove requires_credential from terraform tests (run in terraform jobs directly)
- Split terraform into terraform-build (-n 4) and terraform-local (sequential)
- Mark TestBuildCommand_LayerBuilds as requires_credential
- Add xdist groups: durable (callback, execution), remote_layers, docker_watcher
- Add TestSamPython36HelloWorldIntegrationImages to docker_images group
- Update CONTRIBUTING.md with simplified marker-only approach
* Fix verify_pulled_image runtime mismatch, delete test robustness, xdist groups
- Fix verify_pulled_image/verify_docker_container_cleanedup to use python3.11
matching the actual template runtimes (was incorrectly hardcoded to python3.12)
- Make test_delete_no_prompts_with_s3_prefix_present_zip robust against deploy
failures (CloudFormation EarlyValidation hooks)
- Add xdist groups for TestSamPython36HelloWorldIntegrationImages (docker_images),
TestLocalCallback/TestLocalExecution (durable),
WarmContainersRemoteLayers* (remote_layers),
Watching*Image*/DockerFileLocation* (docker_watcher)
* Split slow jobs, fix layer test robustness, update Node STS SDK
- Split local-start into local-start-api and local-start-lambda
- Split sync into sync-code (-n 2) and sync-watch (sequential)
- Split package-delete-deploy into package (+ delete) and deploy
- Remove Docker image cleanup from layer tests (images persist, reused)
- Add lambda_layers xdist group to all layer test classes
- Add flaky(reruns=3) at class level for TestLayerVersion
- Update Node.js STS SDK to ^3.700.0 (fix @smithy/protocol-http missing)
* Fix setup step conditions for split jobs (sync -> sync-code/sync-watch)
* Split slow jobs into 20 matrix entries for faster CI
- build-x86 -> build-x86-1 (general) + build-x86-2 (language-specific)
- build-x86-container -> build-x86-container-1 + build-x86-container-2
- build-arm64-container -> build-arm64-container-1 (non-java) + build-arm64-container-2 (java)
- terraform-local -> terraform-start-api + terraform-invoke-start-lambda
- Update all setup step conditions for new job names
- Remove samcli image cleanup from layer tests, add lambda_layers xdist group
* Use --ignore pattern for build-x86-2 and build-x86-container-2 so new test files are auto-included
* Rebalance build splits, move sync_build_in_source to sync-watch
- Move test_build_cmd_python.py and test_build_cmd_java.py to build-x86-1/container-1
- Move test_sync_build_in_source.py from sync-code to sync-watch
* Apply autofix formatting to workflow
* Release test account early for local-only jobs, mark docker image tests as requires_credential
- Add early reset step after ECR login for build/local jobs to free test account
- Skip final reset for jobs that already released early
- Mark TestSamPython36HelloWorldIntegrationImages and TestDeleteOldRapidImages as requires_credential
* Apply autofix formatting
* Merge upload_test_report into reset_testing_resources, fix warm container assertion
- Merge S3 report upload into reset_testing_resources.py (always uploads, conditionally resets)
- Use SKIP_ACCOUNT_RESET env var for local-only jobs instead of workflow condition
- Delete standalone upload_test_report.py
- Fix TestWarmContainersHandlesSigTermInterrupt to use assertGreaterEqual for container count
* Add tier1 markers, fix S3 upload role chaining, update report
- Add @pytest.mark.tier1 to 23 test classes across all feature areas
- Fix S3 report upload: use configure-aws-credentials role-chaining for OIDC->RoleA->RoleB
- Simplify upload_test_reports to use default credentials (set by workflow)
- Add tier1 markers for: durable, layers, sync, deploy, terraform, regression, callback, execution
- Update TIER1_TESTS.md with complete coverage table
* Fix: remove secrets context from if condition (not allowed in expressions)
* Fix: add missing pytest import in test_build_cmd_rust.py
* Add tier1-finch as matrix entry, move setup_finch.sh to tests/, fix rust import
- Add tier1-finch to matrix with conditional Finch setup, ECR login, and runtime
- Move setup_finch.sh from scripts/ to tests/
- Add tier1-finch to all toolchain setup conditions
- Remove separate tier1-finch job (now part of matrix)
- Fix missing pytest import in test_build_cmd_rust.py
* Fix Finch install: remove Docker packages first to avoid containerd.io conflict
* Finalize tier1 tests: method-level markers, deduplicate parameterized cases, add missing commands
- Move tier1 from class-level to method-level with dedicated test_tier1_* methods
- Add container + non-container tier1 for each runtime (Python, Java, Node, Dotnet, Rust, Provided)
- Remove duplicated parameterized cases covered by tier1 methods
- Add tier1 for: sam delete, sam package, symlink builds, layer builds, sam sync
- Fix missing pytest imports in delete and package tests
- Revert cargo-lambda to pip install
- Update CONTRIBUTING.md and TIER1_TESTS.md
* Fix tier1 test errors, add ARM64 build tests for all runtimes
- Fix tier1 methods that called parameterized methods (inline logic instead)
- Fix dotnet validate_build_command params (mode=None, use_container=True)
- Fix Python tier1 skip condition to check template and codeuri
- Add ARM64 tier1 build tests: Python, Java, Node, Ruby, Provided, Rust
- Update TIER1_TESTS.md with ARM64 section
* Fix tier1 failures: restore Rust container test, fix dotnet/layer params, skip ARM64 without Docker
- Restore test_tier1_rust_build_in_container (was accidentally deleted)
- Fix dotnet container tier1 to use dotnet8 (dotnet10 container image not available)
- Fix layer tier1 to use correct overrides (LayerContentUri, python3.11)
- Add skipIf(SKIP_DOCKER_TESTS) to ARM64 tier1 tests (need Docker for invoke)
- All container tier1 methods have _in_container suffix for -k filter compatibility
* Fix tier1 tests: match exact params from original parameterized tests
- Dotnet container: use mount_mode=MountMode.WRITE (not use_container=True)
- Layer build: use python3.12 (matching original test_build_single_layer)
- Rust: remove container tier1 (cargo-lambda doesn't support container builds)
- These match the exact parameter combinations that pass in the original tests
* Final cleanup: fix ARM64 naming, QEMU for build-arm64, mypy import, remove temp docs
- Revert ARM64 tier1 names (no _in_container since use_container=False)
- Enable QEMU for build-arm64 job (needed for local invoke)
- Fix mypy import error in setup_testing_resources.py
- Remove Rust container tier1 (cargo-lambda doesn't support container builds)
- Fix dotnet container tier1 to use MountMode.WRITE
- Fix layer tier1 to use python3.12
- Remove CLOUD_VS_LOCAL_REPORT.md and TIER1_TESTS.md
* Skip credentials for non-credential jobs, clear AWS creds after ECR login, add skipIf to dotnet tier1
* Add disk cleanup, mark credential-requiring tests, skip creds for local jobs
- Add tests/free_disk_space.sh: cleanup if <25GB free, reduce swap to 1GB, nohup rm
- Mark git function tests and WarmContainersRemoteLayers as requires_credential
- Skip Get testing resources for non-credential jobs, clear AWS creds after ECR login
- Add skipIf(SKIP_DOCKER_TESTS) to dotnet tier1 non-container test
* remove duplicates
* remove duplicate, add arm64 test with container
* remove dup
* add paramter back, add tier1 extra
* extra
* change temp to mnt if exist1 parent 216c71f commit 8b878af
50 files changed
Lines changed: 1121 additions & 465 deletions
File tree
- .github/workflows
- tests
- integration
- buildcmd
- delete
- deploy
- init
- schemas
- local
- callback
- execution
- generate_event
- invoke
- start_api
- start_lambda
- package
- sync
- testdata/invoke/credential_tests
- incontainer
- java25
- src/main/java/sts
- nodejs
- ruby
- inprocess
- dotnet/STS
- golang
- validate
- regression/package
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
63 | 127 | | |
64 | 128 | | |
65 | 129 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
| 10 | + | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
1 | 4 | | |
2 | 5 | | |
3 | 6 | | |
4 | 7 | | |
5 | 8 | | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
282 | 282 | | |
283 | 283 | | |
284 | 284 | | |
| 285 | + | |
285 | 286 | | |
286 | 287 | | |
287 | 288 | | |
| |||
509 | 510 | | |
510 | 511 | | |
511 | 512 | | |
| 513 | + | |
512 | 514 | | |
513 | 515 | | |
514 | 516 | | |
| |||
529 | 531 | | |
530 | 532 | | |
531 | 533 | | |
| 534 | + | |
532 | 535 | | |
| 536 | + | |
533 | 537 | | |
534 | 538 | | |
535 | 539 | | |
| |||
812 | 816 | | |
813 | 817 | | |
814 | 818 | | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
815 | 824 | | |
816 | 825 | | |
817 | 826 | | |
| |||
1416 | 1425 | | |
1417 | 1426 | | |
1418 | 1427 | | |
1419 | | - | |
1420 | | - | |
| 1428 | + | |
| 1429 | + | |
1421 | 1430 | | |
1422 | 1431 | | |
1423 | 1432 | | |
| |||
1466 | 1475 | | |
1467 | 1476 | | |
1468 | 1477 | | |
1469 | | - | |
1470 | | - | |
| 1478 | + | |
| 1479 | + | |
1471 | 1480 | | |
1472 | 1481 | | |
1473 | 1482 | | |
| |||
1515 | 1524 | | |
1516 | 1525 | | |
1517 | 1526 | | |
1518 | | - | |
1519 | | - | |
| 1527 | + | |
| 1528 | + | |
1520 | 1529 | | |
1521 | 1530 | | |
1522 | 1531 | | |
| |||
1558 | 1567 | | |
1559 | 1568 | | |
1560 | 1569 | | |
| 1570 | + | |
1561 | 1571 | | |
1562 | 1572 | | |
1563 | 1573 | | |
| |||
1676 | 1686 | | |
1677 | 1687 | | |
1678 | 1688 | | |
| 1689 | + | |
1679 | 1690 | | |
1680 | 1691 | | |
1681 | 1692 | | |
| |||
1852 | 1863 | | |
1853 | 1864 | | |
1854 | 1865 | | |
1855 | | - | |
| 1866 | + | |
1856 | 1867 | | |
1857 | 1868 | | |
1858 | 1869 | | |
| |||
1982 | 1993 | | |
1983 | 1994 | | |
1984 | 1995 | | |
| 1996 | + | |
1985 | 1997 | | |
1986 | 1998 | | |
1987 | 1999 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
70 | 83 | | |
71 | 84 | | |
72 | 85 | | |
| |||
89 | 102 | | |
90 | 103 | | |
91 | 104 | | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
92 | 118 | | |
93 | 119 | | |
94 | 120 | | |
| |||
186 | 212 | | |
187 | 213 | | |
188 | 214 | | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
189 | 228 | | |
190 | 229 | | |
191 | 230 | | |
| |||
529 | 568 | | |
530 | 569 | | |
531 | 570 | | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
532 | 592 | | |
533 | 593 | | |
534 | 594 | | |
| |||
605 | 665 | | |
606 | 666 | | |
607 | 667 | | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
608 | 674 | | |
609 | 675 | | |
610 | 676 | | |
| |||
632 | 698 | | |
633 | 699 | | |
634 | 700 | | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
0 commit comments