Skip to content

[DO NOT MERGE] CI timing experiment: parallel vs serial test legs#5802

Closed
pietern wants to merge 2 commits into
mainfrom
ci-timing-serial-defer
Closed

[DO NOT MERGE] CI timing experiment: parallel vs serial test legs#5802
pietern wants to merge 2 commits into
mainfrom
ci-timing-serial-defer

Conversation

@pietern

@pietern pietern commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Measurement-only PR. Not for merge.

Compares CI wall-clock of the test job under two configurations of the test-unit / test-acc legs:

  • Commit 1 (baseline): current parallel deps: configuration, empty commit.
  • Commit 2: sequential cmds: + defer: cat so the concatenated test-output.json is produced even when a leg fails.

Local benchmarking (16-core) showed parallel at 294s vs serial at 329s (~11%), because both legs already saturate cores. This PR measures the real CI delta before deciding whether to fix the failed-upload issue in the Taskfile (serial) or in the workflow (artifact glob).

This pull request and its description were written by Isaac.

@pietern
Baseline: measure CI test job timing (empty commit)
c6b2e23 
Empty commit to capture a baseline CI timing for the parallel
test-unit/test-acc configuration before comparing against a
sequential+defer variant.

Co-authored-by: Isaac
@pietern pietern temporarily deployed to test-trigger-is July 2, 2026 12:52 — with GitHub Actions Inactive
@pietern pietern temporarily deployed to test-trigger-is July 2, 2026 12:52 — with GitHub Actions Inactive
@pietern
Serial test legs + defer cat for test-output.json
e7bef66 
Run test-unit/test-acc (and the three test-unit sub-legs) sequentially
via cmds instead of parallel deps, with the concatenating cat in a
defer so test-output.json is produced even when a leg fails.

Measurement commit for CI timing comparison vs the parallel baseline.

Co-authored-by: Isaac
@pietern pietern temporarily deployed to test-trigger-is July 2, 2026 12:54 — with GitHub Actions Inactive
@pietern pietern temporarily deployed to test-trigger-is July 2, 2026 12:54 — with GitHub Actions Inactive
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: e7bef66

Run: 28591599370

Env ❌​FAIL 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 10 13 230 1041 4:21
🔄​ aws windows 1 10 13 231 1039 4:10
💚​ aws-ucws linux 10 13 314 959 5:09
💚​ aws-ucws windows 10 13 316 957 4:41
💚​ azure linux 4 15 230 1040 4:47
💚​ azure windows 4 15 232 1038 3:52
💚​ azure-ucws linux 4 15 316 956 5:27
💚​ azure-ucws windows 4 15 318 954 4:41
❌​ gcp linux 201 2 15 28 1031 1:53
❌​ gcp windows 203 2 15 28 1029 3:00
226 interesting tests: 203 FAIL, 13 SKIP, 8 RECOVERED, 2 KNOWN
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🟨​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K
❌​ TestAccept/bundle/deploy/mlops-stacks ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/postgres_branches/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/replace_existing 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/update_protected 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/without_branch_id 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_projects/update_display_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
❌​ TestAccept/selftest/record_cloud/basic ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/selftest/record_cloud/basic/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/selftest/record_cloud/basic/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/selftest/record_cloud/error ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/selftest/record_cloud/error/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/selftest/record_cloud/error/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/selftest/record_cloud/pipeline-crud ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/selftest/record_cloud/pipeline-crud/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/selftest/record_cloud/pipeline-crud/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/selftest/record_cloud/workspace-file-io ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/selftest/record_cloud/workspace-file-io/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAccept/selftest/record_cloud/workspace-file-io/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
❌​ TestDashboardAssumptions_WorkspaceImport ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestBundleInitErrorOnUnknownFields ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestBundleInitHelpers ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAlertsCreateErrWhenNoArguments ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestApiGet ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestAuthDescribeSuccess ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestClustersGet ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestClustersList ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCat ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCat/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCatDoesNotSupportOutputModeJson ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCatForDbfsInvalidScheme ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCatOnADir ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCatOnADir/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCatOnNonExistentFile ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCatOnNonExistentFile/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCompletion ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDir ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDir/dbfs_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDir/dbfs_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDir/dbfs_to_uc-volumes 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDir/local_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDir/local_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirFileNotOverwritten ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirFileNotOverwritten/dbfs_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirFileNotOverwritten/dbfs_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirFileNotOverwritten/dbfs_to_uc-volumes 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirFileNotOverwritten/local_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirFileNotOverwritten/local_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirWithOverwriteFlag ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirWithOverwriteFlag/dbfs_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirWithOverwriteFlag/dbfs_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirWithOverwriteFlag/dbfs_to_uc-volumes 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirWithOverwriteFlag/local_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpDirToDirWithOverwriteFlag/local_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpErrorsOnInvalidScheme ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpErrorsWhenSourceIsDirWithoutRecursiveFlag ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpErrorsWhenSourceIsDirWithoutRecursiveFlag/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDir ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDir/dbfs_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDir/dbfs_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDir/dbfs_to_uc-volumes 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDir/local_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDir/local_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirFileNotOverwritten ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirFileNotOverwritten/dbfs_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirFileNotOverwritten/dbfs_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirFileNotOverwritten/dbfs_to_uc-volumes 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirFileNotOverwritten/local_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirFileNotOverwritten/local_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirForWindowsPaths 🙈​s ✅​p 🙈​s ✅​p 🙈​s ✅​p 🙈​s ✅​p 🙈​s ❌​F
❌​ TestFsCpFileToDirWithOverwriteFlag ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirWithOverwriteFlag/dbfs_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirWithOverwriteFlag/dbfs_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirWithOverwriteFlag/dbfs_to_uc-volumes 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirWithOverwriteFlag/local_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToDirWithOverwriteFlag/local_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFile ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFile/dbfs_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFile/dbfs_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFile/dbfs_to_uc-volumes 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFile/local_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFile/local_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileFileNotOverwritten ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileFileNotOverwritten/dbfs_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileFileNotOverwritten/dbfs_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileFileNotOverwritten/dbfs_to_uc-volumes 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileFileNotOverwritten/local_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileFileNotOverwritten/local_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileWithOverwriteFlag ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileWithOverwriteFlag/dbfs_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileWithOverwriteFlag/dbfs_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileWithOverwriteFlag/dbfs_to_uc-volumes 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileWithOverwriteFlag/local_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToFileWithOverwriteFlag/local_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToNonExistentDir ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToNonExistentDir/dbfs_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToNonExistentDir/dbfs_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToNonExistentDir/dbfs_to_uc-volumes 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToNonExistentDir/local_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToNonExistentDir/local_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpFileToNonExistentDirWindowsPaths 🙈​s ✅​p 🙈​s ✅​p 🙈​s ✅​p 🙈​s ✅​p 🙈​s ❌​F
❌​ TestFsCpSourceIsDirectoryButTargetIsFile ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_local ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_uc-volumes 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ❌​F ❌​F
❌​ TestFsCpSourceIsDirectoryButTargetIsFile/local_to_dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsLs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsLs/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsLsForNonexistingDir ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsLsForNonexistingDir/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsLsOnEmptyDir ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsLsOnEmptyDir/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsLsOnFile ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsLsOnFile/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsLsWithAbsolutePaths ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsLsWithAbsolutePaths/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsLsWithoutScheme ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsMkdir ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsMkdir/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsMkdirCreatesIntermediateDirectories ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsMkdirCreatesIntermediateDirectories/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsMkdirWhenDirectoryAlreadyExists ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsMkdirWhenDirectoryAlreadyExists/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsMkdirWhenFileExistsAtPath ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsMkdirWhenFileExistsAtPath/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsRmDirRecursively ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsRmDirRecursively/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsRmEmptyDir ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsRmEmptyDir/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsRmFile ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsRmFile/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsRmForNonExistentFile ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsRmForNonExistentFile/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsRmNonEmptyDirectory ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestFsRmNonEmptyDirectory/dbfs ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSecretsPutSecretBytesValue ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSecretsPutSecretStringValue ✅​p ✅​p 🙈​s 🙈​s ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncEnsureRemotePathIsUsableIfRepoDoesntExist ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncEnsureRemotePathIsUsableIfRepoExists ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncEnsureRemotePathIsUsableInWorkspace ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncFullFileSync ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncIncrementalFileOverwritesFolder ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncIncrementalFileSync ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncIncrementalSyncFileToPythonNotebook ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncIncrementalSyncPythonNotebookDelete ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncIncrementalSyncPythonNotebookToFile ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncNestedFolderDoesntFailOnNonEmptyDirectory ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncNestedFolderSync ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestSyncNestedSpacePlusAndHashAreEscapedSync ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestExport ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestExportDir ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestExportDirDoesNotOverwrite ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestExportDirWithOverwriteFlag ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
❌​ TestExportWithFileFlag ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ❌​F
Top 4 slowest tests (at least 2 minutes):
duration env testname
2:58 aws windows TestAccept
2:57 azure windows TestAccept
2:55 aws-ucws windows TestAccept
2:55 azure-ucws windows TestAccept
(56 table rows omitted to keep the report under 60000 bytes)

@pietern

pietern commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

CI timing result: serial + defer vs parallel deps

Measured the test job's "Run tests" step per matrix cell across both commits on this branch.

Matrix cell Baseline (parallel deps) Serial + defer Δ
linux, direct 312s 294s −18s (−5.8%)
linux, terraform 799s 787s −12s (−1.5%)
macos, direct 556s 546s −10s (−1.8%)
macos, terraform 1433s 1010s −423s (−29.5%)
windows, direct 1057s 1149s +92s (+8.7%)
windows, terraform 1827s 1812s −15s (−0.8%)

Conclusion: no timing regression from serializing the legs. On 5 of 6 cells serial was equal or faster. Both legs already saturate the runners, so running them in parallel mostly created core contention rather than saving wall-clock — matching a local 16-core benchmark (294s parallel vs 329s serial there; CI runners are smaller/more contended, so the parallel benefit shrinks further).

Notes:

  • macos-terraform (−29.5%) is likely inflated by macOS runner variance; discount it heavily. Even so, serial is not slower.
  • windows-direct (+92s) is the only regression, but Windows already runs serially in both configs (TASK_CONCURRENCY=1), so this is runner noise, not a structural effect — its sibling windows-terraform was −15s.

Both runs uploaded all 6 artifacts and the serial run's upload step succeeded, confirming defer: cat produces test-output.json even though it's in cmds.

Separate (non-timing) tradeoff that stands: with sequential cmds, a failing test-unit skips test-acc entirely (and vice-versa), so a red build shows only one suite's results. The parallel deps model runs both regardless.

Closing this experiment PR; implementing the change separately.

This comment was written by Isaac.

@pietern pietern closed this Jul 2, 2026
@pietern pietern deleted the ci-timing-serial-defer branch July 2, 2026 13:34
@pietern pietern mentioned this pull request Jul 2, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pietern @eng-dev-ecosystem-bot