Skip to content

Fix race condition in idp-cli generate-manifest --test-set (#193)#239

Merged
rstrahan merged 2 commits intodevelopfrom
fix/#193-cli-Race-Condition
Mar 17, 2026
Merged

Fix race condition in idp-cli generate-manifest --test-set (#193)#239
rstrahan merged 2 commits intodevelopfrom
fix/#193-cli-Race-Condition

Conversation

@rstrahan
Copy link
Copy Markdown
Contributor

Summary

Fixes #193 — Race condition in idp-cli generate-manifest --test-set that causes test sets to be marked as FAILED with "Missing baseline files" errors when the UI is open during CLI uploads.

Root Cause

When the CLI uploads input files then baselines sequentially to S3, the test set resolver's auto-detection (triggered by UI polling via getTestSets) can validate the test set before all baseline files are uploaded. This creates a premature FAILED DynamoDB entry. The CLI's subsequent resolver call then skips re-validation because the entry already exists.

Solution

Implemented a .uploading S3 marker file protocol:

  1. CLI places marker — Before uploading files, the CLI creates {test_set}/.uploading in the test set bucket
  2. Resolver checks marker_is_valid_test_set_structure() checks for the marker via head_object and skips folders that have it
  3. CLI removes marker — After all uploads complete, the CLI deletes the marker, then invokes the resolver for registration

Changes

File Change
nested/appsync/src/lambda/test_set_resolver/index.py Added .uploading marker check in _is_valid_test_set_structure()
lib/idp_cli_pkg/idp_cli/cli.py Added marker create/delete in generate_manifest() and _create_test_set_from_manifest()
CHANGELOG.md Added fix entry under [Unreleased]

Backward Compatibility

  • Fully backward compatible — Old CLIs without the marker continue to work (resolver only adds an additional skip check)
  • No infrastructure changes — No new DynamoDB fields, Lambda APIs, or AppSync schema changes
  • Fail-safe — If CLI crashes mid-upload, the marker persists and the test set stays hidden from auto-detection (correct behavior)

@rstrahan rstrahan requested a review from kaleko March 15, 2026 22:03
@rstrahan
Copy link
Copy Markdown
Contributor Author

Can you review / verify this fix, @kaleko ? Many tx.

@rstrahan rstrahan merged commit b34f2a9 into develop Mar 17, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant