Skip to content

feat: add storage tier change events script#118

Open
lhoupert wants to merge 2 commits into
mainfrom
feat--add-storage-tier-change-events-script
Open

feat: add storage tier change events script#118
lhoupert wants to merge 2 commits into
mainfrom
feat--add-storage-tier-change-events-script

Conversation

@lhoupert

@lhoupert lhoupert commented Mar 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

branched from on #103

  • Adds submit_storage_tier_workflows.py operator tool that queries STAC in 24h windows over a date range and submits one batch-change-storage-tier webhook payload per window to trigger the eopf-storage-tier-batch-job WorkflowTemplate deployed in https://github.com/EOPF-Explorer/platform-deploy/pull/130 via Argo Events

  • Adds unit tests (covering time-window generation, STAC querying, dry-run, error handling, and payload structure)

  • Updates operator-tools/README.md with full usage docs and a tool-selection reference table

Why

Changing S3 storage classes for thousands of items over a multi-month date range is too slow when done sequentially on a local machine. The new script fans work out to Argo: one workflow per 24h window, with intra-window parallelism handled by the eopf-storage-tier-batch-job template. Empty windows are skipped automatically.

Test plan

  • uv run pytest tests/unit/test_submit_storage_tier_workflows.py passes
  • uv run pytest tests/unit/test_change_storage_tier_commands.py passes
  • Dry run against staging collection: python operator-tools/submit_storage_tier_workflows.py --start-date 2024-01-01 --end-date 2024-01-03 --collection sentinel-2-l2a-staging --dry-run
  • Live run (port-forward active) submits correct payload shape and logs action: batch-change-storage-tier with item IDs per window

@lhoupert

Copy link
Copy Markdown
Collaborator Author

I tested yday with the command:

 python operator-tools/submit_storage_tier_workflows.py \
    --start-date 2026-03-08 \
    --end-date 2026-03-12 \
    --collection sentinel-2-l2a-staging-backup-20260312 \
    --storage-class STANDARD_IA \
    --webhook-url http://localhost:12001/samples

and it worked well:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants