Skip to content

feat: create cli command to change storage tier#103

Merged
lhoupert merged 1 commit into
mainfrom
feat--add-cli-command-to-change-storage-tier-and-update-storage-metadata-based-on-a-time-window
Mar 19, 2026
Merged

feat: create cli command to change storage tier#103
lhoupert merged 1 commit into
mainfrom
feat--add-cli-command-to-change-storage-tier-and-update-storage-metadata-based-on-a-time-window

Conversation

@lhoupert

@lhoupert lhoupert commented Mar 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add change-storage-tier CLI commands to both manage_item.py (single item) and manage_collections.py (collection-level with optional date filtering) in operator-tools/. These commands change S3 object storage classes (STANDARD, STANDARD_IA, EXPRESS_ONEZONE) and update STAC metadata to reflect the new tier.
  • Remove fix_zarr_asset_media_types() from both register_v0.py and register_v1.py — the upstream source now provides correct media types, so runtime patching is no longer needed. The test fixture was updated accordingly.
  • Remove aggregate_items.py script and its documentation/tests — this aggregation functionality is no longer used.
  • Fix URL encoding in S2 visualization/thumbnail query strings (%20+) in both registration scripts.

Details

New change-storage-tier commands

Single item (manage_item.py change-storage-tier):

  • Changes S3 storage class for all objects in a STAC item's Zarr store
  • Updates STAC item metadata only if all S3 operations succeed (zero failures)
  • Supports --include-pattern / --exclude-pattern for filtering objects
  • Safety: requires confirmation prompt unless --dry-run or -y is passed

Collection-level (manage_collections.py change-storage-tier):

  • Operates across all items in a collection
  • Optional --start-date / --end-date filtering via CQL2 between query
  • Progress bar and per-item failure tracking with summary report

Cleanup

  • Removed aggregate_items.py, its README, and 348 lines of tests
  • Removed fix_zarr_asset_media_types() and its 106-line test file
  • Fixed test fixture media types to match upstream corrections

Test plan

  • Added 21 unit tests for the new CLI commands (test_change_storage_tier_commands.py)
  • Verify uv run pytest passes
  • Manual dry-run test of change-storage-tier against staging collection

@lhoupert lhoupert marked this pull request as draft March 3, 2026 14:56
@lhoupert lhoupert marked this pull request as ready for review March 18, 2026 12:51
@lhoupert

lhoupert commented Mar 18, 2026

Copy link
Copy Markdown
Collaborator Author

implements https://github.com/EOPF-Explorer/coordination/issues/182 for reprocessing historical data through CLI

@emmanuelmathot emmanuelmathot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I wonder about the existing assets in the production catalog. Do we still need to update them?

@lhoupert

Copy link
Copy Markdown
Collaborator Author

YEs I think so! I have another PR ready for review for that: #118 and the argo workflow PR that goes with it: https://github.com/EOPF-Explorer/platform-deploy/pull/129

@lhoupert lhoupert merged commit 4fb0ac7 into main Mar 19, 2026
2 checks passed
@github-actions github-actions Bot mentioned this pull request Mar 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants