Find best way to purge files from dynamic cache

[kimandrews]

Outputs of tb-profiler and snippy are stored in a dynamic cache at s3://nextstrain-data/files/workflows/tb/data. As part of the development process of the tb workflow, some of these files were manually deleted from the console. These deleted files still exist on the S3 bucket with a "delete" marker. If the tb workflow changes in the future, we may also need to delete more files. We should figure out the best way to purge files without doing it manually from the console.

[victorlin]

Two questions:

1. Is the presence of the delete marker unwanted because it is still taking up space on S3, or does it interfere with the workflow in some way?
   • Side notes:
     • The delete marker behavior is because s3://nextstrain-data is a versioning-enabled bucket and you're effectively sending a simple DELETE request to the object without specifying a version. It's possible to bypass this by going to the "Versions" tab on the object. More info in AWS doc.
     • If versioning is not useful for these files (not sure if it is?), we could use a bucket without versioning so that simple DELETE requests actually delete instead of adding a delete marker.
2. What scenarios would warrant a purge, and to what extent?
   • Example: updated snippy parameters, all existing files under s3://nextstrain-data/files/workflows/tb/data/snippy/ are considered stale.

[kimandrews]

For now, everything that has a delete marker can be purged because it was generated as part of the development process. Going forward, it's possible we may want to version the files.

[joverlee521]

If versioning is not useful for these files (not sure if it is?), we could use a bucket without versioning so that simple DELETE requests actually delete instead of adding a delete marker.

FWIW, @jameshadfield made the same suggestion of moving these files to a non-versioned bucket in nextstrain/nextstrain.org#1259 (comment)