Production-readiness refactor: drop MinIO, make Crate IDs more robust, add async validation and offline cache

[alexhambley]

TLDR

This is a big refactor that strengthens the validation service by adding:

• An agnostic S3 storage layer;
• Reworking the "Crate ID" model;
• Adding consistent result/error types;
• Adding a faster metadata path;
• Fixing some major production concerns (retries, structured logging, health checks, offline validation);
• Removing reliance on minio-py SDK

PR

I've broken down the changes in more detail below:

Storage and RO-Crates

• Added a "vendor-neutral" StorageBackend abstraction-layer with a boto3 S3Backend (this should work with AWS / MinIO / RustFS / etc.) and an in-memory fake for tests. minio-py SDK is removed;
• Added stricter/safer Crate-ID validation (charset/length, no / or ..). This fixes the bugs/fragility with .zip-in-ID, by treating IDs as opaque;
• Added deterministic crate resolution by direct key checks and explicit zip vs directory, ambiguous (409) and not-found (404) raised;
• Set a canonical layout with separate crate and result prefixes.

Validation

• There is now a single ValidationOutcome type (valid/invalid/error) that replaces the old ValidationResult | str; one boundary to rocrate-validator;
• Metadata-only validation is now synchronous (removes the blocking result.get());
• Bumped rocrate-validator to 0.10.0 for (opt-in) offline validation. Cache warming at build, and --extra-profiles-path.

APIs

• ID endpoints take server-side credentials, so there are no storage secrets in request bodies/logs!
• Resolve-before-queue;
• The celery worker runs a staged fetch → validate → persist → webhook with retries and persisted error outcomes;
• Webhook delivery retries with backoff and surfaces any failures.

DevOps Stuff

• JSON logging with request IDs and secret redaction;
• /healthz and /readyz added to check the health and readiness for the storage + Redis broker ;
• I've tidied up dependencies. pyproject.toml + pip-tools locks, ruff lint/format, CI updated;
• There is now a single parameterised Dockerfile;
• Flask entrypoint is renamed cratey.py → wsgi.py for convention;
• Image and labels renamed to ro-crate-validation-service;
• The README.md has been rewritten and has some architecture/flow mermaid diagrams to help future external collaborators with understanding how the validator works;
• Tests are now organised properly to mirror app/.

Breaking Changes

• Environment variables are renamed;
• ID endpoints no longer accept minio_config / root_path.
• Status codes are clearer
• Storage layout is explicit: crates under crates/, results under validation-results/.
• Requires rocrate-validator >= 0.10.0

Closes #170

[elichad]

@alexhambley to make this slightly more manageable to review, are you able to pop the last three commits (particularly the last one because it touches many files) off into a separate PR that depends on this one?

[AnnieZQH]

I was able to bring the dev stack up locally and successfully tested the API endpoints. I've also read through the changes and the tests. The coverage report looks good.

The only issue I've found: the renamed image is not on eScienceLab's container registry yet, so using docker-compose.yml would throw an error.

[douglowe]

I've a few build comments, and some small swagger api comments.

I do wish that this PR was a series of separate PR's for each feature. I've not reviewed the async validation and offline cache changes yet, because I've been checking other changes.

I like the health checks - I think these are a good addition. And the making crate ID's more robust seems sensible. As does moving to more generic S3 libraries.

I am concerned about the removal of the S3 secrets from the API call. These were added at the explicit request of our project partners, because they access S3 services using ephemeral secrets. We should have a design conversation about this, to decide if this is something that needs to be supported or not.

Regarding the logging checks - I agree we should try to avoid recording secrets in the logs. This will be more difficult if we are allowing ephemeral secrets to be used, as we won't be able to directly grep for known secrets. But we could add field name redaction, or entropy / length checks, so that most secrets which might be passed are screened out.

[manics]

From a K8TRE perspective users should never deal with S3 credentials. /v1/ro_crates/validate_metadata is easy since it never accesses S3.

/v1/ro_crates/{crate_id}/validation is more difficult with or without credentials, especially if a single instance of the validator needs to support multiple isolated projects.

• With credentials: In general users won't have access to S3 tokens, so another front-end service is needed
• Without credentials: This allows users in one project to validate a crate from another project if they know the path, which breaks isolation of projects, so some other auth layer is needed.

For K8TRE if /v1/ro_crates/validate_metadata is sufficient then S3 credentials don't matter, but this will likely affect how the validator is deployed in other TREs.

[alexhambley]

Thanks @manics for the feedback! I agree with the point around users never dealing with S3 credentials. In terms of the {crate_id}/validation issue, I think this is a deployment topology/service point rather the API specifically. I'll pick up with you outside of the PR for a discussion to see if its an RO-Crate Validation Service thing we change, or if another service is needed. This service needs to stay agnostic, and there's different models to support :)

[douglowe]

Thanks for addressing the change requests I made - these all look good to me.

Having discussed the S3 keys offline with @alexhambley, I am okay with the route we're taking here - as is said above, it is more secure to not have these passed via the API call itself.