release: Production-readiness refactor, S3-backend, and dependency updates by alexhambley · Pull Request #189 · eScienceLab/RO-Crate-Validation-Service

alexhambley · 2026-07-22T11:56:47Z

Summary

Merges develop into main for the next release. The bulk of this is the production-readiness refactor (#186), plus the optional-MinIO work (#161) and a series of dependency updates. All of which have been reviewed.

Features

Health endpoints: /healthz and /readyz
Structured JSON logging with request IDs and secret redaction
Webhook delivery retries with backoff; failures are surfaced
ValidationOutcome type and validation runner; metadata validation is now synchronous
Deterministic crate resolver and crate-ID validation
S3 storage via a boto3-based S3Backend; object storage is optional (Make MinIO storage optional #156)

Breaking changes

Container entrypoint and image renamed (image is now ghcr.io/esciencelab/ro-crate-validation-service)
minio-py dropped for generic boto3 S3Backend
Settings are loaded and validated at startup and invalid config now fails early

Tooling & CI

pyproject.toml and ruff; consolidated Dockerfiles and compose files
GitHub Actions pinned to commit SHAs; integration-test and lint deps pinned
RustFS as the local S3-compatible dev store; integration test suite rebuilt to mirror app/ layout
README rewritten

Dependency updates

rocrate-validator 0.11.2, redis 8.0.1, urllib3 2.7.0, idna 3.15, requests 2.34.2, requests-cache 1.3.3

Bumps [roc-validator](https://github.com/crs4/rocrate-validator) from 0.8.1 to 0.9.0. - [Release notes](https://github.com/crs4/rocrate-validator/releases) - [Changelog](https://github.com/crs4/rocrate-validator/blob/develop/CHANGELOG.md) - [Commits](crs4/rocrate-validator@0.8.1...0.9.0) --- updated-dependencies: - dependency-name: roc-validator dependency-version: 0.9.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [requests](https://github.com/psf/requests) from 2.33.0 to 2.33.1. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.33.0...v2.33.1) --- updated-dependencies: - dependency-name: requests dependency-version: 2.33.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

…-0.9.0 Bump roc-validator from 0.8.1 to 0.9.0

Bump requests from 2.33.0 to 2.33.1

Makes MinIO an opt-in and disabled by default, via a MINIO_ENABLED setting. When disabled the service stores nothing and only the stateless validation endpoint is exposed (the ID endpoints are not registered) - config: add get_bool_env helper and MINIO_ENABLED (default False) - routes: split the POST blueprint so /validate_metadata is always available; register the ID-based POST/GET routes only when MINIO_ENABLED is true - compose: move the minio service behind a `minio` profile and drop it from depends_on; MINIO_ENABLED into the flask/celery services - docs/env: document MINIO_ENABLED and the --profile minio workflow - tests: cover enabled and disabled modes; integration tests now starts the minio profile with MINIO_ENABLED=true

Co-authored-by: Eli Chadwick <eli.chadwick@manchester.ac.uk>

feat: Make MinIO object storage optional (#156)

- Replace the import-time Config classes with a validated Settings dataclass built in create_app(). - Fails early with an aggregated error when storage is enabled but the required S3/Celery vars are missing. - Gate storage routes on settings.storage_enabled; Settings injected in tests instead of monkeypatching. - Add app/storage as a "vendor-neutral" abstraction (so, works with MinIO, RustFS, or any other S3 service) - Closes #171, #172

Implement S3Backend using boto3. Supports MinIO SDK features such as pagination. Includes tests Refs #172

Add app/crates/ids.py: validate_crate_id / is_valid_crate_id and an InvalidCrateId exception. IDs are constrained to a safe charset (start alphanumeric, [A-Za-z0-9._-], max 128, no '/' or '..'). Fixes the ".zip in crate ID" fragility. IDs are no longer parsed. Closes #174

Add app/crates/layout.py (separate crate/result key prefixes) and resolver.py. `resolve_crate()` locates a crate by direct stat checks on "canonical keys" instead of prefix-listing. - Distinguishes zip vs directory, - confirms ro-crate-metadata.json for directories, and - reports "AmbiguousCrate" / "CrateNotFound" - Fixes sibling false-match, .zip-in-id, and some missing-metadata gaps. Closes #175

- Add `app/validation/results.py` (`ValidationOutcome` with valid/invalid/error status, detail/error, serialisation) and `runner.py` wrapping rocrate_validator so both entry points always return an outcome; - Replaces the ValidationResult|str / isinstance pattern. Closes #176

- replaces metadata path with `run_metadata_validation()`, which validates inline via the runner and returns a `ValidationOutcome` (200 valid/invalid, 422 unvalidatable). - remove the metadata Celery task and `perform_metadata_validation`, which addresses the result.get() blocking and the return-in-finally bug. closes #169

- Move POST/task/GET onto StorageBackend + resolver + runner with server-side credentials. POST resolves before queueing (400/404/409/503 via app error handlers); - the Celery task (`run_validation_job`) runs fetch->validate->persist->webhook with retries and persists error outcomes; GET reads the results prefix. - Add s3_crate_prefix/s3_results_prefix to Settings. Delete minio_utils and test_minio, remove minio from requirements and recompile the lock. - Covers the store-backed validation flow Closes #177, #178, #179

- Refactor `send_webhook_notification` to retry failures with exponential backoff, add a request timeout, and raise `WebhookDeliveryError` on terminal failure instead of accepting it. - Results are persisted so a delivery failure can be surfaced without losing the result. Closes #180

Rewrite logging_service with a JsonFormatter, a RedactionFilter that masks configured S3 credentials, and a RequestIdFilter closes #177

- Adds endpoints to check if the S3 is reachable and Celery / broker connected; - Returns 503 with a per-check breakdown when a dependency is down. - When storage is disabled both checks report "disabled". Closes #181

- Consolidate packaging and tooling into pyproject.toml and standardise linting and formatting with ruff. - Move metadata and deps into pyproject.toml; compile requirements.txt (runtime) and requirements-dev.txt (runtime + dev: pytest, pytest-mock, moto, ruff) from it via pip-tools; retire requirements.in. - Move pytest config into [tool.pytest.ini_options] and remove pytest.ini. - Add [tool.ruff] config; apply ruff check --fix and ruff format across code. - CI: unit_tests installs requirements-dev.txt; add a lint workflow running ruff check and ruff format --check on PRs to develop. Closes #183, #173

- Replace the MinIO dev container with a vendor-neutral "objectstore" service running rustfs/rustfs. - Update .env/example.env to S3_* app config plus RUSTFS_* container credentials; the app/worker stay vendor-neutral (S3_* only). - Start with `docker compose --profile objectstore up`.

- Move unit tests into app-mirroring sub-packages (storage/, crates/, etc.) and switch pytest to importlib import mode with pythonpath=".". - Rewrite the integration test for the new stack - Update the integration workflow deps Closes #184

Closes #185

… and image A few final things to bundle together: - bump `roc-validator` 0.9.0 -> 0.10.0 to use offline/cache/extra-profiles - rename the WSGI entrypoint and align image/label naming. - Add `VALIDATION_OFFLINE`, `CACHE_PATH` and `EXTRA_PROFILES_PATH` settings - Offline validation is opt-in (default off). - `extra_profiles_path` ADDS to the bundled profiles (roc-validator 0.10.0 fix) - Warm validator HTTP cache at image build (rocrate-validator cache warm) - Replace the site-packages profile bake with extraction to /app/extra-profiles loaded via `EXTRA_PROFILES_PATH` - This is passed as a build arg from the profiles workflow. - Rename entrypoint `cratey.py` -> `wsgi.py` to fit convention - Rename the published image and the five-safes label to ro-crate-validation-service; point image.source at the renamed repo. Closes #162, #163, #164

- Addresses review feedback on #186 - Gives both POST blueprints a shared tag so the Swagger docs show a single POST group whether or not the store-backed routes are registered - APIFlask emits one top-level tag entry per blueprint, so the shared tag would appear twice in openapi.json (OpenAPI requires unique tag names) - Added a spec processor decorator to depulicate list. - Added supporting test

- Addresses review feedback on #186 - Redo profile-fetch and cache-warm RUN commands as BuildKit heredocs - Added set -e to keep old && failure semantics - proposed heredoc blocks would only report exit status of its last command

- Addresses review feedback on #186

- Addresses review feedback on #186 - Idea is that it pins every action to the commit SHA of the latest release in its current major

Production-readiness refactor: drop MinIO, make Crate IDs more robust, add async validation and offline cache

Bumps [redis](https://github.com/redis/redis-py) from 7.4.0 to 8.0.1. - [Release notes](https://github.com/redis/redis-py/releases) - [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES) - [Commits](redis/redis-py@v7.4.0...v8.0.1) --- updated-dependencies: - dependency-name: redis dependency-version: 8.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>

chore(deps): bump redis from 7.4.0 to 8.0.1

Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.6.3 to 2.7.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.6.3...2.7.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.7.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>

chore(deps): bump urllib3 from 2.6.3 to 2.7.0

Bumps [roc-validator](https://github.com/crs4/rocrate-validator) from 0.10.0 to 0.11.2. - [Release notes](https://github.com/crs4/rocrate-validator/releases) - [Changelog](https://github.com/crs4/rocrate-validator/blob/develop/CHANGELOG.md) - [Commits](crs4/rocrate-validator@0.10.0...0.11.2) --- updated-dependencies: - dependency-name: roc-validator dependency-version: 0.10.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

…-0.10.0 chore(deps): bump roc-validator from 0.10.0 to 0.11.2

Bumps [idna](https://github.com/kjd/idna) from 3.10 to 3.15. - [Release notes](https://github.com/kjd/idna/releases) - [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.md) - [Commits](kjd/idna@v3.10...v3.15) --- updated-dependencies: - dependency-name: idna dependency-version: '3.15' dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>

chore(deps): bump idna from 3.10 to 3.15

- requests 2.34 made `RequestsCookieJar` a `TYPE_CHECKING`-only import in requests/models.py, which breaks requests-cache 1.2.1 - Dependabot's #157 bumped only requests and failed the docker build - I also updated to drop async-timeout (redis only needs it on Python < 3.11.3) and typos (no longer required by roc-validator).

chore(deps): bump requests to 2.34.2 and requests-cache to 1.3.3

elichad · 2026-07-22T12:02:55Z

Is it worth putting together an upgrade guide? answering questions like:

what settings should an existing user update (or none)?
if a user already has a minio instance configured, how do they keep using it?

thinking of how we can make this update easy for folks like @AnnieZQH and K8TRE team to adopt.

It would be particularly useful (and maybe sufficient) if all new/changed settings could be listed in our release documentation

alexhambley · 2026-07-22T12:23:49Z

Is it worth putting together an upgrade guide? answering questions like:

what settings should an existing user update (or none)?

if a user already has a minio instance configured, how do they keep using it?

thinking of how we can make this update easy for folks like @AnnieZQH and K8TRE team to adopt.

It would be particularly useful (and maybe sufficient) if all new/changed settings could be listed in our release documentation

Due to release timings and deadlines, this will be an addition to the docs once the site is up and running and pointed to by https://github.com/Health-Informatics-UoN/analysis-types-for-TREs/blob/master/observablehq.config.js#L47

dependabot Bot and others added 30 commits April 27, 2026 13:45

Merge pull request #148 from eScienceLab/dependabot/pip/roc-validator…

5535035

…-0.9.0 Bump roc-validator from 0.8.1 to 0.9.0

Merge pull request #149 from eScienceLab/dependabot/pip/requests-2.33.1

422a4f7

Bump requests from 2.33.0 to 2.33.1

Fix typo in tests/test_integration.py

c7ebfe3

Co-authored-by: Eli Chadwick <eli.chadwick@manchester.ac.uk>

Merge pull request #161 from eScienceLab/feature/optional-minio-156

3ad955b

feat: Make MinIO object storage optional (#156)

feat(storage): add boto3 S3Backend

a3aa2e7

Implement S3Backend using boto3. Supports MinIO SDK features such as pagination. Includes tests Refs #172

feat(logging): structured JSON logs, request IDs, secret redaction

ee62364

Rewrite logging_service with a JsonFormatter, a RedactionFilter that masks configured S3 credentials, and a RequestIdFilter closes #177

feat: add /healthz and /readyz endpoints

f900ae2

- Adds endpoints to check if the S3 is reachable and Celery / broker connected; - Returns 503 with a per-check breakdown when a dependency is down. - When storage is disabled both checks report "disabled". Closes #181

docs: rewrite README for the new architecture with diagrams

3289a9a

Closes #185

chore(docker): consolidate Dockerfiles and tidy up compose

0b340ce

ci: pin integration-test and lint dependencies

6260cca

- Addresses review feedback on #186

ci: pin GitHub Actions to commit SHAs

6f4c389

- Addresses review feedback on #186 - Idea is that it pins every action to the commit SHA of the latest release in its current major

Merge pull request #186 from eScienceLab/refactor-production-readiness

cd34ee9

Production-readiness refactor: drop MinIO, make Crate IDs more robust, add async validation and offline cache

Merge pull request #159 from eScienceLab/dependabot/pip/redis-8.0.0

72c119c

chore(deps): bump redis from 7.4.0 to 8.0.1

dependabot Bot and others added 8 commits July 22, 2026 08:22

Merge pull request #153 from eScienceLab/dependabot/pip/urllib3-2.7.0

9b82b6b

chore(deps): bump urllib3 from 2.6.3 to 2.7.0

Merge pull request #160 from eScienceLab/dependabot/pip/roc-validator…

93e0a8e

…-0.10.0 chore(deps): bump roc-validator from 0.10.0 to 0.11.2

Merge pull request #158 from eScienceLab/dependabot/pip/idna-3.15

0063b87

chore(deps): bump idna from 3.10 to 3.15

Merge pull request #188 from eScienceLab/chore/requests-2.34.2

4279ffe

chore(deps): bump requests to 2.34.2 and requests-cache to 1.3.3

douglowe approved these changes Jul 22, 2026

View reviewed changes

alexhambley merged commit 197861e into main Jul 22, 2026
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

release: Production-readiness refactor, S3-backend, and dependency updates#189

release: Production-readiness refactor, S3-backend, and dependency updates#189
alexhambley merged 38 commits into
mainfrom
develop

alexhambley commented Jul 22, 2026

Uh oh!

elichad commented Jul 22, 2026

Uh oh!

alexhambley commented Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

alexhambley commented Jul 22, 2026

Summary

Features

Breaking changes

Tooling & CI

Dependency updates

Uh oh!

elichad commented Jul 22, 2026

Uh oh!

alexhambley commented Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants