feat(validation): bump rocrate-validator to 0.10.0; rename entrypoint and image

A few final things to bundle together:
- bump `roc-validator` 0.9.0 -> 0.10.0 to use offline/cache/extra-profiles
- rename the WSGI entrypoint and align image/label naming.
- Add `VALIDATION_OFFLINE`, `CACHE_PATH` and `EXTRA_PROFILES_PATH` settings
- Offline validation is opt-in (default off).
- `extra_profiles_path` ADDS to the bundled profiles (roc-validator 0.10.0 fix)
- Warm validator HTTP cache at image build (rocrate-validator cache warm)
- Replace the site-packages profile bake with extraction to /app/extra-profiles loaded via `EXTRA_PROFILES_PATH`
- This is passed as a build arg from the profiles workflow.
- Rename entrypoint `cratey.py` -> `wsgi.py` to fit convention
- Rename the published image and the five-safes label to ro-crate-validation-service; point image.source at the renamed repo.

Closes #162, #163, #164

Author: alexhambley

.github/workflows/build-with-profiles.yml

@@ -40,10 +40,12 @@ jobs:
         with:
           context: .
           push: true
-          # Bake the five-safes profile into the shared Dockerfile via build args.
+          # Add the five-safes profile (via EXTRA_PROFILES_PATH) and warm its
+          # cache, using the shared Dockerfile's build args.
           build-args: |
             FIVE_SAFES_PROFILE_VERSION=five-safes-0.7.4-beta
             PROFILES_ARCHIVE_URL=https://github.com/eScienceLab/rocrate-validator/archive/refs/tags/five-safes-0.7.4-beta.tar.gz
+            EXTRA_PROFILES_PATH=/app/extra-profiles
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
 

Dockerfile

@@ -9,25 +9,38 @@ COPY requirements.txt .
 RUN pip install --upgrade pip
 RUN pip install --no-cache-dir -r requirements.txt
 
-COPY cratey.py LICENSE /app/
+COPY wsgi.py LICENSE /app/
 COPY app /app/app
 
-# Optionally bake an extra RO-Crate profile into the validator's bundled
-# profiles directory. A plain build leaves PROFILES_ARCHIVE_URL empty and skips
-# this entirely; the "with profiles" image build passes these as --build-arg.
-# Baked profiles are then found automatically (no PROFILES_PATH needed).
+# Optionally fetch an extra RO-Crate profile into a normal directory. It is
+# *added* to the bundled profiles at runtime via EXTRA_PROFILES_PATH.
+# A plain build leaves PROFILES_ARCHIVE_URL empty
+# and skips this; the "with profiles" image build passes it as --build-arg.
 ARG PROFILES_ARCHIVE_URL=""
 ARG FIVE_SAFES_PROFILE_VERSION=""
-ARG PY_VER=3.11
+# Set EXTRA_PROFILES_PATH only for the profiles build (passed as a build arg).
+ARG EXTRA_PROFILES_PATH=""
+ENV EXTRA_PROFILES_PATH=${EXTRA_PROFILES_PATH}
+ENV CACHE_PATH=/app/.rocrate-cache
 RUN if [ -n "$PROFILES_ARCHIVE_URL" ]; then \
+        mkdir -p /app/extra-profiles && \
         wget -O /tmp/profiles.tar.gz "$PROFILES_ARCHIVE_URL" && \
         tar -xzf /tmp/profiles.tar.gz \
-            -C "/usr/local/lib/python${PY_VER}/site-packages/rocrate_validator/profiles/" \
+            -C /app/extra-profiles \
             --strip-components=3 \
             "rocrate-validator-${FIVE_SAFES_PROFILE_VERSION}/rocrate_validator/profiles/five-safes-crate" && \
         rm /tmp/profiles.tar.gz ; \
     fi
 
+# Pre-populate the HTTP cache so opt-in offline validation
+# (VALIDATION_OFFLINE=true) works without network at runtime.
+RUN if [ -n "$EXTRA_PROFILES_PATH" ]; then \
+        rocrate-validator cache warm --all-profiles \
+            --extra-profiles-path "$EXTRA_PROFILES_PATH" --cache-path "$CACHE_PATH" ; \
+    else \
+        rocrate-validator cache warm --all-profiles --cache-path "$CACHE_PATH" ; \
+    fi
+
 RUN useradd -ms /bin/bash flaskuser
 RUN chown -R flaskuser:flaskuser /app
 
@@ -37,5 +50,5 @@ EXPOSE 5000
 
 CMD ["flask", "run", "--host=0.0.0.0"]
 
-LABEL org.opencontainers.image.source="https://github.com/eScienceLab/Cratey-Validator"
-LABEL org.cratey.five-safes-profile-version="${FIVE_SAFES_PROFILE_VERSION}"
+LABEL org.opencontainers.image.source="https://github.com/eScienceLab/RO-Crate-Validation-Service"
+LABEL org.ro-crate-validation-service.five-safes-profile-version="${FIVE_SAFES_PROFILE_VERSION}"

README.md

@@ -256,12 +256,28 @@ fails quickly with a clear error rather than at the first request.
 | `S3_RESULTS_PREFIX` | `validation-results` | key prefix for results |
 | `CELERY_BROKER_URL` | — | Redis broker URL |
 | `CELERY_RESULT_BACKEND` | — | Celery result backend URL |
-| `PROFILES_PATH` | — | directory of 'custom' RO-Crate profiles (optional) |
+| `PROFILES_PATH` | — | directory of profiles that **replaces** the bundled set (optional) |
+| `EXTRA_PROFILES_PATH` | — | directory of profiles **added** to the bundled set (optional) |
+| `CACHE_PATH` | per-user dir | HTTP cache location for the validator |
+| `VALIDATION_OFFLINE` | `false` | validate using only the cache (no network) |
 | `FLASK_ENV` | `development` | `production` disables debug |
 
 When `STORAGE_ENABLED=true`, the `S3_*` and `CELERY_*` variables above are
 **required** — startup fails if any are missing.
 
+### Profiles, cache, and offline validation
+
+Custom profiles (e.g. `five-safes-crate`) are best added with
+`EXTRA_PROFILES_PATH`, which **adds** them to the validator's bundled profiles —
+unlike `PROFILES_PATH`, which replaces the bundled set entirely. The published
+"with profiles" image bakes the five-safes profile in this way.
+
+The validator caches the profile/context HTTP resources it fetches. The Docker
+image pre-populates this cache at build time (`rocrate-validator cache warm`),
+so setting `VALIDATION_OFFLINE=true` runs validation entirely from the cache
+with no network access. Online validation (the default) also uses and refreshes
+the cache. (Requires rocrate-validator ≥ 0.10.0.)
+
 ## Running the service
 
 ### Prerequisites
@@ -339,25 +355,25 @@ app/
 ├── __init__.py                 # app factory: config, blueprints, error handlers, request IDs
 ├── health.py                   # /healthz and /readyz
 ├── storage/                    # object-storage abstraction
-│   ├── base.py                 # StorageBackend protocol + ObjectStat
-│   ├── s3.py                   # boto3 implementation (any S3-compatible store)
-│   ├── memory.py               # in-memory backend (tests / local)
-│   └── errors.py               # StorageError, ObjectNotFound
+│   ├── base.py                 #   StorageBackend protocol + ObjectStat
+│   ├── s3.py                   #   boto3 implementation (any S3-compatible store)
+│   ├── memory.py               #   in-memory backend (tests / local)
+│   └── errors.py               #   StorageError, ObjectNotFound
 ├── crates/                     # crate identity, layout, resolution
-│   ├── ids.py                  # 'Crate ID' validation
-│   ├── layout.py               # object keys
-│   └── resolver.py             # deterministic zip/dir resolution
+│   ├── ids.py                  #   'Crate ID' validation
+│   ├── layout.py               #   object keys
+│   └── resolver.py             #   deterministic zip/dir resolution
 ├── validation/                 # validation boundary
-│   ├── results.py              # ValidationOutcome (valid/invalid/error)
-│   └── runner.py               # wraps rocrate-validator
-├── ro_crates/routes/           # HTTP endpoints (metadata + ID-based)
+│   ├── results.py              #   ValidationOutcome (valid/invalid/error)
+│   └── runner.py               #   wraps rocrate-validator
+├── ro_crates/routes/           #   HTTP endpoints (metadata + ID-based)
 ├── services/
-│   ├── validation_service.py   # request handling: resolve, queue, read results
-│   └── logging_service.py      # JSON logging, request IDs, redaction
-├── tasks/validation_tasks.py   # Celery task: fetch → validate → persist → webhook
+│   ├── validation_service.py   #   request handling: resolve, queue, read results
+│   └── logging_service.py      #   JSON logging, request IDs, redaction
+├── tasks/validation_tasks.py   #   Celery task: fetch → validate → persist → webhook
 └── utils/
-    ├── config.py               # validated Settings
-    └── webhook_utils.py        # webhook delivery with retry/backoff
+    ├── config.py               #   validated Settings
+    └── webhook_utils.py        #   webhook delivery with retry/backoff
 ```
 
 ## License

app/ro_crates/routes/post_routes.py

@@ -71,12 +71,15 @@ def validate_ro_crate_metadata(json_data) -> tuple[Response, int]:
     """
 
     crate_json = json_data["crate_json"]
+    profile_name = json_data.get("profile_name")
 
-    if "profile_name" in json_data:
-        profile_name = json_data["profile_name"]
-    else:
-        profile_name = None
-
-    profiles_path = current_app.config["PROFILES_PATH"]
+    settings = current_app.config["SETTINGS"]
 
-    return run_metadata_validation(crate_json, profile_name, profiles_path=profiles_path)
+    return run_metadata_validation(
+        crate_json,
+        profile_name,
+        profiles_path=settings.profiles_path,
+        extra_profiles_path=settings.extra_profiles_path,
+        cache_path=settings.cache_path,
+        offline=settings.validation_offline,
+    )

app/services/validation_service.py

@@ -51,7 +51,12 @@ def queue_ro_crate_validation_task(
 
 
 def run_metadata_validation(
-    crate_json: str, profile_name=None, profiles_path=None
+    crate_json: str,
+    profile_name=None,
+    profiles_path=None,
+    extra_profiles_path=None,
+    cache_path=None,
+    offline=False,
 ) -> tuple[Response, int]:
     """
     Validate RO-Crate metadata synchronously and return the result inline.
@@ -62,7 +67,10 @@ def run_metadata_validation(
 
     :param crate_json: The RO-Crate JSON-LD metadata, as a string.
     :param profile_name: The profile to validate against.
-    :param profiles_path: A path to the profile definition directory.
+    :param profiles_path: A profiles directory that replaces the bundled set.
+    :param extra_profiles_path: A profiles directory added to the bundled set.
+    :param cache_path: HTTP cache location for the validator.
+    :param offline: Validate using only the cache (no network).
     :return: A JSON response and HTTP status code.
     """
     if not crate_json:
@@ -76,7 +84,14 @@ def run_metadata_validation(
     if not metadata:
         return jsonify({"error": "Required parameter crate_json is empty"}), 422
 
-    outcome = validate_metadata(metadata, profile_name=profile_name, profiles_path=profiles_path)
+    outcome = validate_metadata(
+        metadata,
+        profile_name=profile_name,
+        profiles_path=profiles_path,
+        extra_profiles_path=extra_profiles_path,
+        cache_path=cache_path,
+        offline=offline,
+    )
     status_code = 422 if outcome.status is ValidationStatus.ERROR else 200
     return jsonify(outcome.to_dict()), status_code
 

app/tasks/validation_tasks.py

@@ -85,6 +85,9 @@ def run_validation_job(
                 local_path,
                 profile_name=profile_name,
                 profiles_path=settings.profiles_path,
+                extra_profiles_path=settings.extra_profiles_path,
+                cache_path=settings.cache_path,
+                offline=settings.validation_offline,
                 created_at=created_at,
             )
     finally:

app/utils/config.py

@@ -47,6 +47,9 @@ class Settings:
     s3_use_ssl: bool
     s3_crate_prefix: str
     s3_results_prefix: str
+    extra_profiles_path: Optional[str]
+    cache_path: Optional[str]
+    validation_offline: bool
 
     @classmethod
     def from_env(cls, env: Optional[Mapping[str, str]] = None) -> "Settings":
@@ -91,6 +94,9 @@ def from_env(cls, env: Optional[Mapping[str, str]] = None) -> "Settings":
             s3_use_ssl=_parse_bool(env.get("S3_USE_SSL")),
             s3_crate_prefix=_clean(env.get("S3_CRATE_PREFIX")) or "crates",
             s3_results_prefix=_clean(env.get("S3_RESULTS_PREFIX")) or "validation-results",
+            extra_profiles_path=_clean(env.get("EXTRA_PROFILES_PATH")),
+            cache_path=_clean(env.get("CACHE_PATH")),
+            validation_offline=_parse_bool(env.get("VALIDATION_OFFLINE")),
         )
 
 

app/validation/runner.py

@@ -19,15 +19,21 @@ def validate_crate_path(
     rocrate_uri: str,
     profile_name: Optional[str] = None,
     profiles_path: Optional[str] = None,
+    extra_profiles_path: Optional[str] = None,
     skip_checks: Optional[list] = None,
+    cache_path: Optional[str] = None,
+    offline: bool = False,
     created_at: Optional[str] = None,
 ) -> ValidationOutcome:
     """Validate a crate on disk (a directory or zip) at ``rocrate_uri``."""
     return _run(
         {"rocrate_uri": rocrate_uri},
         profile_name=profile_name,
         profiles_path=profiles_path,
+        extra_profiles_path=extra_profiles_path,
         skip_checks=skip_checks,
+        cache_path=cache_path,
+        offline=offline,
         created_at=created_at,
     )
 
@@ -36,15 +42,21 @@ def validate_metadata(
     metadata: dict,
     profile_name: Optional[str] = None,
     profiles_path: Optional[str] = None,
+    extra_profiles_path: Optional[str] = None,
     skip_checks: Optional[list] = None,
+    cache_path: Optional[str] = None,
+    offline: bool = False,
     created_at: Optional[str] = None,
 ) -> ValidationOutcome:
     """Validate an in-memory RO-Crate metadata graph."""
     return _run(
         {"metadata_only": True, "metadata_dict": metadata},
         profile_name=profile_name,
         profiles_path=profiles_path,
+        extra_profiles_path=extra_profiles_path,
         skip_checks=skip_checks,
+        cache_path=cache_path,
+        offline=offline,
         created_at=created_at,
     )
 
@@ -53,16 +65,25 @@ def _run(
     base_settings: dict,
     profile_name: Optional[str],
     profiles_path: Optional[str],
+    extra_profiles_path: Optional[str],
     skip_checks: Optional[list],
+    cache_path: Optional[str],
+    offline: bool,
     created_at: Optional[str],
 ) -> ValidationOutcome:
     options = dict(base_settings)
     if profile_name:
         options["profile_identifier"] = profile_name
     if profiles_path:
         options["profiles_path"] = profiles_path
+    if extra_profiles_path:
+        options["extra_profiles_path"] = extra_profiles_path
     if skip_checks:
         options["skip_checks"] = skip_checks
+    if cache_path:
+        options["cache_path"] = cache_path
+    if offline:
+        options["offline"] = offline
 
     try:
         settings = services.ValidationSettings(**options)

docker-compose-develop.yml

@@ -11,7 +11,7 @@ services:
     ports:
       - "5001:5000"
     environment:
-      - FLASK_APP=cratey.py
+      - FLASK_APP=wsgi.py
       - FLASK_ENV=development
       - CELERY_BROKER_URL=redis://redis:6379/0
       - CELERY_RESULT_BACKEND=redis://redis:6379/0
@@ -23,7 +23,7 @@ services:
       - S3_SECRET_KEY=${S3_SECRET_KEY}
       - S3_BUCKET=${S3_BUCKET}
       - S3_USE_SSL=${S3_USE_SSL:-false}
-      - PROFILES_PATH=/app/profiles
+      - EXTRA_PROFILES_PATH=/app/profiles
     depends_on:
       - redis
     # Metadata validation runs synchronously in this process, so the flask
@@ -47,7 +47,7 @@ services:
       - S3_SECRET_KEY=${S3_SECRET_KEY}
       - S3_BUCKET=${S3_BUCKET}
       - S3_USE_SSL=${S3_USE_SSL:-false}
-      - PROFILES_PATH=/app/profiles
+      - EXTRA_PROFILES_PATH=/app/profiles
     depends_on:
       - redis
     volumes:

docker-compose.yml

@@ -1,4 +1,4 @@
-# Default stack, using the published image (ghcr.io/.../cratey-validator).
+# Default stack, using the published image (ghcr.io/.../ro-crate-validation-service).
 # For running the service. Object storage is opt-in: set STORAGE_ENABLED=true
 # and start the "objectstore" profile (docker compose --profile objectstore up).
 # For local development against a freshly built image, use
@@ -7,11 +7,11 @@
 services:
   flask:
     platform: linux/x86_64
-    image: "ghcr.io/esciencelab/cratey-validator:0.1"
+    image: "ghcr.io/esciencelab/ro-crate-validation-service:0.1"
     ports:
       - "5001:5000"
     environment:
-      - FLASK_APP=cratey.py
+      - FLASK_APP=wsgi.py
       - FLASK_ENV=development
       - CELERY_BROKER_URL=redis://redis:6379/0
       - CELERY_RESULT_BACKEND=redis://redis:6379/0
@@ -28,7 +28,7 @@ services:
 
   celery_worker:
     platform: linux/x86_64
-    image: "ghcr.io/esciencelab/cratey-validator:0.1"
+    image: "ghcr.io/esciencelab/ro-crate-validation-service:0.1"
     command: celery -A app.celery_worker.celery worker --loglevel=info -E
     environment:
       - CELERY_BROKER_URL=redis://redis:6379/0