Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
cba3cc8
v0.1 validator, and specify platform
douglowe May 16, 2025
6116484
validate_metadata api
douglowe Jun 27, 2025
b92649e
metadata api integration test
douglowe Jul 1, 2025
13b4bed
gh action for integration test
douglowe Jul 1, 2025
f1f3dad
load example env for tests
douglowe Jul 1, 2025
9131452
PEP8 conformance for validation classes
douglowe Jul 4, 2025
e004fda
tests without loading docker service
douglowe Jul 4, 2025
252434a
rename metadata_integration to test_integration
douglowe Jul 8, 2025
54fe23b
tests for metadata api
douglowe Jul 8, 2025
463345e
test docstrings
douglowe Jul 8, 2025
1aaa885
switch API root from /ro_crates/ to /v1/
douglowe Jul 8, 2025
b0166fa
switch API root to /v1/ in tests
douglowe Jul 8, 2025
e4e1812
update path in metadata tests
douglowe Jul 14, 2025
e5651eb
integration test for metadata updated to /vi/ro_crates path
douglowe Jul 14, 2025
0a32d98
app path changed to /v1/ro_crates
douglowe Jul 14, 2025
12bb41c
minio util tests
douglowe Jul 14, 2025
bef24d3
remove unneeded commented code
douglowe Jul 17, 2025
0bdad8f
add unit tests to GH workflow
douglowe Jul 17, 2025
3b50b9f
try running pytest as python module
douglowe Jul 17, 2025
dc20f57
add pytest-mock to pip install
douglowe Jul 17, 2025
c700fb3
remove partial option for validating json
douglowe Jul 17, 2025
3c6e3ba
tests for validation tasks
douglowe Jul 14, 2025
1c548c1
correct edgecases for process_validation_task_by_metadata
douglowe Jul 14, 2025
9cd24d7
tidy up function loading in validation tests
douglowe Jul 17, 2025
7c55402
removed unneeded module loads in validation tests
douglowe Jul 17, 2025
f14f595
tidy small parts of minio test code
douglowe Jul 17, 2025
647d2e5
tests for validation services functions
douglowe Jul 18, 2025
3a3888f
remove redundant celery definitions
douglowe Jul 20, 2025
dc670e1
Revert "remove redundant celery definitions"
douglowe Jul 20, 2025
e760dc3
remove only redundant make_celery function
douglowe Jul 20, 2025
6dc565e
dependabot GH action
douglowe Jul 23, 2025
552143e
tidy yaml file
douglowe Jul 23, 2025
28ddb32
move dependabot config to correct location
douglowe Jul 23, 2025
1fc7c69
create InvalidAPIUsage class
douglowe Jul 20, 2025
767b5e6
InvalidAPIUsage class and cleaner API error messages
douglowe Jul 20, 2025
ed39604
add post_routes tests, including metadata tests
douglowe Jul 20, 2025
c828f25
test_metadata.py moved to test_post_routes.py
douglowe Jul 20, 2025
04b302b
specify str type for metadata service function and add checks for this
douglowe Jul 21, 2025
8862cfa
adapt and add tests for crate metadata json strings
douglowe Jul 21, 2025
89d1747
422 error for malformed/missing crate string
douglowe Jul 21, 2025
aaeb5e2
422 errors and make crate_json string for celery test
douglowe Jul 21, 2025
45beaa6
cleaned out unused test check in validate metadata api
douglowe Jul 21, 2025
963e22e
api tests for metadata errors
douglowe Jul 21, 2025
9c10ef0
remove jsonify from test routes
douglowe Jul 21, 2025
cb4bc52
clean validate_by_id function to use data class for validating inputs
douglowe Jul 21, 2025
e82dc46
tests updated for cleaner validate_by_id interface
douglowe Jul 21, 2025
456b538
post route now /v1/ro_crates/{crate_id}/validation
douglowe Jul 21, 2025
0101e71
updated tests to reflect post route change
douglowe Jul 21, 2025
e3ee7da
get route changed to /v1/ro_crates/{crate_id}/validation
douglowe Jul 21, 2025
56120a2
clean get route library imports
douglowe Jul 21, 2025
f8b42e6
validate post and get examples
douglowe Jun 27, 2025
dffc061
remove try to allow pass through of InvalidAPIUsage in get_ro_crate_v…
douglowe Jul 24, 2025
b0afaa8
remove try to allow pass through of InvalidAPIUsage in return_ro_crat…
douglowe Jul 24, 2025
d774ee9
file_utils whitespace cleanup
douglowe Jul 24, 2025
6aaa244
Major updates to minio_utils.py
douglowe Jul 24, 2025
8b49c03
expanded testset for minio functions
douglowe Jul 24, 2025
1b92106
cleaned service and task tests to reflect addition of InvalidAPIUsage…
douglowe Jul 24, 2025
ef4e274
whitespace cleanup
douglowe Jul 24, 2025
2069ceb
whitespace cleanup
douglowe Jul 24, 2025
5c1b1b3
remove erroneous response.release_conn()
douglowe Jul 24, 2025
712f9eb
remove unneeded test for closing connection
douglowe Jul 25, 2025
6963629
clean minio_util doc_strings
douglowe Jul 26, 2025
a2912ab
check crate existence before queuing validation task
douglowe Jul 27, 2025
aa26f6f
find_rocrate returns False not raise error
douglowe Jul 27, 2025
c033cac
tests updated for find_rocrate
douglowe Jul 27, 2025
204f70d
check_ro_crate_exists added to validation_tasks
douglowe Jul 27, 2025
71a72b5
tests for check_ro_crate_exists
douglowe Jul 27, 2025
969d5d8
move invalidapiusage call up to service level
douglowe Jul 27, 2025
97c25c2
rework tests for queue_ro_crate_validation_task
douglowe Jul 27, 2025
abf1f8c
remove unused invalidapiusage import
douglowe Jul 27, 2025
8082bcb
Validation result retrieval is more robust.
douglowe Jul 27, 2025
1480894
integration tests extended, with example files for MinIO store
douglowe Jul 27, 2025
d5aaac8
updated unit tests
douglowe Jul 27, 2025
75fa69d
add minio to integration tests
douglowe Jul 27, 2025
312dfb0
updated example usage in README
douglowe Jul 27, 2025
1688aa0
add path parameters to API docstrings
douglowe Jul 28, 2025
b320dcc
add minio_bucket and root_path to post api
douglowe Jul 28, 2025
144741f
minio_bucket and root_path added to queue_ro_crate_validation
douglowe Jul 28, 2025
0665d09
minio_bucket and root_path for get ro-crate route
douglowe Jul 28, 2025
6797e3c
added get_route tests
douglowe Jul 28, 2025
f10cfcf
rename test_post_routes to test_api_routes
douglowe Jul 28, 2025
94f986a
add minio_bucket and root_path to checks for ro-crates and validation…
douglowe Jul 28, 2025
7eb187c
change to get_minio_client in validation tasks
douglowe Jul 28, 2025
9750607
pass minio_bucket down to minio_utils instead of using get_minio_clie…
douglowe Jul 28, 2025
bbb6987
correct api test call order
douglowe Jul 28, 2025
d9a8f87
add test_bucket and base_path to validation task tests
douglowe Jul 28, 2025
c64d6c2
root_path mandatory for minio_util calls
douglowe Jul 28, 2025
8e16ba8
added rocrate_bucket and root_path to all required functions now
douglowe Jul 29, 2025
de23a96
minio and validation_tasks tests updated
douglowe Jul 29, 2025
f47d676
integration tests will now return docker container logs on failure
douglowe Jul 29, 2025
b7c053e
integration tests updated
douglowe Jul 29, 2025
1d60d2b
add formatting to pytest output
douglowe Jul 29, 2025
773f3fa
remove last storage_path references in tests
douglowe Jul 29, 2025
30717a6
added extra wait for integration tests, for GH actions
douglowe Jul 29, 2025
66f96b5
output more logs from integration tests
douglowe Jul 29, 2025
f229007
add root_path to update_validation_status_in_minio
douglowe Jul 29, 2025
dd782ef
integration tests for ro-crates not in base directory
douglowe Jul 29, 2025
16a4442
merge api missing element tests
douglowe Jul 31, 2025
03bb03b
combine the validate_by_id success tests
douglowe Jul 31, 2025
ac4187e
validate_metadata api tests parameterised
douglowe Jul 31, 2025
4f61567
parameterise get_validation_by_id failures
douglowe Jul 31, 2025
9802790
parameterized service tests
douglowe Jul 31, 2025
87d59cf
add todo notes to validation task
douglowe Aug 1, 2025
9a0d269
parametrize rocrate validation task tests
douglowe Aug 1, 2025
60ea193
parameterise existence task tests
douglowe Aug 1, 2025
2edf60b
clean up minio tests
douglowe Aug 4, 2025
4efc8b3
get_minio_client now takes config dictionary
douglowe Aug 4, 2025
623f488
add minio config dictionary to validate rocrate post route
douglowe Aug 4, 2025
1666585
pass minio config to validation service, and load client there
douglowe Aug 4, 2025
aa9cb90
pass minio client into test_ro_crate_exists
douglowe Aug 4, 2025
5bc2621
pass minio client into get validation function
douglowe Aug 4, 2025
26f3a14
pass minio config to validation celery task
douglowe Aug 4, 2025
6139b60
pass minio client into fetch ro-crate function
douglowe Aug 4, 2025
1629663
pass minio client into update validation function
douglowe Aug 4, 2025
0d4c307
update process validation failure tests
douglowe Aug 4, 2025
d2c730d
add minio config to api get route
douglowe Aug 5, 2025
12b729e
add minio_config and minio client to get validation service
douglowe Aug 5, 2025
ba62fc8
add minio client to return ro-crate validation
douglowe Aug 5, 2025
5b5daf6
pass minio_client into get validation from minio
douglowe Aug 5, 2025
19e7003
update integration tests to pass minio details to api
douglowe Aug 5, 2025
f10baf6
remove testing reqs file
douglowe Aug 7, 2025
2679d27
move requirements to pip-tools input file
douglowe Aug 7, 2025
e737b0e
pip-compile requirements file
douglowe Aug 7, 2025
8555c17
update requirements
douglowe Aug 7, 2025
c208990
add dotenv tools back in
douglowe Aug 7, 2025
63cbff1
Update celery requirement from ~=5.4.0 to ~=5.5.3
dependabot[bot] Aug 7, 2025
8e6a4b9
Bump werkzeug from 3.0.4 to 3.0.6
dependabot[bot] Aug 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
version: 2
updates:
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
35 changes: 35 additions & 0 deletions .github/workflows/test_docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Integration Tests

on:
pull_request:
branches: [ develop ]

jobs:
test:
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pytest requests minio docker

- name: Build Docker Compose Containers
run: |
cp example.env .env
docker compose -f docker-compose-develop.yml build

- name: Spin Up Docker Compose and Run Tests
run: pytest -s -v tests/test_integration.py

- name: Ensure that Docker Compose is Shutdown
if: always()
run: docker compose down
28 changes: 28 additions & 0 deletions .github/workflows/unit_tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Unit Tests

on:
pull_request:
branches: [ develop ]

jobs:
test:
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest pytest-mock

- name: Run tests (excluding integration tests)
run: |
python -m pytest --ignore=tests/test_integration.py
18 changes: 13 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,12 +70,20 @@ For testing locally developed containers use the alternate Docker Compose file:

## Example Usage

```
Submission of validation of RO-Crate with the ID of `ro_crate_1`. No webhook is used here:
```bash
curl -X 'POST' \
'http://localhost:5001/ro_crates/validate_by_id_no_webhook' \
'http://localhost:5001/ro_crates/v1/ro_crate_1/validation' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{}'
```

Retrieval of validation result for RO-Crate `ro_crate_1`:
```bash
curl -X 'GET' \
'http://localhost:5001/ro_crates/v1/ro_crate_1/validation' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"crate_id": "1"
}'
-d '{}'
```
14 changes: 9 additions & 5 deletions app/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@

from apiflask import APIFlask

from app.celery_worker import make_celery, celery
from app.ro_crates.routes import ro_crates_post_bp, ro_crates_get_bp
from app.utils.config import DevelopmentConfig, ProductionConfig, make_celery
from app.ro_crates.routes import v1_post_bp, v1_get_bp
from app.utils.config import DevelopmentConfig, ProductionConfig, InvalidAPIUsage, make_celery
from flask import jsonify


def create_app() -> APIFlask:
Expand All @@ -20,8 +20,12 @@ def create_app() -> APIFlask:
:return: Flask: A configured Flask application instance.
"""
app = APIFlask(__name__)
app.register_blueprint(ro_crates_post_bp, url_prefix="/ro_crates")
app.register_blueprint(ro_crates_get_bp, url_prefix="/ro_crates")
app.register_blueprint(v1_post_bp, url_prefix="/v1/ro_crates")
app.register_blueprint(v1_get_bp, url_prefix="/v1/ro_crates")

@app.errorhandler(InvalidAPIUsage)
def invalid_api_usage(e):
return jsonify(e.to_dict()), e.status_code

# Load configuration:
if os.getenv("FLASK_ENV") == "production":
Expand Down
36 changes: 0 additions & 36 deletions app/celery_worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,42 +5,6 @@
# Copyright (c) 2025 eScience Lab, The University of Manchester

from celery import Celery
from flask import Flask


def make_celery(app: Flask = None) -> Celery:
"""
Create and configure a Celery instance using Flask configuration.

:param app: The Flask application. Configuration will be used to initialise Celery.
:return: Celery: A configured Celery instance.
:raises ValueError: If the Flask configuration values are not provided.
"""
if not app:
raise ValueError(
"A Flask application instance must be provided to configure Celery."
)

if "CELERY_RESULT_BACKEND" not in app.config:
raise ValueError(
"Missing configuration: 'CELERY_RESULT_BACKEND' is not defined in the Flask app config."
)

if "CELERY_BROKER_URL" not in app.config:
raise ValueError(
"Missing configuration: 'CELERY_BROKER_URL' is not defined in the Flask app config."
)

celery_instance = Celery(
app.import_name,
backend=app.config["CELERY_RESULT_BACKEND"],
broker=app.config["CELERY_BROKER_URL"],
)

if app:
celery_instance.conf.update(app.config)

return celery_instance


celery = Celery()
10 changes: 2 additions & 8 deletions app/ro_crates/routes/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,8 @@
# License: MIT
# Copyright (c) 2025 eScience Lab, The University of Manchester

from apiflask import APIBlueprint

from app.ro_crates.routes.post_routes import post_routes_bp
from app.ro_crates.routes.get_routes import get_routes_bp

#ro_crates_bp = APIBlueprint("ro_crates", __name__)

#ro_crates_bp.register_blueprint(post_routes_bp, url_prefix="/post")

ro_crates_post_bp = post_routes_bp
ro_crates_get_bp = get_routes_bp
v1_post_bp = post_routes_bp
v1_get_bp = get_routes_bp
50 changes: 36 additions & 14 deletions app/ro_crates/routes/get_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,45 @@
# Copyright (c) 2025 eScience Lab, The University of Manchester

from apiflask import APIBlueprint, Schema
from apiflask.fields import Integer, String
from flask import request, Response
from apiflask.fields import String, Boolean
from marshmallow.fields import Nested
from flask import Response

from app.services.validation_service import get_ro_crate_validation_task

get_routes_bp = APIBlueprint("get_routes", __name__)

class validate_data(Schema):
crate_id = String(required=True)

class MinioConfig(Schema):
endpoint = String(required=True)
accesskey = String(required=True)
secret = String(required=True)
ssl = Boolean(required=True)
bucket = String(required=True)

@get_routes_bp.get("/get_validation_by_id")
@get_routes_bp.input(validate_data(partial=True), location='json')
def get_ro_crate_validation_by_id(json_data) -> tuple[Response, int]:

class ValidateResult(Schema):
minio_config = Nested(MinioConfig, required=True)
root_path = String(required=False)


@get_routes_bp.get("<string:crate_id>/validation")
@get_routes_bp.input(ValidateResult(partial=False), location='json')
def get_ro_crate_validation_by_id(json_data, crate_id) -> tuple[Response, int]:
"""
Endpoint to obtain an RO-Crate validation result using its ID from MinIO.

Parameters:
- **crate_id**: The ID of the RO-Crate to validate. _Required_.
Path Parameters:
- **crate_id**: The RO-Crate ID. _Required_.

Request Body Parameters:
- **minio_config**: The MinIO bucket containing the RO-Crate. _Required_
- **endpoint**: Endpoint, e.g. 'localhost:9000'
- **accesskey**: Access key / username
- **secret**: Secret / password
- **ssl**: Use SSL encryption? True/False
- **bucket**: The MinIO bucket to access
- **root_path**: The root path containing the RO-Crate. _Optional_

Returns:
- A tuple containing the validation result and an HTTP status code.
Expand All @@ -32,9 +52,11 @@ def get_ro_crate_validation_by_id(json_data) -> tuple[Response, int]:
- KeyError: If required parameters (`crate_id`) are missing.
"""

try:
crate_id = json_data["crate_id"]
except:
raise KeyError("Missing required parameter: 'crate_id'")
minio_config = json_data["minio_config"]

if "root_path" in json_data:
root_path = json_data["root_path"]
else:
root_path = None

return get_ro_crate_validation_task(crate_id)
return get_ro_crate_validation_task(minio_config, crate_id, root_path)
Loading