ci: gate staging on extension tests (+ build-time caching)#205
Merged
cooper667 merged 5 commits intoMay 29, 2026
Conversation
Re-adds the CKAN extension tests that died with the Jenkins pipeline, now as a 'test' job in build-deploy.yml that build/deploy depend on. Brings up the docker-compose dev stack (the same flow the old ci_setup.sh/ci_test.sh drove via adx) on pinned submodule commits and runs the 5 Fjelltopp suites: unaids, validation, scheming, dhis2harvester, emailasusername Gate semantics: - push: tests must pass before build and deploy run - redeploy (workflow_dispatch + image_tag): tests skipped, image already tested when built - test failure blocks both build and deploy Tests run against the dev image + bind-mounted submodules, matching local 'adx test'. Not yet verified green under CKAN 2.11/Py3.10.
- Add pull_request trigger so the extension-test job runs as a visible status check; build/deploy stay gated off pull_request events. - Cache .adxvenv on Pipfile.lock — the dominant cost is bootstrap's 'pipenv sync --dev' (CKAN + ~19 extensions), not the image build. - Cache the unaids React node_modules on its yarn.lock. Both caches are safe on miss, so the first run is a clean cold signal.
- Run all 5 extension suites and aggregate, so one run reports the full picture instead of stopping at the first failing suite. - Split cache restore/save so .adxvenv and node_modules are saved even when tests fail, making triage runs warm instead of cold.
- Add 'mock' and 'pyfakefs' to dev-packages: ckanext-validation, -scheming and -emailasusername import the standalone 'mock' package (and pyfakefs in validation), which weren't installed, causing pytest collection errors. Dev-only — prod uses 'pipenv sync' without --dev, so these don't ship. - run_tests: blank CKAN_SMTP_SERVER for test runs so suites never reach the dev stack's smtp4dev. Fixes ckanext-unaids' test_send_dataset_transfer_emails_errors, which asserts mail sending fails when no server is configured.
The validation/unaids extensions pin frictionless[ckan]==5.13.1 and pyfakefs==4.6.* in their own requirements and pass their own CI against those. Our merged Pipfile had frictionless>=5.0.0,<6.0.0 (drifted to a newer 5.x that dropped Resource.__create__) and pyfakefs=* (6.x dropped the CreateFile API), so ckanext-validation's test suite failed in our stack only. Aligning the pins fixes it without touching the submodule; frictionless is prod-facing but this matches the version the extensions are built and tested against.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-adds the CKAN extension test suite that died with the Jenkins pipeline, as a
testjob inbuild-deploy.yml.What runs
testjob — brings up the docker-compose dev stack viaadxand runs the 5 Fjelltopp suites (unaids, validation, scheming, dhis2harvester, emailasusername). No build/deploy.Build-time caching
.adxvenvkeyed onPipfile.lock— the dominant cost is bootstrap'spipenv sync --dev, not the image build.node_moduleskeyed on itsyarn.lock.Status
Wiring verified (YAML, job graph, test paths). The 5 suites are not yet proven green under CKAN 2.11/Py3.10 — this PR run is the proof. Once green, merging makes it a deploy gate.