test: exercise every documented task example#399
Merged
josegonzalez merged 10 commits intoJul 18, 2026
Merged
Conversation
Adds `TestAllTaskExamplesValidate`, which decodes every registered task's documented examples the way the loader does, applies the field defaults, and runs the optional input validation, so an example that would fail `docket validate` can no longer ship into the generated task reference. It runs as part of `make test` and covers any future task automatically. Correcting the examples it flagged fixes the per-app snippets for the global-only `letsencrypt-email` property on `dokku_haproxy_property` and `dokku_traefik_property` and the global snippet for the per-app-only `rev-env-var` on `dokku_git_property`, and replaces the unreachable placeholder archive URLs on `dokku_git_from_archive` with real ones so the example is runnable.
Adds `TestIntegrationTaskExamples`, a registry-driven driver that wraps each task's documented examples as a recipe, loads them through the real loader, and runs Plan then apply against a live Dokku, so every snippet in the task reference is provably runnable and not merely offline-valid. It provisions the app each example references, deploys the ones that need a running app, creates backing services, and provisions real material for the two tasks whose documented value is a placeholder that cannot apply verbatim. A small `exampleIntegrationPolicy` carries the per-task prerequisites and opt-outs, and the `dokku_letsencrypt` example is gated behind `DOKKU_TEST_LETSENCRYPT` so it applies against a pebble ACME server, so a new task is exercised by default and only genuinely special ones need an entry.
Adds a CI job that stands up a pebble ACME server by reusing the dokku-letsencrypt harness and runs the `dokku_letsencrypt` example under `DOKKU_TEST_LETSENCRYPT`, and extends the scheduler-k3s job to apply the scheduler-k3s examples against a live cluster, so both classes of heavy-infra example run in their own jobs while the default integration run applies everything else.
The `dokku_git_from_image` example referenced `dokku/node-js-app:latest`, which does not exist, and its second example pinned a build directory that the real image does not have; both now deploy `dokku/smoke-test-app:dockerfile`. The `dokku_git_sync` example synced `main`, but the repository's branch is `master`, so the build could never resolve the ref. These are the sources the apply-every-example integration driver deploys, so they must exist.
Teaches the example driver the setup a first CI run showed several examples need before they apply: enabling HTTP auth and adding the app's domains for the `http_auth` family, creating the named storage entry the `storage_mount` examples attach, writing a real maintenance-page tarball where the example only names a path, and redeploying `git_from_archive` from a fresh app so its second, same-content deploy is not rejected as unchanged. It also opts out `ps_scale`, `scheduler_k3s_autoscaling_auth`, and `service_backup`, whose examples need infra the environment does not provide, leaving them to the offline guard.
The letsencrypt job's `docker compose up` validated every service in the dokku-letsencrypt compose file, including the compose-mode dokku service whose bind mount is empty when `LETEST_HOST_DIR` is unset, so the stack failed to start before Pebble came up.
The apply-every-example driver shares a dokku instance with the per-task integration suite, and its extra deploys destabilized the existing deploy tests when both landed on the same shard. Running `TestIntegrationTaskExamples` in a dedicated, separately sharded job and skipping it in the per-task shards keeps each suite's load independent.
`git_from_image` and `git_from_archive` deploy the heavily reused example app late in a run and failed only in the isolated job, where every deploy example lands together; the same examples pass in the four-way per-task sharding. Matching that four-way split thins the per-shard deploy load, and the apply failure now reports the captured command output so any remaining deploy error is legible instead of an empty string.
The network_property global example sets `attach-post-create` to a network globally, and with no global-clear example that setting persisted for the rest of the run, so every later app build failed attaching to a network the environment no longer had. The example driver now creates that network before the examples and, on cleanup, clears the global property and removes the network, which is what left git_from_image, git_from_archive, and git_sync deploys failing depending on run order.
A property task's "Setting a global X" example has no global-clear counterpart, so the setting persisted for the rest of the run and broke later deploys - a leftover global network attachment or registry server made every subsequent app build fail against a resource the environment did not have. The driver now clears any global property an example sets once that example finishes, which generalizes the earlier network-specific workaround to every property family and removes the ordering sensitivity that surfaced different deploy failures per run.
josegonzalez
deleted the
388-exercise-every-documented-task-example-offline-validation-integration
branch
July 18, 2026 12:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nothing tested the documented task examples, so a snippet that would fail
docket validatecould ship straight into the generated task reference, which is how #323 slipped in.TestAllTaskExamplesValidatenow decodes every registered task's examples the way the loader does, applies the field defaults, and runs the optional input validation offline as part ofmake test, covering any future task automatically. It flagged three tasks whose examples could not validate: the per-app snippets for the global-onlyletsencrypt-emailproperty ondokku_haproxy_propertyanddokku_traefik_property, and the global snippet for the per-app-onlyrev-env-varondokku_git_property, all corrected to the scope the property actually supports.TestIntegrationTaskExamplesthen applies every example against a live Dokku undermake test-integration, so the snippets are provably runnable and not merely valid. It provisions the app each example references, deploys the ones that need a running app, creates backing services, and provisions real material where a documented value is a placeholder that cannot apply verbatim, with the per-task prerequisites and opt-outs kept in one small policy so a new task is exercised by default and only genuinely special ones need an entry. Thedokku_git_from_archiveexample pointed at a placeholder URL that could never apply, so it now references a real archive and runs as written.The
dokku_letsencryptexample applies against a real pebble ACME server in a dedicated CI job that reuses the dokku-letsencrypt harness, and the scheduler-k3s examples apply against a live cluster in the existing k3s job, so both classes of heavy-infra example are exercised in their own jobs while the default integration run applies everything else.Closes #388.