fix: include build parameters in script-match cache key#1054
Open
arnoxit wants to merge 5 commits into
Open
Conversation
The per-resource Groovy script-match result cache (added in jenkinsci#966) keyed only on the script text, ignoring the parameter bindings passed into the script's evaluation. When several jobs share an identical match script and differ only by build parameters — a common pattern where the resource to lock is chosen by a parameter, e.g. `resourceName == TARGET_ENVIRONMENT` — the first job's per-resource results are cached and wrongly reused for every other job. Each subsequent job is told it matches the first job's resource (already locked) and blocks, so the jobs serialise behind a single resource instead of each locking its own free resource, for the duration of the cache TTL (default 30s). Fix: build the cache key from the script text plus a deterministic encoding of the parameter bindings, so evaluations with different parameters are cached separately. Adds a regression test (scriptResourceJobsRunConcurrently) submitting N jobs that share one parameter-driven match script; it asserts they run concurrently (peak concurrency N) rather than serialising to one at a time. Fails without this change, passes with it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mPokornyETM
approved these changes
Jun 18, 2026
mPokornyETM
left a comment
Contributor
There was a problem hiding this comment.
I checked the code very slightly and LGTM
@arnoxit thx for contributing
Reduce N from 12 to 5: still distinguishes serial dispatch (peak=1) from concurrent dispatch (peak=N), but needs fewer concurrent JenkinsRule builds and therefore less memory/scheduling headroom. Extend timeouts to match observed Windows agent slowness: - allStarted latch: 20s → 60s - build completion: 30s → 60s - build hold (release latch): 30s → 90s Windows CI agents took ~10 min total in the failing run; the 20s latch was too tight for N=12 builds to all start concurrently.
auto-merge was automatically disabled
June 30, 2026 04:58
Head branch was pushed to by a user without write access
Author
|
I have pushed a small change to the commit to help it pass the windows tests (which seem to have lower resource restrictions than the linux tests). It now passes that test and the Jenkins Security Scan is the last approval needed. Can a maintainer please trigger it? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: include build parameters in script-match cache key
Fixes #1052
Problem
When many jobs are queued together, each needing a distinct, immediately-available lockable resource selected by a Groovy match script, only one job runs at a time — the next dispatches only after the previous build completes. A wave of N such jobs serialises instead of running in parallel.
This shows up with the common pattern where the resource to lock is chosen by a build parameter, e.g. a match script:
…used by many jobs that share the identical script text and differ only by their parameter values (
TARGET_ENVIRONMENT, etc.). Each job needs its own free resource, so all should dispatch at once.Root cause
The per-resource Groovy script-match result cache introduced in #966 keys the cache on the script text only:
But the match result also depends on the parameter bindings passed into evaluation (
resourceName == TARGET_ENVIRONMENTevaluates differently per job). Because the key ignores parameters:slot-0 → true, all others→ false).It manifests as a queue/dispatch serialisation but is actually a cache-correctness bug; queued items show "Waiting for resources identified by custom script".
Fix
Build the cache key from the script text plus a deterministic, order-independent encoding of the parameter bindings, so evaluations with different parameters are cached as distinct entries:
Resources whose match script ignores parameters still share cache entries (key collapses to the script text when there are no params), preserving the original caching benefit.
Test
Adds
FreeStyleProjectTest#scriptResourceJobsRunConcurrently: submits N jobs sharing one parameter-driven, sandboxed match script, each matching a distinct free resource. Because instantaneous builds can't reveal serialisation via start-time spread, each build blocks on a latch and the test records peak concurrency:Verified the test fails without this change and passes with it (Jenkins 2.567, N=12).
Workaround (no plugin upgrade required)
Setting the controller JVM option
-Dorg.jenkins.plugins.lockableresources.SCRIPT_CACHE_TTL_MS=0disables the cache and avoids the serialisation, at the cost of evaluating the match script on every queue-maintenance pass. The cache-key fix is the durable solution that keeps the caching benefit.