Skip to content

fix: include build parameters in script-match cache key#1054

Open
arnoxit wants to merge 5 commits into
jenkinsci:masterfrom
arnoxit:fix/script-cache-key-ignores-params
Open

fix: include build parameters in script-match cache key#1054
arnoxit wants to merge 5 commits into
jenkinsci:masterfrom
arnoxit:fix/script-cache-key-ignores-params

Conversation

@arnoxit

@arnoxit arnoxit commented Jun 13, 2026

Copy link
Copy Markdown

fix: include build parameters in script-match cache key

Fixes #1052

Problem

When many jobs are queued together, each needing a distinct, immediately-available lockable resource selected by a Groovy match script, only one job runs at a time — the next dispatches only after the previous build completes. A wave of N such jobs serialises instead of running in parallel.

This shows up with the common pattern where the resource to lock is chosen by a build parameter, e.g. a match script:

return LOCK_ENVIRONMENT && resourceName == TARGET_ENVIRONMENT

…used by many jobs that share the identical script text and differ only by their parameter values (TARGET_ENVIRONMENT, etc.). Each job needs its own free resource, so all should dispatch at once.

Root cause

The per-resource Groovy script-match result cache introduced in #966 keys the cache on the script text only:

// LockableResource.scriptMatches(...)
String cacheKey = script.getScript();

But the match result also depends on the parameter bindings passed into evaluation (resourceName == TARGET_ENVIRONMENT evaluates differently per job). Because the key ignores parameters:

  1. The first job evaluates the script against every resource with its parameters and populates each resource's cache (e.g. slot-0 → true, all others → false).
  2. Every subsequent job, sharing the identical script text, gets cache hits computed with the first job's parameters. They are all told they match the first job's resource (now locked) and that their own resource does not match.
  3. So every job queues behind the single resource the first job took, and they serialise — for the duration of the cache TTL (default 30s), repeating as entries expire.

It manifests as a queue/dispatch serialisation but is actually a cache-correctness bug; queued items show "Waiting for resources identified by custom script".

Fix

Build the cache key from the script text plus a deterministic, order-independent encoding of the parameter bindings, so evaluations with different parameters are cached as distinct entries:

String cacheKey = buildScriptCacheKey(script.getScript(), params);

Resources whose match script ignores parameters still share cache entries (key collapses to the script text when there are no params), preserving the original caching benefit.

Test

Adds FreeStyleProjectTest#scriptResourceJobsRunConcurrently: submits N jobs sharing one parameter-driven, sandboxed match script, each matching a distinct free resource. Because instantaneous builds can't reveal serialisation via start-time spread, each build blocks on a latch and the test records peak concurrency:

  • With the bug: only the first build starts and blocks; peak concurrency = 1.
  • With the fix: all N run together; peak concurrency = N.

Verified the test fails without this change and passes with it (Jenkins 2.567, N=12).

Workaround (no plugin upgrade required)

Setting the controller JVM option -Dorg.jenkins.plugins.lockableresources.SCRIPT_CACHE_TTL_MS=0 disables the cache and avoids the serialisation, at the cost of evaluating the match script on every queue-maintenance pass. The cache-key fix is the durable solution that keeps the caching benefit.

The per-resource Groovy script-match result cache (added in jenkinsci#966) keyed
only on the script text, ignoring the parameter bindings passed into the
script's evaluation.

When several jobs share an identical match script and differ only by build
parameters — a common pattern where the resource to lock is chosen by a
parameter, e.g. `resourceName == TARGET_ENVIRONMENT` — the first job's
per-resource results are cached and wrongly reused for every other job.
Each subsequent job is told it matches the first job's resource (already
locked) and blocks, so the jobs serialise behind a single resource instead
of each locking its own free resource, for the duration of the cache TTL
(default 30s).

Fix: build the cache key from the script text plus a deterministic encoding
of the parameter bindings, so evaluations with different parameters are
cached separately.

Adds a regression test (scriptResourceJobsRunConcurrently) submitting N jobs
that share one parameter-driven match script; it asserts they run
concurrently (peak concurrency N) rather than serialising to one at a time.
Fails without this change, passes with it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@mPokornyETM mPokornyETM left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the code very slightly and LGTM
@arnoxit thx for contributing

@mPokornyETM mPokornyETM enabled auto-merge (squash) June 24, 2026 20:09
arnoxit and others added 2 commits June 30, 2026 07:54
Reduce N from 12 to 5: still distinguishes serial dispatch (peak=1) from
concurrent dispatch (peak=N), but needs fewer concurrent JenkinsRule
builds and therefore less memory/scheduling headroom.

Extend timeouts to match observed Windows agent slowness:
- allStarted latch: 20s → 60s
- build completion: 30s → 60s
- build hold (release latch): 30s → 90s

Windows CI agents took ~10 min total in the failing run; the 20s latch
was too tight for N=12 builds to all start concurrently.
auto-merge was automatically disabled June 30, 2026 04:58

Head branch was pushed to by a user without write access

@arnoxit

arnoxit commented Jun 30, 2026

Copy link
Copy Markdown
Author

I have pushed a small change to the commit to help it pass the windows tests (which seem to have lower resource restrictions than the linux tests). It now passes that test and the Jenkins Security Scan is the last approval needed. Can a maintainer please trigger it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LockRunListener.onCompleted() dispatches only one blocked job per completion since Jenkins 2.564

2 participants