Epic: Remote lockable resources

[kohtaro-satoh]

What feature do you want to see added?

Follow-up to #321 --- proposes a concrete, minimal-surface design for the "synchronize locked resources between multiple Jenkins instances" idea.

Status of this body: updated to reflect the finalized Phase 1 shape after the discussion in this thread (transparent DSL with optional forcedServerId, plus a GET /resources endpoint and the configuration surface in this comment). Sections "Configuration surface" through "Phase 1 scope" below are the authoritative Phase 1 specification.

Summary

Add an opt-in mechanism that lets lock(...) use a resource managed by another Jenkins. Two operating modes are provided through a single configuration field:

• Peer mode (default): a pipeline can opt into a specific remote with lock(..., serverId: 'X') { body }. Plain lock('X') { body } without serverId keeps its existing single-Jenkins behavior unchanged.
• Delegated mode (when forcedServerId is set on the local Jenkins): the same plain lock('X') { body } notation --- previously a single-Jenkins call --- is transparently routed to the configured remote Jenkins instead. The pipeline does not need to know about remoteness.

In both modes:

• The { body } passed to lock() still executes on the local Jenkins.
• The remote Jenkins is the single source of truth for its resources.
• Authentication uses a Jenkins username/password credential (username = remote service account, password = its API token), referenced by credentialsId on the local side.
• All Jenkins-to-Jenkins traffic is local -> remote only (no inbound connections back from remote to local).
• Communication failures are handled fail-closed; locks are not auto-released.

Inspired by the "lockable-master / lockable-slave" idea in #321 (comment). Delegated mode achieves that centralized arrangement with a single configuration field rather than a new role for the Jenkins; peer mode is preserved so Jenkins instances can also share resources mutually as independent peers.

Note: earlier drafts of this idea used the word "federation". The final shape is a much smaller surface --- minimal remote locking with transparent DSL and an explicit override. Broader federation concerns (multi-server routing, replication, HA, etc.) are intentionally left as future work.

Goals

• lock('X') { body } works transparently against a remote Jenkins when forcedServerId is configured (delegated mode).
• lock(..., serverId: 'X') { body } is available as an explicit per-call override (peer mode, debugging, operational overrides).
• The { body } passed to lock() still executes on the local Jenkins in all modes.
• The remote Jenkins is the single source of truth for its resources (availability, queue, timeout, selection strategy).
• Backward compatible: when forcedServerId is unset and no serverId argument is given, behavior is identical to today's single-Jenkins lock().
• Authentication via a Jenkins username/password credential (service user + API token), referenced by credentialsId on the local side.
• All Jenkins-to-Jenkins traffic is local -> remote only (no inbound connections back from remote to local).
• Safety-first on communication failures (do not auto-release locks).
• Assumed scale: small to medium deployments; a few-seconds polling delay and modest network overhead are acceptable.
• Remote resources must be pre-declared. The remote Jenkins will never auto-create a resource or label that is not already registered (no ephemeral / on-the-fly resource creation over the remote API).
• The "local -> remote only" rule applies per relation, not per Jenkins instance; Jenkins instances may simultaneously hold multiple independent relations (e.g. A->B for B's resources, B->A for A's resources, A->C for C's resources), enabling mutual sharing without any bidirectional channel.

Non-goals (initial)

• No serverId: 'any' / multi-server routing.
• No long-polling or push-based notifications (short-polling only).
• No cross-Jenkins state replication for lock decisions.
• No automatic lease-based release of locks.
• No freestyle project support in the first phase.
• No distributed consensus / HA orchestration.
• No transparent "federation" across multiple Jenkins instances.
• No auto-creation of ad-hoc / ephemeral resources or labels via the remote API (even though local lock() may create them for local use).
• No plugin-specific client allow-list in Phase 1; the remote API is protected by Jenkins' standard authentication and authorization.

High-level design

• The local side is a thin REST client around lock() semantics.
• The remote side exposes a versioned REST API under /lockable-resources/remote/v1/, separate from the existing /lockable-resources/api (different audience: machine-to-machine, not human UI).
• The remote API is off by default (remoteApiEnabled = false); installations are unaffected by upgrade until an administrator opts in.
• All transport is initiated by the local side (local -> remote only).
• The local side uses short-polling (a few-seconds interval) to observe acquisition state; no long-polling is used.
• The remote side tracks lastSeenAt per lease; leases with no heartbeat become STALE in UI but are not auto-released.
• Resource existence is enforced at acquire time. If the requested resource name or label does not match any pre-declared, exposed resource on the remote, the request is rejected immediately (HTTP error), with no lock state created and nothing to poll.
• HTTP method policy:
  • POST acknowledges requests and state transitions only (returns "accepted" / error; never returns acquisition outcome).
  • GET is the single source of truth for acquisition state and lease inspection.
  • Rationale: keeps the client loop uniform (POST /acquire -> poll GET /acquire/{requestId} -> act on state).

DSL resolution rules (Phase 1)

if forcedServerId is set:
    target = (forcedServerId, name)        # all locks are delegated to that remote Jenkins
                                           # an explicit serverId argument is silently ignored
                                           # (an INFO line is written to the build log)
else:
    if lock(..., serverId: 'X') is given:
        target = (X, name)                 # explicit override (peer mode / debugging)
    else:
        target = (LOCAL, name)             # original single-Jenkins behavior

In delegated mode, local resource definitions on this Jenkins are not used at all: resolution always goes to the remote Jenkins, the LR page shows the remote's published resources only, and unknown names fail immediately as UNKNOWN_RESOURCE. This eliminates name-collision questions and prevents "I thought I locked the remote one but actually locked a local one" accidents.

Mutual sharing via multiple independent one-way relations

• The local -> remote only rule is about a single client/server relation, not about the roles of the two Jenkins instances overall.
• Any number of Jenkins instances can freely establish multiple independent one-way relations at the same time. For example, between A and B (the same pattern extends to A<->C, B<->C, and so on):
  • For resources owned by B: A is local, B is remote. (A opens outbound HTTP to B with A's credentialsId.)
  • For resources owned by A: B is local, A is remote. (B opens outbound HTTP to A with B's credentialsId.)
• Each relation still obeys the same rules:
  • the single source of truth is the remote side of that relation,
  • traffic is initiated only by the local side of that relation,
  • failures are handled fail-closed on the local side of that relation.
• This yields mutual sharing without any new concept: no bidirectional channel, no "peer" role at the protocol level, no replication. Just ordinary one-way remote-lock relations coexisting.

Example (peer mode, explicit serverId):

A's pipeline:  lock(resource: 'board-a1',  serverId: 'B') { ... }
   # A is local, B is remote. A -> B HTTP only.

B's pipeline:  lock(resource: 'license-x', serverId: 'A') { ... }
   # B is local, A is remote. B -> A HTTP only.

A's pipeline:  lock(resource: 'staging',   serverId: 'C') { ... }
   # A is local, C is remote. A -> C HTTP only.

Each Jenkins acts as local for its own pipelines and as remote for resources it owns. The roles are per-relation, not per-Jenkins.

REST endpoints (/lockable-resources/remote/v1/*)

Where (base) = /lockable-resources/remote/v1 in the listing below. While remoteApiEnabled = false, all endpoints respond as if the API did not exist.

Acquire lifecycle (request side):

• POST (base)/acquire --- enqueue an acquire request. Returns {requestId} on acceptance.
  Request body may include heartbeatIntervalSeconds (optional in v1; see "Client-declared heartbeat interval" below).
  Does not return the acquisition outcome; callers must read GET (base)/acquire/{requestId} to observe the result.
  Accepts skipIfLocked as a hint; the outcome still materializes via GET as state ACQUIRED or SKIPPED.
  Rejected immediately (HTTP 4xx) if the resource/label is unknown or not exposed (e.g. UNKNOWN_RESOURCE, UNKNOWN_LABEL); no requestId is issued.
• GET (base)/acquire/{requestId} --- authoritative acquisition state: QUEUED / ACQUIRED / SKIPPED / FAILED / CANCELLED / EXPIRED.
  Polled by the local side every few seconds.
• POST (base)/acquire/{requestId}/cancel --- cancel a pending (not yet acquired) request.

Lease lifecycle (after acquisition):

• GET (base)/lease/{leaseId} --- inspect a currently held lease (diagnostics / UI). Response includes the negotiated heartbeatIntervalSeconds and the resulting staleThresholdSeconds.
• POST (base)/lease/{leaseId}/heartbeat --- liveness signal from the local side while the body runs.
• POST (base)/lease/{leaseId}/release --- release the lease when the body finishes (or is aborted).

Discovery:

• GET (base)/resources --- list resources currently exposed by this remote Jenkins (name, labels, description). State is intentionally not included to keep the endpoint cheap and cacheable; lease/state lookups continue to go through the per-lease endpoints. The local side short-caches this list to render its LR page.

Client loop (reference)

requestId = POST /acquire {..., skipIfLocked?, heartbeatIntervalSeconds?}
  # HTTP 4xx if resource/label is unknown or not exposed -> surface error, stop.
loop every few seconds:
    r = GET /acquire/{requestId}
    switch r.state:
      QUEUED    -> continue polling
      ACQUIRED  -> run body; send heartbeat periodically; POST release on exit
      SKIPPED   -> do not run body (skipIfLocked path)
      FAILED    -> surface error
      CANCELLED -> stop
      EXPIRED   -> stop (future: when maxWaitSeconds is set)

Configuration surface (Phase 1)

Roles are per relation, not per Jenkins instance --- a single Jenkins can act as "server" for one relation and as "client" for another at the same time.

Server-side settings (the Jenkins that exposes resources)

Setting	Default	Notes
remoteApiEnabled	false	Master switch. While false, all /remote/v1/* endpoints respond as if the API did not exist. Keeps existing installs unaffected by upgrade.
exposeLabel	(unset)	A single label name. Only resources carrying this label are visible/acquirable through the remote API. When unset, nothing is exposed (opt-in).

Notes:

• No plugin-specific allow-list of clients in Phase 1. The remote API is protected by Jenkins' standard authentication and authorization (API token), exactly like other Jenkins REST endpoints. A plugin-level allow-list / dedicated permission can be revisited in a later phase if there is demand.
• No per-server queue limit, no per-resource QoS in Phase 1.

Client-side settings (the Jenkins that initiates remote locks)

remotes is configured as a map keyed by serverId:

Setting	Notes
remotes[<serverId>]	Map of remote connections, keyed by the logical name assigned on the client side. The key is referenced from lock(..., serverId: 'X') and from forcedServerId.
remotes[<serverId>].url	Base URL of the remote Jenkins.
remotes[<serverId>].credentialsId	Jenkins Credentials ID. Expected to be a username/password credential whose username is the service account name on the remote Jenkins and whose password is that account's API token.
forcedServerId	Optional. When set, must match a key in remotes. Setting this turns the local Jenkins into delegated mode.

Notes:

• serverId is purely a client-side alias for a remote URL + credentials pair. The remote Jenkins is not aware of this name. The LR page on the client side shows this serverId.
• pollIntervalSeconds, heartbeatIntervalSeconds, requestTimeoutSeconds are not exposed as user settings in Phase 1. They are implementation-internal constants for now (see "Client-declared heartbeat interval" below for how the value is still future-proof at the API level).

Validation

• forcedServerId, when set, must match a key in remotes (otherwise: configuration error at save time).
• The "delegated mode" badge is shown clearly on the LR page when forcedServerId is set, so administrators are not surprised by the change in resolution semantics.

Explicitly out of scope for Phase 1 configuration

• Multiple forcedServerId entries / failover.
• A server-side "accept new acquires: ON/OFF" maintenance switch (Phase 2 candidate).
• Plugin-level client allow-list, per-client QoS, per-resource queue limits.

Client-declared heartbeat interval (forward-compatible default)

pollIntervalSeconds and heartbeatIntervalSeconds are not user-configurable in Phase 1, but they are not symmetric: the client decides how often it sends heartbeat, and the server must decide when a lease becomes STALE. To keep these two sides consistent --- and to leave room for making the interval configurable later without bumping the API version --- Phase 1 already carries the heartbeat interval on the wire.

POST /lockable-resources/remote/v1/acquire request body:

{
  "resource": "X",
  "skipIfLocked": false,
  "heartbeatIntervalSeconds": 10   // optional in v1
}

• heartbeatIntervalSeconds is optional in v1.
• If omitted, the server uses its built-in default (currently 10s).
• If outside the server's accepted range, the server rejects the request with HTTP 400 (INVALID_HEARTBEAT_INTERVAL). Silent rounding is intentionally avoided so misconfiguration is visible.

Server-side STALE threshold (Phase 1, hard-coded):

staleThresholdSeconds = max(heartbeatIntervalSeconds * 6, 60)

The factor (6) and the lower bound (60s) are hard-coded in Phase 1. They can be revisited later without changing the API contract.

GET /lockable-resources/remote/v1/lease/{leaseId} response includes both the negotiated heartbeatIntervalSeconds and the resulting staleThresholdSeconds, so operators can see exactly which values are in effect.

If we omitted heartbeatIntervalSeconds from v1 and added it later, every client that wants to use it would force a v2. Adding it now as an optional field means a future "make heartbeat interval configurable" change is just a UI/setting addition --- the API contract does not move.

UI updates on the local (client) side

• Peer mode (forcedServerId not set): the LR page shows local resources as today, plus any active remote leases this Jenkins currently holds (with their serverId).
• Delegated mode (forcedServerId set): the LR page shows the remote's published resources (from GET /resources) and the current remote leases held by this Jenkins. Local resource definitions are hidden or shown as "not used in delegated mode".
• In both modes, the displayed remote state is explicitly labeled as the client-side cached view, not the authoritative state on the remote Jenkins.

This way, what the user sees on the LR page and what lock('X') will actually try to acquire stay consistent.

On the server (remote) side, the LR page shows the client identifier (e.g. authenticated API user) in the status column for active remote leases, so administrators can tell which client holds what.

Phase 1 scope (finalized)

Included:

• REST API on the remote side under /lockable-resources/remote/v1/:
  • POST /acquire (with optional heartbeatIntervalSeconds), GET /acquire/{requestId}, POST /acquire/{requestId}/cancel
  • GET /lease/{leaseId} (returns negotiated heartbeat / stale values), POST /lease/{leaseId}/heartbeat, POST /lease/{leaseId}/release
  • GET /resources
• DSL: lock(..., serverId: 'X') as explicit override; transparent lock('X') resolution under forcedServerId.
• Configuration surface: remoteApiEnabled, exposeLabel on the server side; remotes (map) and forcedServerId on the client side.
• LR page integration on both sides as described above.
• Safety/versioning: heartbeat -> STALE only (no auto-release), fail-closed on errors, versioned path with 404/410 for retired versions, remote API protected by Jenkins' standard authentication/authorization.

Out of scope for Phase 1 (deferred or rejected):

• Multiple remote Jenkins instances with failover / round-robin.
• State mirroring / replication between Jenkins instances.
• Fixed master/slave roles at the Jenkins-instance level.
• serverId: 'any' style automatic selection.
• Cross-server label resolution.
• Server-side maintenance "pause new acquires" switch (good idea --- Phase 2 candidate).
• User-configurable polling / heartbeat / timeout values (the heartbeat interval is already carried on the wire, so enabling configuration later does not require an API version bump).
• Plugin-specific client allow-list or dedicated remote-API permission.

Background & motivation

Detailed background notes live in my sandbox repo (work-in-progress, English drafts):

• Background & motivation
• Realworld usecase (small/medium scale)
• Design rationale --- initial draft; predates the transparent-DSL / forcedServerId direction. The authoritative Phase 1 specification is this issue body (Sections "Configuration surface", "Client-declared heartbeat interval", "UI updates", and "Phase 1 scope").
• Existing plugin architecture notes

(Japanese originals are under docs-j/ in the same repo. The corresponding remote-lock-design-notes-j.md carries the same caveat --- it reflects the original idea and is not the latest specification; this issue is the source of truth.)

Phases (sub-Epics, to be filed)

• Phase 1 --- Remote lock via REST + transparent DSL + remote resource list view
  • M1: Core REST API + explicit lock(..., serverId: 'X') (peer mode only).
  • M2: forcedServerId resolution and the LR page mode-switching behavior.
  • M3: GET /resources and the client-side LR page integration with the remote view.
• Phase 2 --- Operations & observability hardening
  • Server-side maintenance switch ("accept new acquires: ON/OFF").
  • User-configurable polling / heartbeat / timeout values (the wire format already supports this).
  • Optional plugin-level client allow-list / dedicated remote-API permission, if there is demand.
• Phase 3 --- Future extensions (only if demand emerges)
  • Multi-server routing / failover.
  • Folder-level or job-level overrides.
  • Anything that today sits in "Non-goals".

Sub-Epic issues will be filed for each phase as they begin. Please discuss the high-level design here; phase-specific implementation details can wait.

Open questions

Most of the original open questions have been resolved in the discussion above. Remaining items where additional input is welcome:

• Default polling interval (current internal value: 3s).
• Default heartbeat interval / stale threshold (current internal values: 10s / max(heartbeat * 6, 60s)).
• Exact error shape (UNKNOWN_RESOURCE / UNKNOWN_LABEL) and HTTP status codes --- to be finalized during Phase 1 implementation.
• UI integration details for remote entries (merged vs separate tab, badge styling for delegated mode).
• Representation of remote owner/build identity in UI and logs on the server side.

Upstream changes

No. The proposal can be implemented within this plugin alone.

Are you interested in contributing this feature?

Yes. I plan to work on it in phases (see Phases section above). The first draft of Phase 1 (M1) will follow this body update.

[mPokornyETM]

@kohtaro-satoh It looks pretty good.

Q1:
What happens when:

1. lock from A --> B
2. A is dead.
3. What happens with resource on B? Shall I reset it manually in LR page on B side?

Q2:
What happens when:

1. lock from A --> B
2. B is dead.
3. What happens with resource on B? Shall I reset it manually in LR page on A side?, What happens, when the B re-started. Do we have still the same state of resource?

Q3:
How do you want to indicate, that the resource is locked from external API (Jenkins, GutHub ???)
PS - yes github. I am still wondering, that the GitHub still does not action like this. In that case we can provide new GitHub action which will use Jenkins to lock the resources. Jenkins might be started in some small docker image somewhere. This might partly close this gap.

Q4:
How we grant the correct API version?
When I update the B server, do I need update also the A server? Or can we grant backward compatibility for few versions?

[kohtaro-satoh]

@mPokornyETM -san;

Thanks for the questions.
For Q1 and Q2, system behavior and operational procedures (admin actions) are tightly coupled in a safety-first design, so the explanation may look a bit long.

A1

What happens when:

1. lock from A --> B
2. A is dead.
3. What happens with resource on B? Shall I reset it manually in LR page on B side?

Yes --- operationally a manual reset/release is required.

When Jenkins A goes down, heartbeats stop. Then Jenkins B can transition the lease to STALE.
After confirming that (a) the workload on A is really gone (not restarted / or already finished) and (b) the resource is physically healthy and free, an administrator on B can manually set it back to FREE (e.g. via the LR page / admin action).

A2

What happens when:

1. lock from A --> B
2. B is dead.
3. What happens with resource on B? Shall I reset it manually in LR page on A side?, What happens, when the B re-started. Do we have still the same state of resource?

If B goes down after acquisition, A will continue to call lease-related endpoints on B (heartbeat/release/etc.) using the leaseId, but those requests will fail (connection errors / timeouts) while B is unavailable. In any case, A should not automatically abort the running job just because B is unreachable (fail-closed / do not auto-release).

On the B side, before starting/restarting Jenkins B, an administrator should verify by directly checking the actual device/state that the relevant resources are physically healthy and free (rather than relying on Jenkins UI).
This is the same fundamental operational requirement as before introducing remote locking --- however, with remote locking, checking "is it really free?" may require coordinating with the remote client side (e.g. Jenkins A) as well, and this extra step is unavoidable operationally.

A3

How do you want to indicate, that the resource is locked from external API (Jenkins, GutHub ???)

I plan to extend the existing LR UI (<JenkinsURL>/lockable-resources/) to make this visible, e.g. by enhancing the Status and/or Action columns.
For example, Status could show not only the job/build name but also the client URL or client ID (the client would need to provide such an identifier, e.g. Jenkins-tokyo1, GitHub, etc.).

Regarding the GitHub Action idea: I agree this is an ambitious and interesting direction.

A4

How we grant the correct API version?
When I update the B server, do I need update also the A server? Or can we grant backward compatibility for few versions?

A typical timing to introduce a new major version (e.g. v1 -> v2) is when we need changes that are not backward-compatible with v1 semantics.

Even if we introduce v2, we should keep v1 working as long as v1 semantics remain valid and safe (backward compatibility by default).
However, if a future v3 addresses a serious security issue that makes v1 unacceptable, we should drop v1 at that time.

The compatibility model is simple: when a version is not supported anymore, the server would return HTTP 404 Not Found (or 410 Gone) for that versioned path.

Happy to clarify any of these points and update the proposal text to make the behaviors I intend more explicit.

[mPokornyETM]

Great, I calculated with this answer and makes me happy to hear, that it go the correct way. One or 2 points more.

I will have visual indication, that the resource on A is "linked" to resource on B.

Maybe we shall configure resource A as "symLink" from B, so I does not need to care about that in my pipeline.

It will be great, to have some over all Pause mode, that we can provide maintenance on B without disturbance on A.

[mPokornyETM]

@markWait what do you think about this idea and solution? Keep free to mark other jenkins contributors to get feedback, before we start with implementation

[kohtaro-satoh]

@mPokornyETM san;

Thanks again for the additional points and for inviting other maintainers to weigh in. Before more reviewers join, I'd like to consolidate where this design has moved during our discussion, because some parts have changed materially. Concrete answers to your three follow-up points are at the bottom.

1. Design pivot: pipelines should not need to know about "remote"

Your earlier "symLink" idea pushed me to reconsider the explicit serverId parameter on lock(...). After thinking it through, I agree that pipeline authors should not have to encode serverId in their Jenkinsfile just to use a resource that happens to live on another controller. That's an operational/topology concern, not a pipeline concern.

So I want to update the Phase 1 design accordingly, while keeping the safety properties we already discussed (A1–A4) intact.

What changes

• The DSL becomes transparent by default: pipelines keep writing lock('X') { ... }.
• Where that X is resolved is decided by controller-level configuration, not by the pipeline.

What does NOT change (intentionally)

• Communication model: still local → remote only, per-relation, no inbound channel from remote back to local.
• No master/slave role fixed at the controller level: any controller can be a "local" for some relations and a "remote" for other relations at the same time. (This is different from a centralized lockable-master design.)
• Safety semantics from A1–A4: heartbeat → STALE only, no auto-release, fail-closed on communication errors, versioned path with 404/410 for retired versions.
• Remote resources must be pre-declared on the remote side; no auto-creation over the remote API.

2. New configuration: forcedServerId (global)

To make the DSL transparent without re-introducing implicit magic, Phase 1 will add a single global setting in the Lockable Resources section of Manage Jenkins → System:

• forcedServerId (optional, single value)

Resolution rules

if forcedServerId is set:
    target = (forcedServerId, name)        # all locks are delegated to that remote
                                           # an explicit serverId argument is silently ignored
                                           # (an INFO line is written to the build log)
else:
    if lock(..., serverId: 'X') is given:
        target = (X, name)                 # explicit override (also useful for debugging)
    else:
        target = (LOCAL, name)             # original behavior, fully backward compatible

This gives us two coexisting operating modes from a single feature:

• Peer mode (forcedServerId not set): each controller is independent; pipelines can opt into a specific remote with an explicit serverId when they want to. This matches the "mutual sharing via independent one-way relations" model already described in the issue body.
• Delegated mode (forcedServerId set): the controller behaves like a "lockable-slave" that delegates every lock() to a single remote "lockable-master". This is conceptually the configuration you described in #321 (comment), achieved with one config field instead of a new role.

Why "forced" rather than "default"

In delegated mode, local resource definitions on this controller are not used at all:

• Resolution always goes to the remote.
• The LR page on this controller shows the remote's published resources only.
• A lock('X') for an unknown name on the remote fails immediately with UNKNOWN_RESOURCE (no silent fallback to local).

This eliminates name-collision questions entirely (we don't need to decide which X wins) and, more importantly, removes a class of "I thought I locked the remote one but I actually locked a local one" accidents. It is a deliberately strict mode.

Why lock(..., serverId: ...) is preserved

Even after transparency, I want to keep the explicit serverId parameter on lock(...) in the DSL:

• In peer mode it is the way to address a specific remote.
• It is very useful for debugging and operational overrides ("force this one call to go to server C").
• In delegated mode it is silently ignored (INFO-logged), so pipelines remain portable across controllers.

Documentation will describe it as an explicit override, not as a "legacy" parameter.

3. Additional API: list published resources on the remote

To make the A-side LR page useful in delegated mode (and informative in peer mode), Phase 1 will add one more read-only endpoint:

• GET /lockable-resources/remote/v1/resources — returns the list of resources the remote exposes (name, labels, description). State is intentionally not included in this endpoint to keep it cheap and cacheable; lease/state lookups continue to go through the existing per-lease endpoints.

The A side will short-cache this list (a few tens of seconds) and use it to render a remote view on its LR page.

4. Configuration surface (Phase 1)

To keep Phase 1 reviewable, the configuration surface is intentionally minimal. Roles are per relation, not per controller — a single controller can act as "server" for one relation and as "client" for another at the same time.

Server-side settings (the controller that exposes resources)

Setting	Default	Notes
remoteApiEnabled	false	Master switch. While false, all /remote/v1/* endpoints respond as if the API did not exist. Keeps existing installs unaffected by upgrade.
exposeLabel	(unset)	A single label name. Only resources carrying this label are visible/acquirable through the remote API. When unset, nothing is exposed (opt-in).

Notes:

• No plugin-specific allow-list of clients in Phase 1. The remote API is protected by Jenkins' standard authentication and authorization (API token), exactly like other Jenkins REST endpoints. A plugin-level allow-list / dedicated permission can be revisited in a later phase if there is demand.
• No per-server queue limit, no per-resource QoS in Phase 1.

Client-side settings (the controller that initiates remote locks)

Setting	Notes
remotes[]	List of remote connections.
remotes[].serverId	Logical name assigned on the client side for this connection. Referenced from lock(..., serverId: 'X') and from forcedServerId. Must be unique within this controller.
remotes[].url	Base URL of the remote Jenkins.
remotes[].credentialsId	Jenkins Credentials ID. Expected to be a username/password credential whose username is the service account name on the remote and whose password is that account's API token.
forcedServerId	Optional. When set, must match one of remotes[].serverId. Setting this turns the controller into delegated mode.

Notes:

• serverId is purely a client-side alias for a remote URL + credentials pair. The server is not aware of this name. The LR page on the client shows this serverId.
• pollIntervalSeconds, heartbeatIntervalSeconds, requestTimeoutSeconds are not exposed as user settings in Phase 1. They are implementation-internal constants for now (see Section 5 for how heartbeat interval is still made future-proof at the API level).

Validation

• forcedServerId, when set, must match a remotes[].serverId (otherwise: configuration error at save time).
• remotes[].serverId values must be unique within the controller.
• The "delegated mode" badge is shown clearly on the LR page when forcedServerId is set, so administrators are not surprised by the change in resolution semantics.

Explicitly out of scope for Phase 1 configuration

• Multiple forcedServerId entries / failover.
• A server-side "accept new acquires: ON/OFF" maintenance switch (Phase 2 candidate).
• Plugin-level client allow-list, per-client QoS, per-resource queue limits.

5. Client-declared heartbeat interval (forward-compatible default)

pollIntervalSeconds and heartbeatIntervalSeconds are not user-configurable in Phase 1, but they are not symmetric: the client decides how often it sends heartbeat, and the server must decide when a lease becomes STALE. To keep these two sides consistent — and to leave room for making the interval configurable later without bumping the API version — Phase 1 already carries the heartbeat interval on the wire.

POST /lockable-resources/remote/v1/acquire request body

{
  "resource": "X",
  "skipIfLocked": false,
  "heartbeatIntervalSeconds": 10   // optional in v1
}

• heartbeatIntervalSeconds is optional in v1.
• If omitted, the server uses its built-in default (currently 10s).
• If outside the server's accepted range, the server rejects the request with HTTP 400 (INVALID_HEARTBEAT_INTERVAL). Silent rounding is intentionally avoided so misconfiguration is visible.

Server-side STALE threshold (Phase 1, hard-coded)

staleThresholdSeconds = max(heartbeatIntervalSeconds * 6, 60)

The factor (6) and the lower bound (60s) are hard-coded in Phase 1. They can be revisited later without changing the API contract.

GET /lockable-resources/remote/v1/lease/{leaseId} response

The response includes both the negotiated heartbeatIntervalSeconds and the resulting staleThresholdSeconds, so operators can see exactly which values are in effect:

{
  "leaseId": "...",
  "resource": "X",
  "heartbeatIntervalSeconds": 10,
  "staleThresholdSeconds": 60,
  "lastSeenAt": "..."
}

Why declare it on the wire even though it's not user-configurable yet

If we omit heartbeatIntervalSeconds from v1 and add it later, every client that wants to use it forces a v2. Adding it now as an optional field means a future "make heartbeat interval configurable" change is just a UI/setting addition — the API contract does not move.

6. UI updates on A side

• Peer mode: the LR page shows local resources as today, plus any active remote leases this controller currently holds (with their serverId).
• Delegated mode: the LR page shows the remote's published resources (from the new GET /resources) and the current remote leases held by this controller. Local resource definitions are hidden or shown as "not used in delegated mode".
• In both modes, the displayed remote state is explicitly labeled as the A-side cached view, not the authoritative state on B.

This way, what the user sees on the LR page and what lock('X') will actually try to acquire stay consistent.

7. Phase 1 scope (updated and finalized)

Included:

• REST API on remote side under /lockable-resources/remote/v1/:
  • POST /acquire (with optional heartbeatIntervalSeconds), GET /acquire/{requestId}, POST /acquire/{requestId}/cancel
  • GET /lease/{leaseId} (returns negotiated heartbeat / stale values), POST /lease/{leaseId}/heartbeat, POST /lease/{leaseId}/release
  • GET /resources (new)
• DSL: lock(..., serverId: 'X') as explicit override; transparent lock('X') resolution under forcedServerId.
• Configuration surface as in Section 4 (remoteApiEnabled, exposeLabel on the server side; remotes[], forcedServerId on the client side).
• A-side LR page integration as described above.
• B-side LR page: show client identifier (e.g. authenticated API user) in the status column for active remote leases (this is the visualization piece from A3).
• Safety/versioning per A1–A4. Remote API protected by Jenkins' standard authentication/authorization.

Out of scope for Phase 1 (deferred or rejected):

• Multiple remote servers with failover / round-robin.
• State mirroring / replication between controllers.
• Fixed master/slave roles at the controller level.
• serverId: 'any' style automatic selection.
• Cross-server label resolution.
• Server-side maintenance "pause new acquires" switch (good idea — Phase 2 candidate).
• User-configurable polling / heartbeat / timeout values (the heartbeat interval is already carried on the wire, so enabling configuration later does not require an API version bump).
• Plugin-specific client allow-list or dedicated remote-API permission.

8. Direct answers to your three follow-up points

I will have visual indication, that the resource on A is "linked" to resource on B.

Agreed and included in Phase 1 (see the UI section above). A side will display remote leases it currently holds and, in delegated mode, the remote's published resources. The display is clearly marked as the A-side cached view.

Maybe we shall configure resource A as "symLink" from B, so I do not need to care about that in my pipeline.

I think forcedServerId (delegated mode) achieves the same end result you're after — pipelines stay as lock('X') and don't need to know about remote-ness — without introducing a per-resource symlink construct. A symlink-style mechanism (per-resource alias, mixed local + remote on the same controller) raises non-trivial questions (collision handling, failure semantics when the target is unreachable, configuration consistency) and I'd prefer to keep that out of Phase 1.

It will be great to have some over all Pause mode, that we can provide maintenance on B without disturbance on A.

Agreed in spirit. A reasonable shape would be a B-side admin switch "accept new acquires: ON / OFF": when OFF, new acquire requests are rejected (e.g. HTTP 503) so the A side can back off and retry, while existing leases (heartbeat / release) keep working so in-flight jobs are not disturbed. I'd like to put this in Phase 2 rather than Phase 1, to keep Phase 1 focused.

9. About implementation pace

There is no fixed deadline on my side. Given the scope above, I'm planning to implement Phase 1 in three internal milestones, so each step can be reviewed independently rather than as one large PR:

1. Core REST API + explicit lock(..., serverId: 'X') (peer mode only).
2. forcedServerId resolution and the LR page mode-switching behavior.
3. GET /resources and the A-side LR page integration with the remote view.

I will update the issue body to reflect this finalized Phase 1 shape after this comment, so newcomers don't have to reconstruct the design from the discussion thread. Feedback on the updated direction — especially on the forcedServerId semantics, the configuration surface in Section 4, and the new GET /resources endpoint — is very welcome.

[mPokornyETM]

one small point.

about remotes[]. It willbe etter to use map [:] instead of list, where the serverId is the map key as well. The search will be much more faster. Everything else LGTM and you can start with the first draft.

[kohtaro-satoh]

@mPokornyETM san;

Good catch --- Map/[:] keyed by serverId makes more sense given that serverId is unique and is the lookup key everywhere (resolution, UI, forcedServerId). It also makes the JCasC representation cleaner.

I'll update the issue body accordingly when I refresh it with the finalized Phase 1 shape, and use a Map<String, RemoteConfig> (or its equivalent in the Jenkins config UI: a repeatable list internally with a uniqueness constraint on the key, exposed as a map at the API/lookup layer).

Thanks for the LGTM --- I'll start the first draft.

[kohtaro-satoh]

@mPokornyETM san;

An initial implementation for Phase 1 / Milestone 1 is now available in my fork:

• Repository: kohtaro-satoh/lockable-resources-plugin
• Branch: feature/1025-remote-lockable-resources-p1-m1

I also ran focused end-to-end tests for the current Phase 1 / M1 scope, covering peer-mode routing, contention / waiting behavior, skipIfLocked, fail-closed cases, and several multi-controller scenarios.

Implementation notes, work logs, and E2E materials are tracked in my notes repo:

• Repository: kohtaro-satoh/lockable-resources-remote-notes
• E2E specification: dev/docs-e/E2E_TEST_SPECIFICATION.md
• Latest full E2E report: dev/reports/20260523133947-e2e-test.md

I am preparing a draft PR against jenkinsci/lockable-resources-plugin:master.

[kohtaro-satoh]

Small follow-up: this work is expected to conflict with #1035, so I plan to rebase and resolve those conflicts after #1035 is merged, and then open the draft PR.

[adityajalkhare]

@kohtaro-satoh, #1035 has been merged. Thanks for waiting :)

[kohtaro-satoh]

@mPokornyETM san, @adityajalkhare san;

Thanks for the discussion here and for merging #1035.

A small update from my side: while waiting for #1035, I kept working on and rethinking the Phase 1 / M1 implementation. The M1 implementation and tests themselves are already in place, but during that process I came to feel that M1 is not the right PR boundary anymore.

I now think the minimum practically usable and API-stable slice is what I currently call M1A, not M1.

So instead of opening a PR for M1, I would like to first share the M1A direction here and then move the implementation to that point before opening the PR.

I realize this is a change from my earlier plan, so apologies for the shift. My intention here is to avoid submitting an intermediate API shape that I already expect to revise soon after.

I will post the M1 -> M1A delta in a follow-up comment shortly. Feedback would be very welcome.

[kohtaro-satoh]

Here is the high-level M1 -> M1A direction I would like to discuss before opening the PR.

In short, I now want the first PR boundary to be the point where the remote layer already behaves as a transparent wrapper for lock(...) semantics, rather than just a thinner remote acquire/release transport.

What stays unchanged

The following Phase 1 principles stay unchanged:

• communication remains local -> remote only
• the remote side remains the single source of truth
• { body } still runs on the local Jenkins
• failures remain fail-closed
• no automatic release on communication failure
• the remote API remains versioned under /lockable-resources/remote/v1/

Why M1 is no longer the right PR boundary

After implementing and testing M1, I no longer think it is the right externally reviewable slice.

M1 is implementable, but it still leaves the protocol in a shape that feels too much like a first transport draft. I would likely revise that contract immediately in the next step in order to preserve better semantic equivalence with the existing local lock() behavior.

So my current view is that M1 is a valid internal milestone, but not the best upstream review boundary.

What changes from M1 to M1A

At a high level, M1A changes the design in the following way:

1. The acquire payload becomes a transparent lockRequest

   • POST /acquire carries the lock-semantics payload more directly.
   • This includes parameters such as resource, label, quantity, variable, skipIfLocked, priority, and timeoutForAllocateResource.
   • serverId and forcedServerId remain routing/configuration concerns and are not part of lockRequest.

2. The acquired result returns lockEnvVars

   • GET /acquire/{lockId} returns lockEnvVars when state = ACQUIRED.
   • The intention is to preserve the same environment-variable expansion model as the local lock(), e.g. LOCKED_RESOURCE, LOCKED_RESOURCE0, LOCKED_RESOURCE1, etc.

3. Identifier handling is simplified

   • Instead of keeping separate requestId and leaseId, M1A unifies them as a single lockId.
   • The same lockId is then used for polling, heartbeat, and release.

4. cancel is removed from this first PR slice

   • I no longer think a separate cancel endpoint is necessary in the first externally reviewed boundary.
   • For the first PR, I would rather keep the client loop focused on acquire / poll / heartbeat / release.

5. M1A includes both explicit peer routing and controller-side delegated routing

   • lock(..., serverId: 'X') remains the explicit peer-mode path.
   • forcedServerId is also included in M1A as a controller-side routing setting, while remaining outside the lock-semantics payload itself.
   • In other words, M1A aims to establish the transport/protocol shape and the routing model together, instead of separating them across two PR boundaries.

Why I want M1A to be the first PR boundary

The main reason is API stability.

If I submit M1 first, I may end up freezing an intermediate remote-lock contract and then immediately changing:

• the request shape,
• the response shape,
• the identifier model,
• and the way local lock() semantics are represented over the wire.

I would rather make the first PR the point where the protocol already reflects the intended semantic direction more faithfully.

I have written up the current M1A design note in English here:

• LRR_DESIGN_P1_M1A.md in kohtaro-satoh/lockable-resources-remote-notes

I will continue working toward this M1A shape from here, and any feedback during that process would be very welcome. If there are suggestions on the design direction, I would be glad to consider them and incorporate them where appropriate before opening the PR.

[mPokornyETM]

lets try it. Please ping me here, when oyu have some draft. I will try to test it afterwards