Jenkins and plugins versions report
Newest LTS Jenkins version and Newest git-client-plugin
What Operating System are you using (both controller, and any agents involved in the problem)?
RHEL 9
RKE 2 1.34.6
Reproduction steps
Inside a Jenkins container (controller or agent):
touch /tmp/test && chmod 400 /tmp/testt
ls -lZ /tmp/test # container_file_t readable
chcon --type=ssh_home_t /tmp/test; echo $? # exits 0
ls /tmp/test # Permission denied
After the relabel succeeds, the calling process can no longer access its own file. ls -l from the calling context shows ?????????? for the file mode, because stat() is denied.
When the git-client-plugin runs this same chcon against the temporary key it writes to tmp/jenkins-gitclient-ssh*.key, the subsequent ssh invocation fails with:
Warning: Identity file …/jenkins-gitclient-ssh….key not accessible: Permission denied.
and the git operation failss
On some runs on some clusters we have also seen the relabel itself be denied by the kernel generating an AVC like:
type=AVC msg=audit(...): avc: denied { relabelto } for pid=... comm="chcon"
name="jenkins-gitclient-ssh...key"
scontext=system_u:system_r:container_t:s0:c290,c1002
tcontext=system_u:object_r:ssh_home_t:s0:c290,c1002
tclass=file permissive=0
In that case the file keeps its container_file_t label and the build often succeeds despite the [WARNING] Failed (1) performing chcon helper command for SELinux log line. Whether a given run takes the "relabel denied" path or the "relabel succeeds, subsequent reads denied" path appears to depend on the underlying filesystem and host policy root cause of the variation is still under investigaton on our side.
Expected Results
The plugin does not relabel files to a type that the calling process cannot then read
Actual Results
Summary
fixSELinuxLabel() runs chcon --type='ssh_home_t on the temporary SSH key file when it detects a labeled security context. This is correct when the ssh client that will read the key runs under sshd_t (or another domain for which ssh_home_t is the expected file tyype).
When Jenkins runs inside a container under the standard container-selinux policy (Kubernetes pods labeled container_t, including RKE2, OpenShift, plain containerd/CRI-O), the ssh client invoked by git also runs as container_t. Under that policy:
container_t can read container_file_t (the default workspace label).container_t cannot getattr/read/open files labeled ssh_home_t.
So relabeling the tmp key to `ssh_home_t makes it unreadable to the same process that needs to read it.
We have observed this failure on RKE2 with rke2-selinux and container-selinux installed and SELinux enforcing, with the Jenkins controller and inbound agent containers both running as container_t.
Anything else?
The function currently checks:
- `/proc/self/attr/current contains a labeled context.
/sys/fs/selinux/enforce reads as 1.
Neither check answers the question that actually matters: is the process domain one for which ssh_home_t is the correct file type? A process running as container_t will not benefit from the relabel and will be actively hamed by it.
Suggested fixes
-
Add a system property to skip the relabel. Somthing loke skipSELinuxRelabeldefaulting tofalse` to preserve existing behavior, settable by operators who know their environment doesn't need or want it. Lowest-risk change and allows site self-mitigation without rebuilding base images.
-
Detect the calling SELinux domain and skip the relabel for container domains. Read the type from /proc/self/attr/current; if it's container_t (or matches a list of container-family domains: container_init_t, container_user_t, etc.), skip the chcon. The temp key keeps its container_file_t label, which container_t ssh can already read.
-
Verify after relabel. After chcon exits 0, do a Files.isReadable(key) (or equivalent) check from the plugin. If the file is no longer readable, revert the label or surface a clear error explaining what happened
-
Document the failure mode. Note in the plugin docs that on systems where Jenkins runs under container_t, the chcon step is at best a no-op and at worst breaks the build, and that shimming chcon to exit 0 inside the image is a known workaround.
Workaround we're using
RUN printf '#!/bin/sh\nexit 0\n' > /usr/local/bin/chcon && chmod 0755 /usr/local/bin/chcon
Placed in both controller and agent images. Verified to fix the failure across all affected clusters.
Are you interested in contributing a fix?
No response
Jenkins and plugins versions report
Newest LTS Jenkins version and Newest git-client-plugin
What Operating System are you using (both controller, and any agents involved in the problem)?
RHEL 9
RKE 2 1.34.6
Reproduction steps
Inside a Jenkins container (controller or agent):
After the relabel succeeds, the calling process can no longer access its own file.
ls -lfrom the calling context shows??????????for the file mode, becausestat()is denied.When the git-client-plugin runs this same chcon against the temporary key it writes to
tmp/jenkins-gitclient-ssh*.key, the subsequent ssh invocation fails with:and the git operation failss
On some runs on some clusters we have also seen the relabel itself be denied by the kernel generating an AVC like:
In that case the file keeps its
container_file_tlabel and the build often succeeds despite the[WARNING] Failed (1) performing chcon helper command for SELinuxlog line. Whether a given run takes the "relabel denied" path or the "relabel succeeds, subsequent reads denied" path appears to depend on the underlying filesystem and host policy root cause of the variation is still under investigaton on our side.Expected Results
The plugin does not relabel files to a type that the calling process cannot then read
Actual Results
Summary
fixSELinuxLabel()runschcon --type='ssh_home_ton the temporary SSH key file when it detects a labeled security context. This is correct when the ssh client that will read the key runs undersshd_t(or another domain for whichssh_home_tis the expected file tyype).When Jenkins runs inside a container under the standard
container-selinuxpolicy (Kubernetes pods labeledcontainer_t, including RKE2, OpenShift, plain containerd/CRI-O), the ssh client invoked by git also runs ascontainer_t. Under that policy:container_tcan readcontainer_file_t(the default workspace label).container_tcannotgetattr/read/openfiles labeledssh_home_t.So relabeling the tmp key to `ssh_home_t makes it unreadable to the same process that needs to read it.
We have observed this failure on RKE2 with
rke2-selinuxandcontainer-selinuxinstalled and SELinux enforcing, with the Jenkins controller and inbound agent containers both running ascontainer_t.Anything else?
The function currently checks:
/sys/fs/selinux/enforcereads as1.Neither check answers the question that actually matters: is the process domain one for which
ssh_home_tis the correct file type? A process running ascontainer_twill not benefit from the relabel and will be actively hamed by it.Suggested fixes
Add a system property to skip the relabel. Somthing loke skipSELinuxRelabel
defaulting tofalse` to preserve existing behavior, settable by operators who know their environment doesn't need or want it. Lowest-risk change and allows site self-mitigation without rebuilding base images.Detect the calling SELinux domain and skip the relabel for container domains. Read the type from
/proc/self/attr/current; if it'scontainer_t(or matches a list of container-family domains:container_init_t,container_user_t, etc.), skip the chcon. The temp key keeps itscontainer_file_tlabel, whichcontainer_tssh can already read.Verify after relabel. After
chconexits 0, do aFiles.isReadable(key)(or equivalent) check from the plugin. If the file is no longer readable, revert the label or surface a clear error explaining what happenedDocument the failure mode. Note in the plugin docs that on systems where Jenkins runs under
container_t, thechconstep is at best a no-op and at worst breaks the build, and that shimmingchconto exit 0 inside the image is a known workaround.Workaround we're using
Placed in both controller and agent images. Verified to fix the failure across all affected clusters.
Are you interested in contributing a fix?
No response