Error: OCI runtime error: crun: Failed to re-execute libcrun via memory file descriptor

[hpcpony]

Complete novice trying my first experiments with podman.

I also tried pulling some containers ("hello-world", "busybox") and get the same error.

This is the simplest I could come up with to illustrate the problem.

[ex1] cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.6 (Plow)

[ex1] cat Containerfile
FROM alpine:latest
CMD ["echo", "Hello from Podman container!"]

[ex1] podman build -t ex1 .
STEP 1/2: FROM alpine:latest
STEP 2/2: CMD ["echo", "Hello from Podman container!"]
--> Using cache 39cef35724e789ee1c41384b5dd9424c79a52dd1d338ec2e9db2f6819f2002c8
COMMIT ex1
--> 39cef35724e7
Successfully tagged localhost/ex1:latest
39cef35724e789ee1c41384b5dd9424c79a52dd1d338ec2e9db2f6819f2002c8

[ex1] podman run ex1
Failed to re-execute libcrun via memory file descriptor
ERRO[0000] Removing container 9aedcf6f44061c7726f744c95770ba94bb636a5eb0e88ef3c98c5a1fe83c115f from runtime after creation failed
Error: OCI runtime error: crun: Failed to re-execute libcrun via memory file descriptor

[ex1] podman --version
podman version 5.4.0

[ex1] crun --version
crun version 1.23.1
commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
rundir: /run/user/1008/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

As best as I can tell these are the newest podman and crun available from RedHat.

Looking at the source it looks like it might be that this (crun/src/libcrun/clonedbinary.c) is unhappy but
it's not clear to me which part is the problem. (not sure this is the code for my version of podman).

int ensure_cloned_binary(void)
{
	cleanup_close int execfd = -1;
	char **argv = NULL;

	/* Check that we're not self-cloned, and if we are then bail. */
	int cloned = is_self_cloned();
	if (cloned > 0 || cloned == -ENOTRECOVERABLE)
		return cloned;

	if (fetchve(&argv) < 0)
		return -EINVAL;

	execfd = clone_binary();
	if (execfd < 0)
		return -EIO;

	if (putenv(CLONED_BINARY_ENV "=1"))
		goto error;

	fexecve(execfd, argv, environ);
error:
	return -ENOEXEC;
}

[kolyshkin]

@hpcpony yes, you are right, ensure_cloned_binary failed. As to why, there can be a number of reasons, but generally this should never happen.

The best way to check is to run crun under strace and see what results in an error.

Or, you can modify crun source to print the error returned from ensure_clone_binary, which should give a hint. Something like this:

--- a/src/libcrun/linux.c
+++ b/src/libcrun/linux.c
@@ -5965,9 +5965,11 @@ int ensure_cloned_binary ();
 __attribute__ ((constructor)) static void
 libcrun_rexec (void)
 {
-  if (ensure_cloned_binary () < 0)
+  int ret;
+  ret = ensure_cloned_binary ();
+  if (ret < 0)
     {
-      fprintf (stderr, "Failed to re-execute libcrun via memory file descriptor\n");
+      fprintf (stderr, "Failed to re-execute libcrun via memory file descriptor: %d (%s)\n", -ret, strerror (-ret));
       _exit (EXIT_FAILURE);
     }
 }

If you can't do either strace or patch and recompile, please let us know more about how we can reproduce this.

[hpcpony]

My system lacks the prerequisites to build from source (and I'm not sure I can get the sysadmins to load them).

In the meantime I tried using strace (strace -f podman run -t ex1). It's a bit beyond me, but this seemed suspicious. Does it help?

.
.
.
[pid 293102] munmap(0x7feb30688000, 72891) = 0
[pid 293102] getrandom("\x2f\xad\x6a\x73\x11\x52\x82\xfd", 8, GRND_NONBLOCK) = 8
[pid 293102] brk(NULL)                  = 0x55e438802000
[pid 293102] brk(0x55e438823000)        = 0x55e438823000
[pid 293102] access("/etc/gcrypt/fips_enabled", F_OK) = -1 ENOENT (No such file or directory)
[pid 293102] openat(AT_FDCWD, "/proc/sys/crypto/fips_enabled", O_RDONLY) = 3
[pid 293102] fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid 293102] read(3, "0\n", 1024)       = 2
[pid 293102] close(3)                   = 0
[pid 293102] prctl(PR_CAPBSET_READ, CAP_MAC_OVERRIDE) = 1
[pid 293102] prctl(PR_CAPBSET_READ, 0x30 /* CAP_??? */) = -1 EINVAL (Invalid argument)
[pid 293102] prctl(PR_CAPBSET_READ, CAP_CHECKPOINT_RESTORE) = 1
[pid 293102] prctl(PR_CAPBSET_READ, 0x2c /* CAP_??? */) = -1 EINVAL (Invalid argument)
[pid 293102] prctl(PR_CAPBSET_READ, 0x2a /* CAP_??? */) = -1 EINVAL (Invalid argument)
[pid 293102] prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument)
[pid 293102] openat(AT_FDCWD, "/proc/self/exe", O_RDONLY|O_CLOEXEC) = -1 EPERM (Operation not permitted)
[pid 293102] write(2, "Failed to re-execute libcrun via"..., 56) = 56
[pid 293102] exit_group(1)              = ?
[pid 293102] +++ exited with 1 +++
[pid 293101] <... poll resumed>)        = 1 ([{fd=16, revents=POLLIN}])
[pid 293101] write(5, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 293101] read(16, "\21\0\0\0\0\0\0\0\1\0\0\0\356x\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 128) = 128
.
.
.

[giuseppe]

in what environment are you running?

If we can't access proc, we can't really setup the container anyway

[hpcpony]

RedHat 9.6. I don't think there are any special tweaks to the kernel but there's probably some security/account stuff setup which my gut says is the issue. I've done some more poking around and:

1. SELinux isn't the problem
2. our /tmp is noexec, but I've tried with TMPDIR and ~/.config/containers/containers.conf setting it to somewhere else.
3. things seem to work fine with runc

Maybe since runc works, once I figure out how to do some podman stuff I can build a container with the dependencies to build an instrumented version of crun and see why it's having problems.

You can close this if you're out of ideas. If I figure anything out I'll follow up.

[hpcpony]

I was able to create a container (using runc) in which I could build crun from source and see what what is happening. I grabbed 1.23.1 source (the same version as what comes with RedHat 9). I added the following to the beginning of "is_self_cloned" in cloned_binary.c.

.
.
.
  char filename[] = "/proc/self/exe";
  fprintf(stderr, ">> is_self_cloned\n");
  if (stat(filename, &statbuf) == 0) {
    fprintf(stderr, "stat %s uid/gid: %d/%d mode: %o uid: %d pid: %d\n", filename, statbuf.st_uid, statbuf.st_gid, statbuf.st_mode, getuid(), getpid());
  }
  if (lstat(filename, &statbuf) == 0) {
    fprintf(stderr, "lstat %s uid/gid: %d/%d mode: %o uid: %d pid: %d\n", filename, statbuf.st_uid, statbuf.st_gid, statbuf.st_mode, getuid(), getpid());
  }
  char buf[1024] ={0};
  if (readlink(filename, buf, sizeof(buf)) == -1) {
    perror("readlink:");
    exit(1);
  }
  fprintf(stderr, "%s links to %s\n", filename, buf);
  for (int i=0; i<50; i++) fprintf(stderr, "%02X", buf[i]); fprintf(stderr,"\n");

  fd = open(filename, O_RDONLY|O_CLOEXEC);
  fprintf(stderr, "open: %d errno: %d\n", fd, errno);
  if (fd < 0) {
    perror("open:");
    return -ENOTRECOVERABLE;
  }
.
.
.

Then trying to start a container (simple hello world) using the newly built version

[work] podman run --runtime /scr/work/crun ex1
crun call: delete --force ab0d806f80f5c8defdac3f0067a20a603a991faeb31dc474518097c7f6e4cdbb
>> is_self_cloned
stat /proc/self/exe uid/gid: 0/0 mode: 100755 uid: 0 pid: 817673
lstat /proc/self/exe uid/gid: 0/0 mode: 120777 uid: 0 pid: 817673
/proc/self/exe links to /scr/work/crun-1.23.1/build/crun
2F7363722F636672656573652F776F726B2F6372756E2D312E32332E312F6275696C642F6372756E00000000000000000000
open: 3 errno: 1
>> is_self_cloned
stat /proc/self/exe uid/gid: 0/0 mode: 100755 uid: 0 pid: 817673
lstat /proc/self/exe uid/gid: 0/0 mode: 120777 uid: 0 pid: 817673
/proc/self/exe links to /
2F00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
open: -1 errno: 1
open:: Operation not permitted
ensure_cloned_binary= -131
Failed to re-execute libcrun via memory file descriptor
ERRO[0000] Removing container ab0d806f80f5c8defdac3f0067a20a603a991faeb31dc474518097c7f6e4cdbb from runtime after creation failed 
Error: /scr/work/crun: open:: Operation not permitted: OCI permission denied

Couple things that are really weird (assuming I didn't do something stupid):

• the uid/gid for both /proc/self/exe and the thing it links to is 0/0-
• the mode for /proc/self/exe is actual that of /scr/work/crun-1.23.1/build/crun. I can change the permissions on the crun and that's what will show up in the output.
• /proc/self/exe seems to link to '/'

Just for yucks I ran the modified crun just by itself and it seems more reasonable.

[work] /scr/work/crun --version
crun call: --version
>> is_self_cloned
stat /proc/self/exe uid/gid: 1008/1020 mode: 100755 uid: 1008 pid: 817695
lstat /proc/self/exe uid/gid: 1008/1020 mode: 120777 uid: 1008 pid: 817695
/proc/self/exe links to /scr/work/crun-1.23.1/build/crun
2F7363722F636672656573652F776F726B2F6372756E2D312E32332E312F6275696C642F6372756E00000000000000000000
open: 3 errno: 1
>> is_self_cloned
stat /proc/self/exe uid/gid: 1008/1020 mode: 100777 uid: 1008 pid: 817695
lstat /proc/self/exe uid/gid: 1008/1020 mode: 120777 uid: 1008 pid: 817695
/proc/self/exe links to /memfd:crun_cloned:/proc/self/exe (deleted)
2F6D656D66643A6372756E5F636C6F6E65643A2F70726F632F73656C662F657865202864656C657465642900000000000000
open: 3 errno: 1
crun version 1.23.1
commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
rundir: /run/user/1008/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

At this point I'm thinking the problem starts somewhere in podman. I'm not sure where podman makes entry into the crun code. I looked at what it takes to build podman from source I don't see me being able to tackle that anytime soon.

I tried crun-1.24-linux-amd64 and it has the same issues.

[jnovy]

The key finding is that after fexecve() from the memfd, /proc/self/exe resolves to / instead of the expected /memfd:crun_cloned:... path. This only happens when podman invokes crun (not standalone), which suggests something in the container setup (likely a pivot_root or mount namespace change) causes the kernel to lose track of the executable's dentry.

Could you check:

1. mount | grep proc -- specifically whether /proc is mounted with hidepid= or any unusual options
2. readlink /proc/self/exe from within the same context where crun is invoked by podman
3. Whether installing crun on a read-only filesystem works around the issue (the clone mechanism is skipped when the binary is on a read-only mount)