New clean install using provided manifests fails to start due to failure to open /etc/ceph/keyring: read-only file system

[eriklogtenberg]

Describe the bug

I installed ceph-csi-rbd according to these instructions: https://github.com/ceph/ceph-csi/blob/devel/docs/rbd/deploy.md

The csi-rbdplugin and csi-rbdplugin-provisioner fail to come online due to this error:
I0307 04:14:48.391550 2616501 cephcsi.go:204] Driver version: v3.15.0 and Git version: 72cdc74
F0307 04:14:48.391716 2616501 cephcsi.go:242] failed to write ceph configuration file (open /etc/ceph/keyring: read-only file system)

I checked the /etc/ceph mount and it is indeed read-only, since the ceph-config ConfigMap is mounted there.
I checked around lines 245 in cephcsi.go and it does seem to check (unconditionally) in func main() if it can write there. I think an alternative location could be specified with --config-dir, but the manifests don't seem to do that, so this is what it defaults to.

I'm kind of stuck here, am I overlooking anything?

I did verify these two requirements and they are both met: "Your Kubernetes cluster must allow privileged pods (i.e. --allow-privileged flag must be set to true for both the API server and the kubelet). Moreover, as stated in the mount propagation docs, the Docker daemon of the cluster nodes must allow shared mounts."

Environment details

• Image/version of Ceph CSI driver : v3.15.0
• Kubernetes cluster version : v1.35.1
• Ceph cluster version : 19.2.3 (not that it gets to connect to it)

Steps to reproduce

Steps to reproduce the behavior:

Basically follow the installation instructions. As soon as we deploy the csi-rbdplugin, this error appears.

[Madhu-1]

@eriklogtenberg this is inside a container and we are mouting a configmap to the path. it should work.

/etc/ceph is not readyonly inside the container may be you have some extra configuration that is making it as readyonly. We are testing with 1.35 in our E2E testing, we dont have any problem

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[carlosaml500]

The cephcsi image seems to only use ceph-csi-config as it has all te info, and only writes the keyring to /etc/ceph.
/etc/ceph has a config map mounted, thus it is read-only. and the image returns error.
As a workaround I just mounted it as an empty dir, and it worked fine.
must do it on both on csi-rbdplugin.yaml and csi-rbdplugin-provisioner.yaml

spec:
...
template:
...
spec:
...
volumes:
...
- name: ceph-config
configMap:
name: ceph-config
*replace the 3 lines above with:
- name: ceph-config
emptyDir: {}