Documentation doesn't mention who/how to create secrets and storageclasses

[remram44]

Describe the bug

I am looking to migrate from ceph-csi to the operator as recommended by the ceph-csi release notes. However it is puzzling that the ceph-csi README doesn't mention it at all, and that the ceph-csi-operator docs don't mention any of the other steps necessary to get a complete setup.

I am left wondering what needs to be picked from each documentation to reach a working configuration without losing data.

Please make sure your documentation lists all necessary steps. Stopping at the point where the operator is installed but no storage is available is not appropriate and definitely not a way to "quick start".

Steps to reproduce

Steps to reproduce the behavior:

1. Try to install operator
2. Find operator docs, follow steps
3. Give up when steps end abruptly after setting up a "ClientProfile" with no idea how to set up CSIDrivers or StorageClasses and link them with the cluster

Actual results

My planned maintenance window ended and I am staying with ceph-csi because the operator is obviously not ready for prime time.

Expected behavior

The "recommended" installation method is recommended consistently and sufficiently documented to be used. It is fine for some of the docs to live in the ceph-csi repo, but then you need to reference them.

[Starttoaster]

In particular, the Helm usage documentation is basically non-existent 😄 It's like, "follow the guide to install with Helm!" Click on the link to the Helm documentation "here's the Helm chart repo URL, and a hundred options with no descriptions, and no minimal example setup values." And then that's the end of the documentation. I'm just going to re-install ceph-csi for now, but waiting for more documentation.

[Madhu-1]

@Starttoaster #431 should take care of adding more documentation for helm values

[Starttoaster]

@Madhu-1 Thanks, this helps a little bit, but not a lot. There's no explanation of where to add or reference a Secret containing your ceph userID and userKey. There's no explanation of what field(s) in the driver helm chart need to be set to your clusterID from the old ceph-csi. Do you have to apply your own StorageClass or does the operator somehow do that for you? If you have to apply one separately, why isn't it part of the helm chart? I'm sure the charts make a lot of sense to people who have either wrote the operator or have already deciphered all the various settings the hard way, but some example installation values files would go a long way to helping people understand the settings.

In addition to that, some of the documentation for these settings don't make sense. cephConnections[0].monitors has a description of Ceph monitors (key-value pairs, typically IP addresses of the Ceph monitors) (default: {}). The monitors field in ceph-csi used to be a list of addresses (ip:port) to your Ceph monitors, and I'm pretty sure it still is, and not a key-value pair.

[Starttoaster]

I managed to install ceph-csi-operator using the helm chart, and the drivers via kubectl now (which wasn't an easy task with the current documentation either, but it was better than the ceph-csi-driver Helm chart's documentation.) But now I found a situation which the current migration.md documentation didn't account for: all existing PersistentVolume resources in the cluster reference the old ceph user Secret

spec:
  capacity:
    storage: 1Gi
  csi:
    driver: rbd.csi.ceph.com
    fsType: ext4
    volumeAttributes:
       ...
    nodeStageSecretRef:
      name: csi-rbd-secret
      namespace: ceph-csi
    controllerExpandSecretRef:
      name: csi-rbd-secret
      namespace: ceph-csi

This is taken from one of my PersistentVolume resources, and shows the nodeStageSecretRef and controllerExpandSecretRef referencing the previous ceph-csi Secret. This appears to be causing my PersistentVolume to be unmountable from the following error:

MountVolume.MountDevice failed for volume "pvc-23388706-8854-473a-a6f9-d92b5b8ec7fd" : fetching NodeStageSecretRef ceph-csi/csi-rbd-secret failed: kubernetes.io/csi: failed to find the secret csi-rbd-secret in the namespace ceph-csi with error: secrets "csi-rbd-secret" not found. (12m)

That is an unpatchable field on a PersistentVolume... so is migration from ceph-csi even possible? Has it been attempted before? If so, why is none of this documented?

[Starttoaster]

Okay, I got a working setup, through some trial and tribulation. This should really have been better documented before it was communicated that the original ceph-csi would stop being supported.

Steps

Uninstall Ceph-CSI

I installed ceph-csi with Helm so for me it was just deleting the ceph-csi helm release.

Install ceph-csi-operator via the Helm chart

I just used the default values. helm upgrade --install --namespace ceph-csi-operator-system --repo "https://ceph.github.io/ceph-csi-operator/" ceph-csi-operator ceph-csi-operator

Install all the driver resources

I just use an rbd driver. Install these into the same namespace as the operator (ceph-csi-operator-system).

Replace the variables below with the same authentication information you had for the old ceph-csi:

apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
type: Opaque
stringData:
  userID: "$CEPH_USER_ID"
  userKey: "$CEPH_USER_KEY"

Replace the ceph monitor IPs below with yours. And change the crushLocationLabels if you need.

kind: CephConnection
apiVersion: csi.ceph.io/v1
metadata:
  name: ceph-connection
spec:
  monitors:
    - "$MONITOR1_IP:6789"
    - "$MONITOR2_IP:6789"
    - "$MONITOR3_IP:6789"
  readAffinity:
    crushLocationLabels:
      - kubernetes.io/hostname
      - topology.kubernetes.io/region
      - topology.kubernetes.io/zone

Set the name of the ClientProfile resource to your old clusterId field from ceph-csi. Make sure the cephConnectionRef points to the name of the CephConnection resource we just made, and the controllerPublishSecret points to the authentication Secret we made before that.

apiVersion: csi.ceph.io/v1
kind: ClientProfile
metadata:
  name: $CLUSTER_ID
spec:
  cephConnectionRef:
    name: ceph-connection
  rbd:
    cephCsiSecrets:
      controllerPublishSecret:
        name: csi-rbd-secret

I have a custom kubelet path so I needed to specify it in the driver's nodePlugin config.

apiVersion: csi.ceph.io/v1
kind: Driver
metadata:
  name: rbd.csi.ceph.com
spec:
  nodePlugin:
    kubeletDirPath: /var/lib/k0s/kubelet

And then finally the StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd
  annotations:
    storageclass.kubernetes.io/is-default-class: 'true'
provisioner: rbd.csi.ceph.com
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate
parameters:
  # Should match the ClientProfile name from before, this is your clusterId from your old ceph-csi config
  clusterID: $CLUSTER_ID

  # The name of your ceph pool that will store your RBD images
  pool: $POOL_NAME

  # storageclass configuration details
  imageFeatures: layering
  csi.storage.k8s.io/fstype: ext4

  # I don't understand why we need to set these here AND in the ClientProfile, it's super redundant, but it seems to be required.
  # But this just points back to the authentication Secret we made above.
  csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-operator-system
  csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-operator-system
  csi.storage.k8s.io/controller-publish-secret-name: csi-rbd-secret
  csi.storage.k8s.io/controller-publish-secret-namespace: ceph-csi-operator-system
  csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-operator-system

Hacky way to make old volumes work

Now this is where it gets tricky. You probably didn't install the old ceph-csi to the ceph-csi-operator-system namespace before, right? Well all your old PersistentVolumes have references to your old ceph authentication Secret in the namespace(s) that you previously installed ceph-csi to, and these fields on the PersistentVolume are immutable. So you will need to ensure you still have your old ceph-csi namespace(s), and just install the same authentication Secret to each one. Otherwise, what happens is your existing volumes that you're bringing to the operator from ceph-csi will fail to mount with the error MountVolume.MountDevice failed for volume "pvc-23388706-8854-473a-a6f9-d92b5b8ec7fd" : fetching NodeStageSecretRef ceph-csi/csi-rbd-secret failed: kubernetes.io/csi: failed to find the secret csi-rbd-secret in the namespace ceph-csi with error: secrets "csi-rbd-secret" not found. (12m)

Notes

As best as I can tell, you only need to install the Driver resources you will actually use for StorageClasses. For each StorageClass, it will require its own ClientProfile. And each ClientProfile can share one CephConnection resource, and share or use different Ceph authentication Secrets.
Though I have no clue how you're supposed to set up ClientProfiles for multiple different drivers, because they probably share the same ClusterId, and would therefore have the same resource name. Maybe one ClientProfile could encompass both driver types? Haven't tried it, so it's impossible for me to say.

[Madhu-1]

@Starttoaster will reply to all the questions after some time.

https://github.com/ceph/ceph-csi-operator/blob/main/docs/migration.md covers the migration for users from cephcsi.
cephcsi operator doesn't do anything specific to storageclasses and secrets, it will remain the same as before
#367 open issue to create the classes and their automation

The above PR is just for existing values in the operator and csi drivers. Please feel free to open issues/PR's for any improvements on this repo. Thank you

[Starttoaster]

@Madhu-1 The biggest problem was that the migration steps don't actually show you how to migrate from the old ceph-csi. At the end of those migration steps, the operator would still fail to mount volumes made by the old ceph-csi installation. See Hacky way to make old volumes work from my previous comment.

The second problem was just that it was not at all clear how to configure each of the driver-related custom resources to end up with a usable storage class with the new operator. My previous comment explained a working way to do that for an RBD-backed storage class. Hopefully that helps someone in the future who was similarly confused by the lack of documentation on how the driver custom resources should be configured when migrating from the old ceph-csi.