VolumePopulator scheduling on the wrong node?

[remlabm]

Hello,

I'm having an issue that only recently started after upgrading to v0.14.0.

I have three nodes with the same OpenEBS-ZFS StorageClass. I have WaitForFirstConsumer set, but when scheduling the pod, the VolumePopulator will create the PVC/PVs on the wrong node. Even when forcing the node using nodeSelector, the VolumePopulator will schedule on a different node.

Similar to this issue:
#1255

Any ideas on why or where to start debugging this issue? I haven't tried to downgrade back to v0.13.1 yet.

[tesshuflower]

@remlabm unfortunately these can be a bit hard to debug. I can't think of anything specifically in v0.14.0 that would suddenly cause this to happen.

I guess from your description I'm assuming this stems from the fact that your volumesnapshots are not portable.

I think this can be an issue even if you created your PVC from snapshot (without volumepopulator being involved). With waitForFirstConsumer, the pod (which will mount the PVC) gets assigned to a node, and then it will get populated from the volumesnapshot - which may not work depending on what node was picked.

I'm not sure if there's some mechanism in the storage driver to get around this.

Having said all the above, it sounds like you are intentionally assigning the pod that will mount the PVC to a specific node which should be able to use the volumesnapshot?

Let me list some of the steps the volsync volumepopulator controller does, in case this helps (the PVC here is your volumesnapshot PVC):

• When the storage driver for the PVC is using waitForFirstConsumer, then the volume snapshot controller will not do anything with the PVC until it gets assigned to a node. That should happen after your pod gets created.
• Essentially after the pod gets created and assigned to a node (this is not done by the volume populator but normally the kube-scheduler), the PVC should get the annotation: volume.kubernetes.io/selected-node to indicate the node that has been selected.
• until the annotation is set on the PVC, the volume populator will not do anything further, in the VolSync log you may see entries something like this: "VolumeBindingMode is WaitForFirstConsumer, need to wait for nodeName annotation"
• only after the PVC gets this annotation will the volsync volume populator controller proceed and try to populate a PVC (on that assigned node) from the volumesnapshot.

So - wondering if perhaps your pod is actually not getting scheduled correctly? Can you recreate this same thing with the pod on its own (not using this particular PVC)? That might be good to see as a comparison.

[erkiaas]

Same here - OpenEBS zfs-localpv - same storage class for all nodes (matching pool name across those)

[tesshuflower]

Same here - OpenEBS zfs-localpv - same storage class for all nodes (matching pool name across those)

@erkiaas Are you also trying to schedule your app pod that mounts the volume populator PVC to a specific node? Some details here may help debug (along with the other questions I've asked above).

[remlabm]

@tesshuflower - It doesn't matter if I set the nodeSelector or let the pod affinity scheduler handle the pods, there is always a conflict with the volsync-${APP}-src selector.

If I set the node to worker1, the PV/PVC will provision properly on worker1, but the recovery volume will attach to worker2. If I disable scheduling on worker2, the recovery volume will then provision properly on worker1.

This also will happen with the backup and cache volumes.

[tesshuflower]

@remlabm I'm not sure I understand exactly what you mean above - maybe you can answer the following:

• The original problem appeared to be that your volume populator PVC (this is your "recovery volume" I assume) was not getting scheduled on the node you want - since you use waitForFirstConsumer, this volume should get scheduled when you try to use it. Is the first usage of this volume by your app? Or by VolSync itself (i.e. with the ReplicationSource).
• Are you setting spec.restic.moverAffinity on your replicationSource? And is this being respected?
• Are you adding any custom moverVolumes in your replicationsource ?