Skip to content

multi-region-clouds.md

Latest commit

 

History

History
357 lines (325 loc) · 10.7 KB

multi-region-clouds.md

File metadata and controls

357 lines (325 loc) · 10.7 KB

Multi Az/Region/Openstack Configuration

Multi cluster Configuration file

Create a configuration file with a subsection per openstack cluster to manage (pay attention to enable ignore-volume-az in BlockStorage section).

Example of configuration with 3 regions (The default is backward compatible with mono cluster configuration but not mandatory).

apiVersion: v1
kind: Secret
metadata:
  name: cloud-config
  namespace: kube-system
type: Opaque
stringData:
  cloud.conf: |-
    [BlockStorage]
    bs-version=v3
    ignore-volume-az=True

    [Global]
    auth-url="https://auth.cloud.openstackcluster.region-default.local/v3"
    username="region-default-username"
    password="region-default-password"
    region="default"
    tenant-id="region-default-tenant-id"
    tenant-name="region-default-tenant-name"
    domain-name="Default"

    [Global "region-one"]
    auth-url="https://auth.cloud.openstackcluster.region-one.local/v3"
    username="region-one-username"
    password="region-one-password"
    region="one"
    tenant-id="region-one-tenant-id"
    tenant-name="region-one-tenant-name"
    domain-name="Default"

    [Global "region-two"]
    auth-url="https://auth.cloud.openstackcluster.region-two.local/v3"
    username="region-two-username"
    password="region-two-password"
    region="two"
    tenant-id="region-two-tenant-id"
    tenant-name="region-two-tenant-name"
    domain-name="Default"

Create region/cloud secrets

Create a secret per openstack cluster which contains a key cloud and as value the subsection's name of corresponding openstack cluster in configuration file.

These secrets are referenced in storageClass definitions to identify openstack cluster associated to the storageClass.

apiVersion: v1
kind: Secret
metadata:
  name: openstack-config-region-one
  namespace: kube-system
type: Opaque
stringData:
  cloud: region-one
---
apiVersion: v1
kind: Secret
metadata:
  name: openstack-config-region-two
  namespace: kube-system
type: Opaque
stringData:
  cloud: region-two

Create storage Class for dedicated cluster

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: sc-region-one
allowVolumeExpansion: true
allowedTopologies:
- matchLabelExpressions:
  - key: topology.cinder.csi.openstack.org/zone
    values:
    - nova
  - key: topology.kubernetes.io/region
    values:
    - region-one
parameters:
  csi.storage.k8s.io/controller-publish-secret-name: openstack-config-region-one
  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
  csi.storage.k8s.io/node-publish-secret-name: openstack-config-region-one
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system
  csi.storage.k8s.io/node-stage-secret-name: openstack-config-region-one
  csi.storage.k8s.io/node-stage-secret-namespace: kube-system
  csi.storage.k8s.io/provisioner-secret-name: openstack-config-region-one
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  csi.storage.k8s.io/controller-expand-secret-name: openstack-config-region-one
  csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
provisioner: cinder.csi.openstack.org
reclaimPolicy: Delete
volumeBindingMode: Immediate
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: sc-region-two
allowVolumeExpansion: true
allowedTopologies:
- matchLabelExpressions:
  - key: topology.cinder.csi.openstack.org/zone
    values:
    - nova
  - key: topology.kubernetes.io/region
    values:
    - region-two
parameters:
  csi.storage.k8s.io/controller-publish-secret-name: openstack-config-region-two
  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
  csi.storage.k8s.io/node-publish-secret-name: openstack-config-region-two
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system
  csi.storage.k8s.io/node-stage-secret-name: openstack-config-region-two
  csi.storage.k8s.io/node-stage-secret-namespace: kube-system
  csi.storage.k8s.io/provisioner-secret-name: openstack-config-region-two
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  csi.storage.k8s.io/controller-expand-secret-name: openstack-config-region-two
  csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
provisioner: cinder.csi.openstack.org
reclaimPolicy: Delete
volumeBindingMode: Immediate

Create a csi-cinder-nodeplugin daemonset per cluster openstack

Daemonsets should deploy pods on nodes from proper openstack context. We suppose that the node have a label topology.kubernetes.io/region with the openstack cluster name as value (you could manage this with kubespray, manually, whatever, it should be great to implement this in openstack cloud controller manager).

Do as follows:

  • Use nodeSelector to match proper nodes labels
  • Add cli argument --additional-topology topology.kubernetes.io/region=region-one, which should match node labels, to container cinder-csi-plugin
  • Add cli argument --cloud-name="region-one", which should match configuration file subsection name, to container cinder-csi-plugin.
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: csi-cinder-nodeplugin-region-one
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-cinder-nodeplugin-region-one
  template:
    metadata:
      labels:
        app: csi-cinder-nodeplugin-region-one
    spec:
      containers:
      - name: node-driver-registrar
        ...
      - name: liveness-probe
        ...
      - name: cinder-csi-plugin
        image: registry.k8s.io/provider-os/cinder-csi-plugin:v1.36.0
        args:
        - /bin/cinder-csi-plugin
        - --endpoint=$(CSI_ENDPOINT)
        - --cloud-config=$(CLOUD_CONFIG)
        - --cloud-name="region-one"
        - --additional-topology
        - topology.kubernetes.io/region=region-one
        env:
        - name: CSI_ENDPOINT
          value: unix://csi/csi.sock
        - name: CLOUD_CONFIG
          value: /etc/config/cloud.conf
        ...
        volumeMounts:
        ...
        - mountPath: /etc/config
          name: secret-cinderplugin
          readOnly: true
        ...
      nodeSelector:
        topology.kubernetes.io/region: region-one
      volumes:
      ...
      - name: secret-cinderplugin
        secret:
          defaultMode: 420
          secretName: cloud-config
      ...
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: csi-cinder-nodeplugin-region-two
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-cinder-nodeplugin-region-two
  template:
    metadata:
      labels:
        app: csi-cinder-nodeplugin-region-two
    spec:
      containers:
      - name: node-driver-registrar
        ...
      - name: liveness-probe
        ...
      - name: cinder-csi-plugin
        image: registry.k8s.io/provider-os/cinder-csi-plugin:v1.36.0
        args:
        - /bin/cinder-csi-plugin
        - --endpoint=$(CSI_ENDPOINT)
        - --cloud-config=$(CLOUD_CONFIG)
        - --cloud-name="region-two"
        - --additional-topology
        - topology.kubernetes.io/region=region-two
        env:
        - name: CSI_ENDPOINT
          value: unix://csi/csi.sock
        - name: CLOUD_CONFIG
          value: /etc/config/cloud.conf
        ...
        volumeMounts:
        ...
        - mountPath: /etc/config
          name: secret-cinderplugin
          readOnly: true
        ...
      nodeSelector:
        topology.kubernetes.io/region: region-two
      volumes:
      ...
      - name: secret-cinderplugin
        secret:
          defaultMode: 420
          secretName: cloud-config
      ...

Configure csi-cinder-controllerplugin deployment

Enable Topology feature-gate on container csi-provisioner of csi-cinder-controllerplugin deployment by adding cli argument ``--feature-gates="Topology=true"

Add cli argument --cloud-name="region-one" for each managed openstack cluster, name should match configuration file subsection name, to container cinder-csi-plugin.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  name: csi-cinder-controllerplugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-cinder-controllerplugin
  template:
    metadata:
      labels:
        app: csi-cinder-controllerplugin
    spec:
      containers:
      - name: csi-provisioner
        image: registry.k8s.io/sig-storage/csi-provisioner:v3.0.0
        args:
        - --csi-address=$(ADDRESS)
        - --timeout=3m
        - --default-fstype=ext4
        - --extra-create-metadata
        - --feature-gates
        - Topology=true
        ...
      - name: cinder-csi-plugin
        image: registry.k8s.io/provider-os/cinder-csi-plugin:v1.36.0
        args:
        - /bin/cinder-csi-plugin
        - --endpoint=$(CSI_ENDPOINT)
        - --cloud-config=$(CLOUD_CONFIG)
        - --cluster=$(CLUSTER_NAME)
        - --cloud-name="region-one"
        - --cloud-name="region-two"
        env:
        - name: CSI_ENDPOINT
          value: unix://csi/csi.sock
        - name: CLOUD_CONFIG
          value: /etc/config/cloud.conf
        - name: CLUSTER_NAME
          value: kubernetes
        volumeMounts:
        - mountPath: /etc/config
          name: secret-cinderplugin
          readOnly: true
        ...
      - name: csi-attacher
        ...
      - name: csi-snapshotter
        ...
      - name: csi-resizer
        ...
      - name: liveness-probe
        ...
      volumes:
      - name: secret-cinderplugin
        secret:
          defaultMode: 420
          secretName: cloud-config
      ...

When Using the cinder-csi-plugin Helm Chart

When running the cinder-csi-plugin in a multi-region setup, you need to specify different extraArgs for the cinder-csi-plugin containers in both the Deployment and the DaemonSet.

When using the Helm chart, set the different extraArgs using plugin.nodePlugin.extraArgs and plugin.controllerPlugin.extraArgs.

If you set the extraArgs in plugin.extraArgs, the same arguments will be applied to both the Deployment and the DaemonSet cinder-csi-plugin containers.

You will still need to manually create additional DaemonSets for your extra regions.

nodePlugin:
  extraArgs: |-
    - --cloud-name=region-one
    - --additional-topology
    - topology.kubernetes.io/region=region-one
controllerPlugin:
  extraArgs: |-
    - --cloud-name=region-one
    - --cloud-name=region-two

In addition, if you use the resizer and the snapshotter, you will need them to be able to read the secrets you defined in the storage class' annotations in order to determine which cloud to address. You will need to add some extraRbac in YAML format, like this:

snapshotter:
  extraRbac:
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["get", "list"]
resizer:
  extraRbac:
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["get", "list", "watch"]