Skip to content

Multiple etcd-managers in the same AWS AZ can claim each other's volumes #14560

Closed as not planned
#18294
Closed as not planned
Multiple etcd-managers in the same AWS AZ can claim each other's volumes#14560
#18294
Labels
kind/bugCategorizes issue or PR as related to a bug.lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.
@diversario

Description

@diversario
diversario
opened on Nov 15, 2022

/kind bug

1. What kops version are you running? The command kops version, will display
this information.

Version 1.20.3 (git-6995b12559257e37a05aee54939222c97d0f3c8f)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

1.20.15

3. What cloud provider are you using?

AWS EC2

4. What commands did you run? What is the simplest way to reproduce this issue?

Create a cluster with two or more master nodes within the same AWS AZ, with etcdClusters configured similar to

  etcdClusters:
  - enableEtcdTLS: true
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a-1
      name: etcd-us-east-1a-1
      volumeSize: 100
    - encryptedVolume: true
      instanceGroup: master-us-east-1a-4
      name: etcd-us-east-1a-4
      volumeSize: 100
    - encryptedVolume: true
      instanceGroup: master-us-east-1b-5
      name: etcd-us-east-1b-5
      volumeSize: 100
    name: main

5. What happened after the commands executed?

Occasionally, after the cluster is created, master node master-us-east-1a-1 will attach etcd volumes belonging to master-us-east-1a-4 and vise versa.

6. What did you expect to happen?

Master nodes attach etcd volumes belonging to themselves only.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

Details 
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  name: k8s.staging.ue1.company.io
spec:
  api:
    loadBalancer:
      class: Network
      crossZoneLoadBalancing: true
      sslCertificate: arn:aws:acm:us-east-1:475133402591:certificate/341a3f43-46f2-4616-a1e2-95b75a12fa66
      sslPolicy: ELBSecurityPolicy-2016-08
      type: Internal
  authorization:
    rbac: {}
  cloudLabels:
    environment: staging
    k8s.io/cluster-autoscaler/enabled: "1"
    k8s.io/cluster-autoscaler/k8s.staging.ue1.company.io: "1"
  cloudProvider: aws
  configBase: s3://kops-company-io/k8s.staging.ue1.company.io
  dnsZone: Z045098225JE6A9UNA5TR
  docker:
    version: null
  etcdClusters:
  - enableEtcdTLS: true
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a-1
      name: etcd-us-east-1a-1
      volumeSize: 100
    - encryptedVolume: true
      instanceGroup: master-us-east-1b-2
      name: etcd-us-east-1b-2
      volumeSize: 100
    - encryptedVolume: true
      instanceGroup: master-us-east-1c-3
      name: etcd-us-east-1c-3
      volumeSize: 100
    - encryptedVolume: true
      instanceGroup: master-us-east-1a-4
      name: etcd-us-east-1a-4
      volumeSize: 100
    - encryptedVolume: true
      instanceGroup: master-us-east-1b-5
      name: etcd-us-east-1b-5
      volumeSize: 100
    manager:
      env:
      - name: ETCD_QUOTA_BACKEND_BYTES
        value: "5368709120"
      - name: ETCD_LISTEN_METRICS_URLS
        value: http://0.0.0.0:8081
      - name: ETCD_METRICS
        value: extensive
    name: main
    version: null
  - enableEtcdTLS: true
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a-1
      name: etcd-us-east-1a-1
      volumeSize: 100
    - encryptedVolume: true
      instanceGroup: master-us-east-1b-2
      name: etcd-us-east-1b-2
      volumeSize: 100
    - encryptedVolume: true
      instanceGroup: master-us-east-1c-3
      name: etcd-us-east-1c-3
      volumeSize: 100
    - encryptedVolume: true
      instanceGroup: master-us-east-1a-4
      name: etcd-us-east-1a-4
      volumeSize: 100
    - encryptedVolume: true
      instanceGroup: master-us-east-1b-5
      name: etcd-us-east-1b-5
      volumeSize: 100
    manager:
      env:
      - name: ETCD_QUOTA_BACKEND_BYTES
        value: "5368709120"
      - name: ETCD_LISTEN_METRICS_URLS
        value: http://0.0.0.0:8082
      - name: ETCD_METRICS
        value: extensive
    name: events
    version: null
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeAPIServer:
    apiAudiences:
    - api
    - istio-ca
    auditLogFormat: json
    auditLogPath: '-'
    disableBasicAuth: false
    enableBootstrapTokenAuth: false
    featureGates:
      EphemeralContainers: "true"
      StartupProbe: "true"
      TTLAfterFinished: "true"
    maxMutatingRequestsInflight: 200
  kubeControllerManager:
    featureGates:
      EphemeralContainers: "true"
      StartupProbe: "true"
      TTLAfterFinished: "true"
  kubeDNS:
    memoryLimit: 1Gi
    memoryRequest: 128Mi
    provider: CoreDNS
  kubeProxy:
    featureGates:
      EphemeralContainers: "true"
      StartupProbe: "true"
      TTLAfterFinished: "true"
    metricsBindAddress: 0.0.0.0
  kubeScheduler:
    featureGates:
      EphemeralContainers: "true"
      StartupProbe: "true"
      TTLAfterFinished: "true"
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
    evictionHard: memory.available<500Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
    featureGates:
      EphemeralContainers: "true"
      StartupProbe: "true"
      TTLAfterFinished: "true"
    imageGCHighThresholdPercent: 75
    imageGCLowThresholdPercent: 70
    kubeReserved:
      cpu: "1"
      ephemeral-storage: 1Gi
      memory: 2Gi
    maxPods: 110
    systemReserved:
      cpu: 500m
      ephemeral-storage: 1Gi
      memory: 1Gi
  kubernetesApiAccess:
  - 10.0.0.0/16
  - 10.100.0.0/16
  - 10.102.0.0/16
  - 10.130.0.0/16
  - 10.131.0.0/16
  - 10.132.0.0/16
  - 10.206.0.0/16
  - 10.3.0.0/24
  - 10.4.0.0/24
  - 10.7.0.0/24
  kubernetesVersion: 1.20.15
  masterPublicName: api.k8s.staging.ue1.company.io
  networkCIDR: 172.20.0.0/16
  networkID: vpc-123
  networking:
    cni: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  rollingUpdate:
    maxSurge: 14%
    maxUnavailable: 0
  sshAccess:
  - 10.100.0.0/16
  - 10.130.0.0/16
  - 10.131.0.0/16
  - 10.206.0.0/16
  - 10.3.0.0/24
  - 10.4.0.0/24
  - 10.7.0.0/24
  subnets:
  - cidr: 172.20.8.0/21
    id: subnet-0a5f08e5d52b90c6f
    name: staging.ue1-private-us-east-1b
    type: Private
    zone: us-east-1b
  - cidr: 172.20.0.0/21
    id: subnet-05acfeade5d6e5120
    name: staging.ue1-private-us-east-1a
    type: Private
    zone: us-east-1a
  - cidr: 172.20.16.0/21
    id: subnet-0d0cf97dac1a37cc8
    name: staging.ue1-private-us-east-1c
    type: Private
    zone: us-east-1c
  - cidr: 172.20.101.0/24
    id: subnet-0250e429197016158
    name: staging.ue1-public-us-east-1b
    type: Utility
    zone: us-east-1b
  - cidr: 172.20.102.0/24
    id: subnet-0fdb80c6be19dd404
    name: staging.ue1-public-us-east-1c
    type: Utility
    zone: us-east-1c
  - cidr: 172.20.100.0/24
    id: subnet-0f8f0faf0a57f8d66
    name: staging.ue1-public-us-east-1a
    type: Utility
    zone: us-east-1a
  sysctlParameters: []
  topology:
    dns:
      type: Private
    masters: private
    nodes: private
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
  name: master-us-east-1a-1
spec:
  additionalSecurityGroups:
  - sg-0123380c16589c89e
  associatePublicIp: false
  cloudLabels:
    environment: staging

  image: ami-123
  machineType: m5.4xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
    kops.k8s.io/instancegroup: master-us-east-1a-1
  role: Master
  rootVolumeEncryption: true
  subnets:
  - staging.ue1-private-us-east-1a
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
  name: master-us-east-1b-2
spec:
  additionalSecurityGroups:
  - sg-0123380c16589c89e
  associatePublicIp: false
  cloudLabels:
    environment: staging
  image: ami-123
  machineType: m5.4xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
    kops.k8s.io/instancegroup: master-us-east-1b-2
  role: Master
  rootVolumeEncryption: true
  subnets:
  - staging.ue1-private-us-east-1b
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
  name: master-us-east-1c-3
spec:
  additionalSecurityGroups:
  - sg-0123380c16589c89e
  associatePublicIp: false
  cloudLabels:
    environment: staging

  image: ami-123
  machineType: m5.4xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
    kops.k8s.io/instancegroup: master-us-east-1c-3
  role: Master
  rootVolumeEncryption: true
  subnets:
  - staging.ue1-private-us-east-1c
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
  name: master-us-east-1a-4
spec:
  additionalSecurityGroups:
  - sg-0123380c16589c89e
  associatePublicIp: false
  cloudLabels:
    environment: staging
  image: ami-123
  machineType: m5.4xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
    kops.k8s.io/instancegroup: master-us-east-1a-4
  role: Master
  rootVolumeEncryption: true
  subnets:
  - staging.ue1-private-us-east-1a
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
  name: master-us-east-1b-5
spec:
  additionalSecurityGroups:
  - sg-0123380c16589c89e
  associatePublicIp: false
  cloudLabels:
    environment: staging
  image: ami-123
  machineType: m5.4xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
    kops.k8s.io/instancegroup: master-us-east-1b-5
  role: Master
  rootVolumeEncryption: true
  subnets:
  - staging.ue1-private-us-east-1b
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
  name: on-demand-workers-subnet-05acfeade5d6e5120
spec:
  additionalSecurityGroups:
  - sg-0123380c16589c89e
  associatePublicIp: false
  cloudLabels:
    environment: staging
  image: ami-123
  machineType: m5a.2xlarge
  maxSize: 1
  minSize: 0
  nodeLabels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
    kops.k8s.io/instancegroup: on-demand-workers-subnet-05acfeade5d6e5120
  role: Node
  rootVolumeEncryption: true
  rootVolumeSize: 100
  subnets:
  - staging.ue1-private-us-east-1a
  taints:
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
  name: on-demand-workers-subnet-0a5f08e5d52b90c6f
spec:
  additionalSecurityGroups:
  - sg-0123380c16589c89e
  associatePublicIp: false
  cloudLabels:
    environment: staging
  image: ami-123
  machineType: m5a.2xlarge
  maxSize: 1
  minSize: 0
  nodeLabels:
    kops.k8s.io/cluster: k8s.staging.ue1.company.io
    kops.k8s.io/instancegroup: on-demand-workers-subnet-0a5f08e5d52b90c6f
  role: Node
  rootVolumeEncryption: true
  rootVolumeSize: 100
  subnets:
  - staging.ue1-private-us-east-1b
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:

    kops.k8s.io/cluster: k8s.staging.ue1.company.io
  name: on-demand-workers-subnet-0d0cf97dac1a37cc8
spec:
  additionalSecurityGroups:
  - sg-0123380c16589c89e
  associatePublicIp: false
  cloudLabels:
    environment: staging
  image: ami-123
  machineType: m5a.2xlarge
  maxSize: 1
  minSize: 0
  nodeLabels:

    kops.k8s.io/cluster: k8s.staging.ue1.company.io
    kops.k8s.io/instancegroup: on-demand-workers-subnet-0d0cf97dac1a37cc8
  role: Node
  rootVolumeEncryption: true
  rootVolumeSize: 100
  subnets:
  - staging.ue1-private-us-east-1c

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

etcd-manager used is v3.4.13.

etcd-manager logs say what tags are used to discover volumes:

Mounting available etcd volumes matching tags [k8s.io/etcd/main k8s.io/role/master=1 kubernetes.io/cluster/k8s-foo.cluster.io=owned]; nameTag=k8s.io/etcd/main

This matches all etcd main volumes, and filtered down to a specific AZ it matches multiple etcd's volumes within the AZ.

In my case, I ended up with one master having 3 volumes attached, and with one with 1 volume attached. Not only the volumes were switched between the two masters, but one of them had two volumes of the same kind (events).

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions