Skip to content

Root object was present, but now absent #241

Description

@HeinzM

Still getting an error related to Issue 194
PR 198

If I don't provide a project id to egress firewall the outcome is:

except from terraform config:
main.tf

terraform {
  required_providers {
    cloudstack = {
      source = "local/cloudstack/cloudstack"
      version = "0.6.0-rc2"
    }
    talos = {
      source  = "siderolabs/talos"
      version = "0.9.0"
    }
  }
}

provider "cloudstack" {
  # Configuration options
  api_url    = var.api_url
  api_key    = var.api_key
  secret_key = var.secret_key
}

network,tf

# Guest network used for K8s
resource "cloudstack_network" "k8s_nw_01" {
  name             = var.k8s_cluster_network_name
  cidr             = "${local.network_cidr}"
  network_offering = "DefaultIsolatedNetworkOfferingWithSourceNatService"
  zone             = var.zone
  project             = var.project_id
}

# Egress firewall and rules for outside communication
resource "cloudstack_egress_firewall" "default_egress_fw_01" {
  network_id          = cloudstack_network.k8s_nw_01.id
  depends_on          = [ cloudstack_instance.controller ]

  rule {
    cidr_list = ["${local.network_cidr}"]
    protocol  = "tcp"
    ports     = ["53", "80", "443"]
  }

  rule {
    cidr_list = ["${local.network_cidr}"]
    protocol  = "udp"
    ports     = ["53", "123"]
  }

 
  rule {
    cidr_list = ["${local.network_cidr}"]
    protocol  = "tcp"
    ports     = ["8443"]
  }
}

# SNAT ip address
resource "cloudstack_ipaddress" "k8s_ips01" {
  network_id          = cloudstack_network.k8s_nw_01.id
  project             = var.project_id
}

# Load balancer for K8s API
resource "cloudstack_loadbalancer_rule" "k8s_lb_k8s_api" {
  #depends_on = [ cloudstack_instance.controller ]
  project             = var.project_id 
  algorithm           = "roundrobin"
  ip_address_id       = cloudstack_ipaddress.k8s_ips01.id
  member_ids          = [ cloudstack_instance.controller[0].id ]
  name                = "lb-k8s-api"
  private_port        = 6443
  public_port         = 6443
}

# Loadbalancer for Talos API
resource "cloudstack_loadbalancer_rule" "k8s_lb_talos_api" {
  #depends_on = [ cloudstack_instance.controller ]
  project             = var.project_id 
  algorithm           = "roundrobin"
  ip_address_id       = cloudstack_ipaddress.k8s_ips01.id
  member_ids          = [ cloudstack_instance.controller[0].id ]
  name                = "lb-talos-api"
  private_port        = 50000
  public_port         = 50000
}

# Firewall for accessing the SNAT ip address
# TODO: set a valid range
resource "cloudstack_firewall" "default_lbfw01" {
  ip_address_id       = cloudstack_ipaddress.k8s_ips01.id
  depends_on = [ cloudstack_instance.controller ]

  rule {
    cidr_list         = ["0.0.0.0/0"]
    protocol          = "tcp"
    ports             = ["6443", "50000"]
  }
}

output apply:

module.k8s.talos_machine_secrets.talos: Creating...
module.k8s.cloudstack_network.k8s_nw_01: Creating...
module.k8s.cloudstack_network.k8s_nw_01: Creation complete after 1s [id=103adc96-9c51-446b-ac93-055f098364b3]
module.k8s.cloudstack_ipaddress.k8s_ips01: Creating...
module.k8s.talos_machine_secrets.talos: Creation complete after 2s [id=machine_secrets]
module.k8s.data.talos_client_configuration.talos: Reading...
module.k8s.data.talos_machine_configuration.worker: Reading...
module.k8s.data.talos_client_configuration.talos: Read complete after 0s [id=test-cluster-mm]
module.k8s.data.talos_machine_configuration.worker: Read complete after 0s [id=test-cluster-mm]
module.k8s.cloudstack_ipaddress.k8s_ips01: Creation complete after 1s [id=f4f4162d-510c-4ff2-b630-c82eab3717a1]
module.k8s.data.talos_machine_configuration.controller[0]: Reading...
module.k8s.data.talos_machine_configuration.controller[0]: Read complete after 0s [id=test-cluster-mm]
module.k8s.cloudstack_instance.controller[0]: Creating...
module.k8s.cloudstack_instance.controller[0]: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [30s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [40s elapsed]
module.k8s.cloudstack_instance.controller[0]: Creation complete after 46s [id=9dc18c37-d743-49f4-ae97-b19a952efe17]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creating...
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creating...
module.k8s.cloudstack_instance.worker[1]: Creating...
module.k8s.cloudstack_instance.worker[0]: Creating...
module.k8s.cloudstack_firewall.default_lbfw01: Creating...
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Creating...
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Still creating... [10s elapsed]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.worker[1]: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.worker[0]: Still creating... [10s elapsed]
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Still creating... [10s elapsed]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creation complete after 10s [id=480f288c-06d2-4d9a-97c4-b5abf02fdf00]
module.k8s.cloudstack_instance.worker[0]: Creation complete after 15s [id=1054386c-00e5-4a35-98b1-0ca3e57deb4e]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creation complete after 16s [id=f20ae740-6f1d-4a69-b7b1-821ac4222048]
module.k8s.cloudstack_instance.worker[1]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.worker[1]: Creation complete after 28s [id=82b4ddd2-e297-439c-94ff-cd1af1276b22]
module.k8s.talos_machine_bootstrap.this: Creating...
module.k8s.talos_machine_bootstrap.this: Creation complete after 0s [id=machine_bootstrap]
module.k8s.talos_cluster_kubeconfig.kubeconfig: Creating...
module.k8s.talos_cluster_kubeconfig.kubeconfig: Creation complete after 0s [id=test-cluster-mm]
module.k8s.local_file.kubeconfig: Creating...
module.k8s.local_file.kubeconfig: Creation complete after 0s [id=dd623c39ef9e7a1af82c8c2962ef66f27b9fb1df]
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to module.k8s.cloudstack_egress_firewall.default_egress_fw_01, provider "module.k8s.provider[\"local/cloudstack/cloudstack\"]" produced an unexpected new value: root object was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to module.k8s.cloudstack_firewall.default_lbfw01, provider "module.k8s.provider[\"local/cloudstack/cloudstack\"]" produced an unexpected new value: root object was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

output state list

module.k8s.data.talos_client_configuration.talos
module.k8s.data.talos_machine_configuration.controller[0]
module.k8s.data.talos_machine_configuration.worker
module.k8s.cloudstack_instance.controller[0]
module.k8s.cloudstack_instance.worker[0]
module.k8s.cloudstack_instance.worker[1]
module.k8s.cloudstack_ipaddress.k8s_ips01
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api
module.k8s.cloudstack_network.k8s_nw_01
module.k8s.local_file.kubeconfig
module.k8s.talos_cluster_kubeconfig.kubeconfig
module.k8s.talos_machine_bootstrap.this
module.k8s.talos_machine_secrets.talos

cloudstack_egress_firewall is missing in state.

with the project id:

network.tf

# Guest network used for K8s
resource "cloudstack_network" "k8s_nw_01" {
  name             = var.k8s_cluster_network_name
  cidr             = "${local.network_cidr}"
  network_offering = "DefaultIsolatedNetworkOfferingWithSourceNatService"
  zone             = var.zone
  project             = var.project_id
}

# Egress firewall and rules for outside communication
resource "cloudstack_egress_firewall" "default_egress_fw_01" {
  network_id          = cloudstack_network.k8s_nw_01.id
  project             = var.project_id
  depends_on          = [ cloudstack_instance.controller ]

  rule {
    cidr_list = ["${local.network_cidr}"]
    protocol  = "tcp"
    ports     = ["53", "80", "443"]
  }

  rule {
    cidr_list = ["${local.network_cidr}"]
    protocol  = "udp"
    ports     = ["53", "123"]
  }

  
  rule {
    cidr_list = ["${local.network_cidr}"]
    protocol  = "tcp"
    ports     = ["8443"]
  }
}

# SNAT ip address
resource "cloudstack_ipaddress" "k8s_ips01" {
  network_id          = cloudstack_network.k8s_nw_01.id
  project             = var.project_id
}

# Load balancer for K8s API
resource "cloudstack_loadbalancer_rule" "k8s_lb_k8s_api" {
  #depends_on = [ cloudstack_instance.controller ]
  project             = var.project_id 
  algorithm           = "roundrobin"
  ip_address_id       = cloudstack_ipaddress.k8s_ips01.id
  member_ids          = [ cloudstack_instance.controller[0].id ]
  name                = "lb-k8s-api"
  private_port        = 6443
  public_port         = 6443
}

# Loadbalancer for Talos API
resource "cloudstack_loadbalancer_rule" "k8s_lb_talos_api" {
  #depends_on = [ cloudstack_instance.controller ]
  project             = var.project_id 
  algorithm           = "roundrobin"
  ip_address_id       = cloudstack_ipaddress.k8s_ips01.id
  member_ids          = [ cloudstack_instance.controller[0].id ]
  name                = "lb-talos-api"
  private_port        = 50000
  public_port         = 50000
}

# Firewall for accessing the SNAT ip address
# TODO: set a valid range
resource "cloudstack_firewall" "default_lbfw01" {
  ip_address_id       = cloudstack_ipaddress.k8s_ips01.id
  depends_on = [ cloudstack_instance.controller ]

  rule {
    cidr_list         = ["0.0.0.0/0"]
    protocol          = "tcp"
    ports             = ["6443", "50000"]
  }
}

output apply:

module.k8s.talos_machine_secrets.talos: Creating...
module.k8s.cloudstack_network.k8s_nw_01: Creating...
module.k8s.talos_machine_secrets.talos: Creation complete after 0s [id=machine_secrets]
module.k8s.data.talos_client_configuration.talos: Reading...
module.k8s.data.talos_machine_configuration.worker: Reading...
module.k8s.data.talos_client_configuration.talos: Read complete after 0s [id=test-cluster-mm]
module.k8s.data.talos_machine_configuration.worker: Read complete after 0s [id=test-cluster-mm]
module.k8s.cloudstack_network.k8s_nw_01: Creation complete after 0s [id=eee99502-d2f3-43f4-a0ad-6a15e621c84b]
module.k8s.cloudstack_ipaddress.k8s_ips01: Creating...
module.k8s.cloudstack_ipaddress.k8s_ips01: Creation complete after 1s [id=f4f4162d-510c-4ff2-b630-c82eab3717a1]
module.k8s.data.talos_machine_configuration.controller[0]: Reading...
module.k8s.data.talos_machine_configuration.controller[0]: Read complete after 0s [id=test-cluster-mm]
module.k8s.cloudstack_instance.controller[0]: Creating...
module.k8s.cloudstack_instance.controller[0]: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [30s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [40s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [50s elapsed]
module.k8s.cloudstack_instance.controller[0]: Creation complete after 56s [id=47e3b974-5f08-4dbf-bf0b-37f5b32cfde7]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creating...
module.k8s.cloudstack_firewall.default_lbfw01: Creating...
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creating...
module.k8s.cloudstack_instance.worker[0]: Creating...
module.k8s.cloudstack_instance.worker[1]: Creating...
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Creating...
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creation complete after 6s [id=b47f9269-55f3-4189-a342-90e44e236190]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.worker[0]: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.worker[1]: Still creating... [10s elapsed]
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Still creating... [10s elapsed]
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Creation complete after 13s [id=eee99502-d2f3-43f4-a0ad-6a15e621c84b]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creation complete after 16s [id=82ca7dfc-a7a1-4846-b1b4-a9196db0d702]
module.k8s.cloudstack_instance.worker[0]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.worker[1]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.worker[0]: Creation complete after 22s [id=39be1280-853b-4f22-9b72-f5c698dad14e]
module.k8s.cloudstack_instance.worker[1]: Creation complete after 29s [id=fd08a259-8165-45f7-958a-1315d47b1efc]
module.k8s.talos_machine_bootstrap.this: Creating...
module.k8s.talos_machine_bootstrap.this: Still creating... [10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m50s elapsed]
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to module.k8s.cloudstack_firewall.default_lbfw01, provider "module.k8s.provider[\"local/cloudstack/cloudstack\"]" produced an unexpected new value: root object was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Error bootstrapping node
│ 
│   with module.k8s.talos_machine_bootstrap.this,
│   on modules/k8s/k8s-01.tf line 167, in resource "talos_machine_bootstrap" "this":
│  167: resource "talos_machine_bootstrap" "this" {
│ 
│ rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for xxx.xxx.xxx.xxx

output state list:

module.k8s.data.talos_client_configuration.talos
module.k8s.data.talos_machine_configuration.controller[0]
module.k8s.data.talos_machine_configuration.worker
module.k8s.cloudstack_egress_firewall.default_egress_fw_01
module.k8s.cloudstack_instance.controller[0]
module.k8s.cloudstack_instance.worker[0]
module.k8s.cloudstack_instance.worker[1]
module.k8s.cloudstack_ipaddress.k8s_ips01
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api
module.k8s.cloudstack_network.k8s_nw_01
module.k8s.talos_machine_secrets.talos

With the project id provided to cloudstack_egress_firewall the egress firewall is available in the state, but bootstrapping the kubernetes cluster fails afterwards.
We testet multiple times with and without projectid. Without the bootstrap is working flawlessly.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions