Still getting an error related to Issue 194
PR 198
If I don't provide a project id to egress firewall the outcome is:
except from terraform config:
main.tf
terraform {
required_providers {
cloudstack = {
source = "local/cloudstack/cloudstack"
version = "0.6.0-rc2"
}
talos = {
source = "siderolabs/talos"
version = "0.9.0"
}
}
}
provider "cloudstack" {
# Configuration options
api_url = var.api_url
api_key = var.api_key
secret_key = var.secret_key
}
network,tf
# Guest network used for K8s
resource "cloudstack_network" "k8s_nw_01" {
name = var.k8s_cluster_network_name
cidr = "${local.network_cidr}"
network_offering = "DefaultIsolatedNetworkOfferingWithSourceNatService"
zone = var.zone
project = var.project_id
}
# Egress firewall and rules for outside communication
resource "cloudstack_egress_firewall" "default_egress_fw_01" {
network_id = cloudstack_network.k8s_nw_01.id
depends_on = [ cloudstack_instance.controller ]
rule {
cidr_list = ["${local.network_cidr}"]
protocol = "tcp"
ports = ["53", "80", "443"]
}
rule {
cidr_list = ["${local.network_cidr}"]
protocol = "udp"
ports = ["53", "123"]
}
rule {
cidr_list = ["${local.network_cidr}"]
protocol = "tcp"
ports = ["8443"]
}
}
# SNAT ip address
resource "cloudstack_ipaddress" "k8s_ips01" {
network_id = cloudstack_network.k8s_nw_01.id
project = var.project_id
}
# Load balancer for K8s API
resource "cloudstack_loadbalancer_rule" "k8s_lb_k8s_api" {
#depends_on = [ cloudstack_instance.controller ]
project = var.project_id
algorithm = "roundrobin"
ip_address_id = cloudstack_ipaddress.k8s_ips01.id
member_ids = [ cloudstack_instance.controller[0].id ]
name = "lb-k8s-api"
private_port = 6443
public_port = 6443
}
# Loadbalancer for Talos API
resource "cloudstack_loadbalancer_rule" "k8s_lb_talos_api" {
#depends_on = [ cloudstack_instance.controller ]
project = var.project_id
algorithm = "roundrobin"
ip_address_id = cloudstack_ipaddress.k8s_ips01.id
member_ids = [ cloudstack_instance.controller[0].id ]
name = "lb-talos-api"
private_port = 50000
public_port = 50000
}
# Firewall for accessing the SNAT ip address
# TODO: set a valid range
resource "cloudstack_firewall" "default_lbfw01" {
ip_address_id = cloudstack_ipaddress.k8s_ips01.id
depends_on = [ cloudstack_instance.controller ]
rule {
cidr_list = ["0.0.0.0/0"]
protocol = "tcp"
ports = ["6443", "50000"]
}
}
output apply:
module.k8s.talos_machine_secrets.talos: Creating...
module.k8s.cloudstack_network.k8s_nw_01: Creating...
module.k8s.cloudstack_network.k8s_nw_01: Creation complete after 1s [id=103adc96-9c51-446b-ac93-055f098364b3]
module.k8s.cloudstack_ipaddress.k8s_ips01: Creating...
module.k8s.talos_machine_secrets.talos: Creation complete after 2s [id=machine_secrets]
module.k8s.data.talos_client_configuration.talos: Reading...
module.k8s.data.talos_machine_configuration.worker: Reading...
module.k8s.data.talos_client_configuration.talos: Read complete after 0s [id=test-cluster-mm]
module.k8s.data.talos_machine_configuration.worker: Read complete after 0s [id=test-cluster-mm]
module.k8s.cloudstack_ipaddress.k8s_ips01: Creation complete after 1s [id=f4f4162d-510c-4ff2-b630-c82eab3717a1]
module.k8s.data.talos_machine_configuration.controller[0]: Reading...
module.k8s.data.talos_machine_configuration.controller[0]: Read complete after 0s [id=test-cluster-mm]
module.k8s.cloudstack_instance.controller[0]: Creating...
module.k8s.cloudstack_instance.controller[0]: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [30s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [40s elapsed]
module.k8s.cloudstack_instance.controller[0]: Creation complete after 46s [id=9dc18c37-d743-49f4-ae97-b19a952efe17]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creating...
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creating...
module.k8s.cloudstack_instance.worker[1]: Creating...
module.k8s.cloudstack_instance.worker[0]: Creating...
module.k8s.cloudstack_firewall.default_lbfw01: Creating...
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Creating...
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Still creating... [10s elapsed]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.worker[1]: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.worker[0]: Still creating... [10s elapsed]
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Still creating... [10s elapsed]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creation complete after 10s [id=480f288c-06d2-4d9a-97c4-b5abf02fdf00]
module.k8s.cloudstack_instance.worker[0]: Creation complete after 15s [id=1054386c-00e5-4a35-98b1-0ca3e57deb4e]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creation complete after 16s [id=f20ae740-6f1d-4a69-b7b1-821ac4222048]
module.k8s.cloudstack_instance.worker[1]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.worker[1]: Creation complete after 28s [id=82b4ddd2-e297-439c-94ff-cd1af1276b22]
module.k8s.talos_machine_bootstrap.this: Creating...
module.k8s.talos_machine_bootstrap.this: Creation complete after 0s [id=machine_bootstrap]
module.k8s.talos_cluster_kubeconfig.kubeconfig: Creating...
module.k8s.talos_cluster_kubeconfig.kubeconfig: Creation complete after 0s [id=test-cluster-mm]
module.k8s.local_file.kubeconfig: Creating...
module.k8s.local_file.kubeconfig: Creation complete after 0s [id=dd623c39ef9e7a1af82c8c2962ef66f27b9fb1df]
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.k8s.cloudstack_egress_firewall.default_egress_fw_01, provider "module.k8s.provider[\"local/cloudstack/cloudstack\"]" produced an unexpected new value: root object was present, but now absent.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.k8s.cloudstack_firewall.default_lbfw01, provider "module.k8s.provider[\"local/cloudstack/cloudstack\"]" produced an unexpected new value: root object was present, but now absent.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
output state list
module.k8s.data.talos_client_configuration.talos
module.k8s.data.talos_machine_configuration.controller[0]
module.k8s.data.talos_machine_configuration.worker
module.k8s.cloudstack_instance.controller[0]
module.k8s.cloudstack_instance.worker[0]
module.k8s.cloudstack_instance.worker[1]
module.k8s.cloudstack_ipaddress.k8s_ips01
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api
module.k8s.cloudstack_network.k8s_nw_01
module.k8s.local_file.kubeconfig
module.k8s.talos_cluster_kubeconfig.kubeconfig
module.k8s.talos_machine_bootstrap.this
module.k8s.talos_machine_secrets.talos
cloudstack_egress_firewall is missing in state.
with the project id:
network.tf
# Guest network used for K8s
resource "cloudstack_network" "k8s_nw_01" {
name = var.k8s_cluster_network_name
cidr = "${local.network_cidr}"
network_offering = "DefaultIsolatedNetworkOfferingWithSourceNatService"
zone = var.zone
project = var.project_id
}
# Egress firewall and rules for outside communication
resource "cloudstack_egress_firewall" "default_egress_fw_01" {
network_id = cloudstack_network.k8s_nw_01.id
project = var.project_id
depends_on = [ cloudstack_instance.controller ]
rule {
cidr_list = ["${local.network_cidr}"]
protocol = "tcp"
ports = ["53", "80", "443"]
}
rule {
cidr_list = ["${local.network_cidr}"]
protocol = "udp"
ports = ["53", "123"]
}
rule {
cidr_list = ["${local.network_cidr}"]
protocol = "tcp"
ports = ["8443"]
}
}
# SNAT ip address
resource "cloudstack_ipaddress" "k8s_ips01" {
network_id = cloudstack_network.k8s_nw_01.id
project = var.project_id
}
# Load balancer for K8s API
resource "cloudstack_loadbalancer_rule" "k8s_lb_k8s_api" {
#depends_on = [ cloudstack_instance.controller ]
project = var.project_id
algorithm = "roundrobin"
ip_address_id = cloudstack_ipaddress.k8s_ips01.id
member_ids = [ cloudstack_instance.controller[0].id ]
name = "lb-k8s-api"
private_port = 6443
public_port = 6443
}
# Loadbalancer for Talos API
resource "cloudstack_loadbalancer_rule" "k8s_lb_talos_api" {
#depends_on = [ cloudstack_instance.controller ]
project = var.project_id
algorithm = "roundrobin"
ip_address_id = cloudstack_ipaddress.k8s_ips01.id
member_ids = [ cloudstack_instance.controller[0].id ]
name = "lb-talos-api"
private_port = 50000
public_port = 50000
}
# Firewall for accessing the SNAT ip address
# TODO: set a valid range
resource "cloudstack_firewall" "default_lbfw01" {
ip_address_id = cloudstack_ipaddress.k8s_ips01.id
depends_on = [ cloudstack_instance.controller ]
rule {
cidr_list = ["0.0.0.0/0"]
protocol = "tcp"
ports = ["6443", "50000"]
}
}
output apply:
module.k8s.talos_machine_secrets.talos: Creating...
module.k8s.cloudstack_network.k8s_nw_01: Creating...
module.k8s.talos_machine_secrets.talos: Creation complete after 0s [id=machine_secrets]
module.k8s.data.talos_client_configuration.talos: Reading...
module.k8s.data.talos_machine_configuration.worker: Reading...
module.k8s.data.talos_client_configuration.talos: Read complete after 0s [id=test-cluster-mm]
module.k8s.data.talos_machine_configuration.worker: Read complete after 0s [id=test-cluster-mm]
module.k8s.cloudstack_network.k8s_nw_01: Creation complete after 0s [id=eee99502-d2f3-43f4-a0ad-6a15e621c84b]
module.k8s.cloudstack_ipaddress.k8s_ips01: Creating...
module.k8s.cloudstack_ipaddress.k8s_ips01: Creation complete after 1s [id=f4f4162d-510c-4ff2-b630-c82eab3717a1]
module.k8s.data.talos_machine_configuration.controller[0]: Reading...
module.k8s.data.talos_machine_configuration.controller[0]: Read complete after 0s [id=test-cluster-mm]
module.k8s.cloudstack_instance.controller[0]: Creating...
module.k8s.cloudstack_instance.controller[0]: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [30s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [40s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [50s elapsed]
module.k8s.cloudstack_instance.controller[0]: Creation complete after 56s [id=47e3b974-5f08-4dbf-bf0b-37f5b32cfde7]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creating...
module.k8s.cloudstack_firewall.default_lbfw01: Creating...
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creating...
module.k8s.cloudstack_instance.worker[0]: Creating...
module.k8s.cloudstack_instance.worker[1]: Creating...
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Creating...
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creation complete after 6s [id=b47f9269-55f3-4189-a342-90e44e236190]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.worker[0]: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.worker[1]: Still creating... [10s elapsed]
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Still creating... [10s elapsed]
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Creation complete after 13s [id=eee99502-d2f3-43f4-a0ad-6a15e621c84b]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creation complete after 16s [id=82ca7dfc-a7a1-4846-b1b4-a9196db0d702]
module.k8s.cloudstack_instance.worker[0]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.worker[1]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.worker[0]: Creation complete after 22s [id=39be1280-853b-4f22-9b72-f5c698dad14e]
module.k8s.cloudstack_instance.worker[1]: Creation complete after 29s [id=fd08a259-8165-45f7-958a-1315d47b1efc]
module.k8s.talos_machine_bootstrap.this: Creating...
module.k8s.talos_machine_bootstrap.this: Still creating... [10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [1m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [2m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [3m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [4m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [5m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [6m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [7m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [8m50s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m0s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m10s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m20s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m30s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m40s elapsed]
module.k8s.talos_machine_bootstrap.this: Still creating... [9m50s elapsed]
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.k8s.cloudstack_firewall.default_lbfw01, provider "module.k8s.provider[\"local/cloudstack/cloudstack\"]" produced an unexpected new value: root object was present, but now absent.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Error bootstrapping node
│
│ with module.k8s.talos_machine_bootstrap.this,
│ on modules/k8s/k8s-01.tf line 167, in resource "talos_machine_bootstrap" "this":
│ 167: resource "talos_machine_bootstrap" "this" {
│
│ rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for xxx.xxx.xxx.xxx
output state list:
module.k8s.data.talos_client_configuration.talos
module.k8s.data.talos_machine_configuration.controller[0]
module.k8s.data.talos_machine_configuration.worker
module.k8s.cloudstack_egress_firewall.default_egress_fw_01
module.k8s.cloudstack_instance.controller[0]
module.k8s.cloudstack_instance.worker[0]
module.k8s.cloudstack_instance.worker[1]
module.k8s.cloudstack_ipaddress.k8s_ips01
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api
module.k8s.cloudstack_network.k8s_nw_01
module.k8s.talos_machine_secrets.talos
With the project id provided to cloudstack_egress_firewall the egress firewall is available in the state, but bootstrapping the kubernetes cluster fails afterwards.
We testet multiple times with and without projectid. Without the bootstrap is working flawlessly.
Still getting an error related to Issue 194
PR 198
If I don't provide a project id to egress firewall the outcome is:
except from terraform config:
main.tf
network,tf
output apply:
output state list
cloudstack_egress_firewall is missing in state.
with the project id:
network.tf
output apply:
output state list:
With the project id provided to cloudstack_egress_firewall the egress firewall is available in the state, but bootstrapping the kubernetes cluster fails afterwards.
We testet multiple times with and without projectid. Without the bootstrap is working flawlessly.