Fix creation of firewall & Egress firewall rules when created in a project

[Pearl1594]

Fixes: #194

Terraform config:

terraform {
  required_providers {
    cloudstack = {
      source  = "cloudstack/cloudstack"
      version = "0.5.0"
    }
  }
}

provider "cloudstack" {
  api_url    = "http://xx.xx.xx.xx:8080/client/api"
  api_key    = "LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8"
  secret_key = "R6QPwRUz09TVXBjXNwZk7grTjcPtsFRphH6xhN1oPvnc12YUk296t4KHytg8zRLczDA0X5NsLVi4d8rfMMx3g"

}


# Guest network used for K8s
resource "cloudstack_network" "k8s_nw_01" {
  name                = "terraform-net"
  cidr                = "10.0.0.0/16"
  network_offering    = "DefaultIsolatedNetworkOfferingWithSourceNatService"
  zone                = "2cf3d03d-f8ba-44f9-9543-d05e7af3a9b9"
  project             = "22a47bf0-67f3-4eaa-92dc-6b8070a55e42"
}

# Egress firewall and rules for outside communication
resource "cloudstack_egress_firewall" "default_egress_fw_01" {
  network_id          = cloudstack_network.k8s_nw_01.id
  project             = "22a47bf0-67f3-4eaa-92dc-6b8070a55e42"
  depends_on          = [ cloudstack_network.k8s_nw_01,cloudstack_ipaddress.k8s_ips01 ]

  rule {
    cidr_list         = ["10.0.0.0/16"]
    protocol          = "tcp"
    ports             = ["53", "80", "443"]
  }

  rule {
    cidr_list         = ["10.0.0.0/16"]
    protocol          = "udp"
    ports             = ["53", "123"]
  }

  # TODO: Find out where to limit this rule to the destination ip of the Cloudstack API
  rule {
    cidr_list         = ["10.0.0.0/16"]
    ports             = ["8443"]
    protocol          = "tcp"
  }
}

# SNAT ip address
resource "cloudstack_ipaddress" "k8s_ips01" {
  network_id          = cloudstack_network.k8s_nw_01.id
  project             = "22a47bf0-67f3-4eaa-92dc-6b8070a55e42"
}

# Load balancer for K8s API
resource "cloudstack_loadbalancer_rule" "k8s_lb_k8s_api" {
  depends_on = [ cloudstack_instance.controller,cloudstack_ipaddress.k8s_ips01]
  project             = "22a47bf0-67f3-4eaa-92dc-6b8070a55e42" 
  algorithm           = "roundrobin"
  ip_address_id       = cloudstack_ipaddress.k8s_ips01.id
  member_ids          = [ cloudstack_instance.controller.id ]
  name                = "lb-k8s-api"
  private_port        = 6443
  public_port         = 6443
}

# Loadbalancer for Talos API
resource "cloudstack_loadbalancer_rule" "k8s_lb_talos_api" {
  depends_on = [ cloudstack_instance.controller,cloudstack_ipaddress.k8s_ips01 ]
  project             = "22a47bf0-67f3-4eaa-92dc-6b8070a55e42"
  algorithm           = "roundrobin"
  ip_address_id       = cloudstack_ipaddress.k8s_ips01.id
  member_ids          = [cloudstack_instance.controller.id]
  name                = "lb-talos-api"
  private_port        = 50000
  public_port         = 50000
}

# Firewall for accessing the SNAT ip address
# TODO: set a valid range
resource "cloudstack_firewall" "default_lbfw01" {
  ip_address_id       = cloudstack_ipaddress.k8s_ips01.id
  project             = "22a47bf0-67f3-4eaa-92dc-6b8070a55e42"
  depends_on = [ cloudstack_network.k8s_nw_01,cloudstack_ipaddress.k8s_ips01 ]

  rule {
    cidr_list         = ["0.0.0.0/0"]
    protocol          = "tcp"
    ports             = ["6443", "50000"]
  }
}



resource "cloudstack_instance" "controller" {
  depends_on = [ cloudstack_network.k8s_nw_01 ]
  name             = "server-1"
  service_offering = "Small Instance"
  network_id       = cloudstack_network.k8s_nw_01.id
  template         = "fed7ee6e-60e2-11f0-afea-1e0030000314"
  zone             = "2cf3d03d-f8ba-44f9-9543-d05e7af3a9b9"
  project          = "22a47bf0-67f3-4eaa-92dc-6b8070a55e42"
}

project field added to egress / firewall rule resources. Successfully created all 7 resources

O/p of terraform apply:

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

----------------------------------------------------------------------------------------------------------
$ terraform state list
cloudstack_egress_firewall.default_egress_fw_01
cloudstack_firewall.default_lbfw01
cloudstack_instance.controller
cloudstack_ipaddress.k8s_ips01
cloudstack_loadbalancer_rule.k8s_lb_k8s_api
cloudstack_loadbalancer_rule.k8s_lb_talos_api
cloudstack_network.k8s_nw_01

----------------------------------------------------------------------------------------------------------
$ terraform state show cloudstack_egress_firewall.default_egress_fw_01
# cloudstack_egress_firewall.default_egress_fw_01:
resource "cloudstack_egress_firewall" "default_egress_fw_01" {
    id          = "87d2be23-9259-4801-9417-126e1da36673"
    managed     = false
    network_id  = "87d2be23-9259-4801-9417-126e1da36673"
    parallelism = 2
    project     = "22a47bf0-67f3-4eaa-92dc-6b8070a55e42"

    rule {
        cidr_list = [
            "10.0.0.0/16",
        ]
        icmp_code = 0
        icmp_type = 0
        ports     = [
            "123",
            "53",
        ]
        protocol  = "udp"
        uuids     = {
            "123" = "c48aaf7e-635c-49ce-9b72-93a9d61246e5"
            "53"  = "737ccba0-73a9-4cb6-bc3a-ccf7998e1fea"
        }
    }
    rule {
        cidr_list = [
            "10.0.0.0/16",
        ]
        icmp_code = 0
        icmp_type = 0
        ports     = [
            "443",
            "53",
            "80",
        ]
        protocol  = "tcp"
        uuids     = {
            "443" = "d2f8edc5-c819-4706-8a7b-8d2e74f3b7e8"
            "53"  = "2cdd864a-cd35-42bd-8591-f3f4d641259a"
            "80"  = "ad708783-85d4-4c03-951a-bbd7a15ab22b"
        }
    }
    rule {
        cidr_list = [
            "10.0.0.0/16",
        ]
        icmp_code = 0
        icmp_type = 0
        ports     = [
            "8443",
        ]
        protocol  = "tcp"
        uuids     = {
            "8443" = "18b7305d-39ef-42fd-bc39-2eaffc023ac0"
        }
    }
}

----------------------------------------------------------------------------------------------------------

$ terraform state show cloudstack_firewall.default_lbfw01
# cloudstack_firewall.default_lbfw01:
resource "cloudstack_firewall" "default_lbfw01" {
    id            = "ef3c5b9b-7f2e-4be5-be00-9ed1e6a11131"
    ip_address_id = "ef3c5b9b-7f2e-4be5-be00-9ed1e6a11131"
    managed       = false
    parallelism   = 2
    project       = "22a47bf0-67f3-4eaa-92dc-6b8070a55e42"

    rule {
        cidr_list = [
            "0.0.0.0/0",
        ]
        icmp_code = 0
        icmp_type = 0
        ports     = [
            "50000",
            "6443",
        ]
        protocol  = "tcp"
        uuids     = {
            "50000" = "928f50ad-04af-42b8-9004-4ca2ab611866"
            "6443"  = "35734418-446f-496e-9408-c38416a71d35"
        }
    }
}

[baltazorbest]

Cool, now it's work for me, many thanks 👍

[DaanHoogland]

clgtm

[DaanHoogland]

@kiranchavala I think we can merge as a 3rd person test has been done by @baltazorbest , but I leave you the honour.

[baltazorbest]

Could you please make new release with the fix?

[HeinzM]

Tested the creation of a talos cluster with
Cloudstack: 4.21.0
cloudstack-terraform-provider: 0.6.0.-rc2
siderolabs-talos-provider: 0.9.0

module.k8s.talos_machine_secrets.talos: Creating...
module.k8s.cloudstack_network.k8s_nw_01: Creating...
module.k8s.cloudstack_network.k8s_nw_01: Creation complete after 1s [id=103adc96-9c51-446b-ac93-055f098364b3]
module.k8s.cloudstack_ipaddress.k8s_ips01: Creating...
module.k8s.talos_machine_secrets.talos: Creation complete after 2s [id=machine_secrets]
module.k8s.data.talos_client_configuration.talos: Reading...
module.k8s.data.talos_machine_configuration.worker: Reading...
module.k8s.data.talos_client_configuration.talos: Read complete after 0s [id=test-cluster-mm]
module.k8s.data.talos_machine_configuration.worker: Read complete after 0s [id=test-cluster-mm]
module.k8s.cloudstack_ipaddress.k8s_ips01: Creation complete after 1s [id=f4f4162d-510c-4ff2-b630-c82eab3717a1]
module.k8s.data.talos_machine_configuration.controller[0]: Reading...
module.k8s.data.talos_machine_configuration.controller[0]: Read complete after 0s [id=test-cluster-mm]
module.k8s.cloudstack_instance.controller[0]: Creating...
module.k8s.cloudstack_instance.controller[0]: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [30s elapsed]
module.k8s.cloudstack_instance.controller[0]: Still creating... [40s elapsed]
module.k8s.cloudstack_instance.controller[0]: Creation complete after 46s [id=9dc18c37-d743-49f4-ae97-b19a952efe17]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creating...
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creating...
module.k8s.cloudstack_instance.worker[1]: Creating...
module.k8s.cloudstack_instance.worker[0]: Creating...
module.k8s.cloudstack_firewall.default_lbfw01: Creating...
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Creating...
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Still creating... [10s elapsed]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.worker[1]: Still creating... [10s elapsed]
module.k8s.cloudstack_instance.worker[0]: Still creating... [10s elapsed]
module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Still creating... [10s elapsed]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creation complete after 10s [id=480f288c-06d2-4d9a-97c4-b5abf02fdf00]
module.k8s.cloudstack_instance.worker[0]: Creation complete after 15s [id=1054386c-00e5-4a35-98b1-0ca3e57deb4e]
module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creation complete after 16s [id=f20ae740-6f1d-4a69-b7b1-821ac4222048]
module.k8s.cloudstack_instance.worker[1]: Still creating... [20s elapsed]
module.k8s.cloudstack_instance.worker[1]: Creation complete after 28s [id=82b4ddd2-e297-439c-94ff-cd1af1276b22]
module.k8s.talos_machine_bootstrap.this: Creating...
module.k8s.talos_machine_bootstrap.this: Creation complete after 0s [id=machine_bootstrap]
module.k8s.talos_cluster_kubeconfig.kubeconfig: Creating...
module.k8s.talos_cluster_kubeconfig.kubeconfig: Creation complete after 0s [id=test-cluster-mm]
module.k8s.local_file.kubeconfig: Creating...
module.k8s.local_file.kubeconfig: Creation complete after 0s [id=dd623c39ef9e7a1af82c8c2962ef66f27b9fb1df]
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to module.k8s.cloudstack_egress_firewall.default_egress_fw_01, provider "module.k8s.provider[\"local/cloudstack/cloudstack\"]" produced an unexpected new value: root object was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to module.k8s.cloudstack_firewall.default_lbfw01, provider "module.k8s.provider[\"local/cloudstack/cloudstack\"]" produced an unexpected new value: root object was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Tested before with the self-build 0.5.0 after the PR was merged. Same result.
The error still exists.

[Pearl1594]

@HeinzM could you please share your configuration if possible? I wasn't able to reproduce this issue.