Skip to content

Commit b92bc59

Browse files
privateipprivateip
committed
chore: update containerlab deploy for galactic-router
- Replace galactic-agent with galactic-router container image - Replace BGPInstance/BGPPeer CRDs with BGPRouter/BGPPeer/BGPAdvertisement - Replace infra cluster with dfw cluster (three-region: dfw, iad, sjc) - Replace infra route reflector with iad-worker-rr node - Remove cosmos operator deployment from containerlab - Update NAD configs to use galacticRouter instead of gobgp - Add BGP CRD patches to fix ASN maximum for kubebuilder v0.18.0 - Update all documentation, Taskfile, and scripts accordingly
1 parent efc8cb5 commit b92bc59

33 files changed

Lines changed: 324 additions & 302 deletions

‎AGENTS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,8 @@ Summary:
5050

5151
## Deployments
5252

53-
- **`deploy/galactic-router/`** — Kustomize manifests for the router DaemonSet, RBAC, and ServiceAccount. Apply with `kubectl apply -k deploy/galactic-router/`.
54-
- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (iad, sjc, infra) wired over an IPv6 SRv6 transit mesh. FRR runs as a hostNetwork DaemonSet on each worker for eBGP underlay; `galactic-router` (tenant role) handles EVPN path distribution over iBGP. See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands.
53+
- **`deploy/galactic-router/`** — Production manifests for the router DaemonSet, RBAC, and ServiceAccount. Apply with `kubectl apply -f deploy/galactic-router/`.
54+
- **`deploy/containerlab/`** — ContainerLab topology (`gvpc.clab.yaml`) for three Kind clusters (dfw, iad, sjc) wired over an IPv6 SRv6 transit mesh. FRR runs as a hostNetwork DaemonSet on each worker for eBGP underlay; `galactic-router` (tenant role) handles EVPN path distribution over iBGP. See `deploy/containerlab/README.md` and `deploy/containerlab/Taskfile.yaml` for bring-up commands.
5555

5656
## New Developer Entry Points
5757

‎deploy/containerlab/README.md‎

Lines changed: 75 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -1,52 +1,56 @@
11
# Galactic VPC Lab Deployment
22

3-
Three Kind clusters connected over an IPv6 SRv6 transit mesh. Each cluster runs FRR
4-
as a node routing daemon (hostNetwork DaemonSet) to peer with the transit layer via
5-
BGP unnumbered. GoBGP runs alongside FRR on the iad and sjc workers to exchange
6-
L3VPN type-5 routes with the infra route reflector over iBGP.
3+
Three Kind clusters (dfw, iad, sjc) connected over an IPv6 SRv6 transit mesh. Each cluster
4+
runs FRR as a node routing daemon (hostNetwork DaemonSet) to peer with the transit layer via
5+
BGP unnumbered. galactic-router runs alongside FRR on the workers to distribute EVPN routes
6+
over iBGP to the route reflector on iad-rr.
77

88
## Topology
99

1010
```
11-
iad-worker ──eth1── tr1 ──────────── tr2 ──eth1── sjc-worker
11+
dfw-worker ──eth1── tr1 ──────────── tr2 ──eth1── sjc-worker
1212
│ ╲ ╱ │
1313
│ tr3 ── tr4 │
1414
│ ╱ ╲ │
1515
(mesh) (mesh)
16-
tr3 ──eth5── infra-worker
16+
tr3 ──eth5── iad-worker
17+
tr3 ──eth4── iad-worker-rr
1718
```
1819

1920
### Node roles
2021

2122
| Node | Kind | Role |
2223
|-----------------------|---------------|---------------------------------------------------|
24+
| `dfw` | k8s-kind | Kind cluster definition (dfw region) |
25+
| `dfw-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus, cert-mgr |
26+
| `dfw-worker` | ext-container | Kind worker; runs FRR PE + galactic-router PE |
2327
| `iad` | k8s-kind | Kind cluster definition (iad region) |
2428
| `iad-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus, cert-mgr |
25-
| `iad-worker` | ext-container | Kind worker; runs FRR PE + GoBGP PE |
29+
| `iad-worker` | ext-container | Kind worker; runs FRR PE + galactic-router PE |
30+
| `iad-worker-rr` | ext-container | Kind worker; runs FRR PE + galactic-router RR |
2631
| `sjc` | k8s-kind | Kind cluster definition (sjc region) |
2732
| `sjc-control-plane` | ext-container | Kind control-plane; runs Cilium, Multus, cert-mgr |
28-
| `sjc-worker` | ext-container | Kind worker; runs FRR PE + GoBGP PE |
29-
| `infra` | k8s-kind | Kind cluster definition (infra) |
30-
| `infra-control-plane` | ext-container | Kind control-plane; runs Cilium |
31-
| `infra-worker` | ext-container | Kind worker; runs FRR route reflector |
33+
| `sjc-worker` | ext-container | Kind worker; runs FRR PE + galactic-router PE |
3234
| `tr1`–`tr4` | linux (FRR) | iBGP full mesh, AS 65100 |
3335

3436
### BGP design
3537

3638
```
37-
AS 65000 (iad-underlay / FRR) ──eBGP unnumbered── tr1 (AS 65100)
38-
AS 65000 (sjc-underlay / FRR) ──eBGP unnumbered── tr2 (AS 65100)
39-
AS 65000 (infra-control-plane / FRR) ──eBGP unnumbered── tr3 (AS 65100)
40-
41-
AS 65000 (iad-overlay / GoBGP) ──iBGP── infra-control-plane (AS 65000 RR)
42-
AS 65000 (sjc-overlay / GoBGP) ──iBGP── infra-control-plane (AS 65000 RR)
39+
AS 65000 (dfw-underlay / FRR) ──eBGP unnumbered── tr1 (AS 65100)
40+
AS 65000 (iad-underlay / FRR) ──eBGP unnumbered── tr3:eth5 (AS 65100)
41+
AS 65000 (iad-rr-underlay / FRR) ──eBGP unnumbered── tr3:eth4 (AS 65100)
42+
AS 65000 (sjc-underlay / FRR) ──eBGP unnumbered── tr2 (AS 65100)
43+
44+
AS 65000 (dfw-overlay / galactic-router) ──iBGP── iad-rr (AS 65000 RR)
45+
AS 65000 (iad-overlay / galactic-router) ──iBGP── iad-rr (AS 65000 RR)
46+
AS 65000 (sjc-overlay / galactic-router) ──iBGP── iad-rr (AS 65000 RR)
4347
```
4448

45-
- All clusters use a single AS (65000) for both the FRR underlay and the GoBGP overlay.
49+
- All clusters use a single AS (65000) for both the FRR underlay and the galactic-router overlay.
4650
- The transit mesh carries IPv6 unicast (SRv6 locator prefixes and loopbacks) via iBGP within AS 65100.
4751
- FRR PE nodes originate their SRv6 forwarding prefix (`2001:db8:ffXX::/48`) and SRv6 SID block (`fc00:0:X::/48`) toward the transit layer via eBGP unnumbered.
4852
- `allowas-in 1` is configured on all cluster FRR instances so each site accepts prefixes that carry AS 65000 in the path — necessary because the transit reflects routes from one AS 65000 site to another.
49-
- GoBGP instances on iad/sjc workers peer with infra-control-plane over iBGP (AS 65000) for `l3vpn-ipv4-unicast` (type-5 VPN routes). GoBGP runs with `port = -1`; FRR owns TCP/179.
53+
- galactic-router instances on dfw/iad/sjc workers peer with iad-worker-rr over iBGP (AS 65000) for `l2vpn-evpn` routes. GoBGP runs with outbound-only mode (`listenPort=-1`); all BGP sessions are initiated outbound.
5054

5155
## Addressing
5256

@@ -72,38 +76,41 @@ AS 65000 (sjc-overlay / GoBGP) ──iBGP── infra-control-plane (AS 65000
7276

7377
### Worker–TR links (BGP unnumbered, link-local only)
7478

75-
| Link | TR interface |
76-
|--------------------|--------------|
77-
| iad-worker – tr1 | eth1 |
78-
| sjc-worker – tr2 | eth1 |
79-
| infra-worker – tr3 | eth5 |
79+
| Link | TR interface |
80+
|------------------------|--------------|
81+
| dfw-worker – tr1 | eth1 |
82+
| sjc-worker – tr2 | eth1 |
83+
| iad-worker – tr3 | eth5 |
84+
| iad-worker-rr – tr3 | eth4 |
8085

8186
### Cluster SRv6 addressing
8287

83-
| Cluster | FRR loopback / SID block | SRv6 forwarding prefix | GoBGP local-address |
84-
|---------|--------------------------|------------------------|---------------------|
85-
| iad | fc00:0:2::1/48 | 2001:db8:ff01::/48 | fc00:0:2::1 |
86-
| sjc | fc00:0:3::1/48 | 2001:db8:ff02::/48 | fc00:0:3::1 |
87-
| infra | fc00:0:4::1/128 | — | — |
88+
| Cluster | FRR loopback / SID block | SRv6 forwarding prefix | galactic-router address |
89+
|-----------|--------------------------|------------------------|-------------------------|
90+
| dfw | fc00:0:2::1/48 | 2001:db8:ff01::/48 | fc00:0:2::1 |
91+
| iad | fc00:0:4::1/48 | 2001:db8:ff03::/48 | fc00:0:4::1 |
92+
| iad-rr | fc00:0:8::1/48 | — | fc00:0:8::1 |
93+
| sjc | fc00:0:3::1/48 | 2001:db8:ff02::/48 | fc00:0:3::1 |
8894

8995
Worker SRv6 node SIDs (on `lo-galactic`):
9096

91-
| Node | Address |
92-
|--------------|--------------------------------------------|
93-
| iad-worker | 2001:db8:ff01:100:ffff:ffff:ffff:ffff/128 |
94-
| sjc-worker | 2001:db8:ff02:100:ffff:ffff:ffff:ffff/128 |
95-
| infra-worker | 2001:db8:ff03:100:ffff:ffff:ffff:ffff/128 |
97+
| Node | Address |
98+
|---------------|--------------------------------------------|
99+
| dfw-worker | 2001:db8:ff01:100:ffff:ffff:ffff:ffff/128 |
100+
| iad-worker | 2001:db8:ff03:100:ffff:ffff:ffff:ffff/128 |
101+
| sjc-worker | 2001:db8:ff02:100:ffff:ffff:ffff:ffff/128 |
96102

97103
### Management network (172.20.20.0/24)
98104

99105
| Node | Address |
100106
|-----------------------|---------------|
101-
| iad | 172.20.20.101 |
102-
| iad-control-plane | 172.20.20.102 |
103-
| iad-worker | 172.20.20.103 |
104-
| infra | 172.20.20.111 |
105-
| infra-control-plane | 172.20.20.112 |
106-
| infra-worker | 172.20.20.113 |
107+
| dfw | 172.20.20.101 |
108+
| dfw-control-plane | 172.20.20.102 |
109+
| dfw-worker | 172.20.20.103 |
110+
| iad | 172.20.20.111 |
111+
| iad-control-plane | 172.20.20.112 |
112+
| iad-worker | 172.20.20.113 |
113+
| iad-worker-rr | 172.20.20.114 |
107114
| sjc | 172.20.20.121 |
108115
| sjc-control-plane | 172.20.20.122 |
109116
| sjc-worker | 172.20.20.123 |
@@ -116,16 +123,16 @@ deploy/containerlab/
116123
├── Taskfile.yaml
117124
├── containers/
118125
│ ├── kindest-node-galactic/ # Custom Kind node image (Cilium, Multus, cert-manager, galactic)
119-
│ ├── gobgp/ # GoBGP container built from upstream release binary
126+
│ ├── galactic-router/ # galactic-router container built from Go source
120127
│ └── frr/ # FRR container built from Alpine edge
121128
├── resources/
122-
│ ├── underlay/ # FRR DaemonSet kustomize overlays (iad, sjc, infra)
123-
│ ├── overlay/ # GoBGP DaemonSet kustomize overlays (iad, sjc)
124-
│ └── cosmos/ # Cosmos BGP CRs (BGPInstance, BGPSession, BGPProvider)
129+
│ ├── underlay/ # FRR DaemonSet kustomize overlays (dfw, iad, iad-rr, sjc)
130+
│ ├── overlay/ # galactic-router DaemonSet kustomize overlays (dfw, iad, sjc)
131+
│ └── bgp/ # BGP CRs (BGPRouter, BGPPeer, BGPAdvertisement)
125132
├── node_files/
133+
│ ├── dfw/ config.yaml
126134
│ ├── iad/ config.yaml
127135
│ ├── sjc/ config.yaml
128-
│ ├── infra/ config.yaml
129136
│ ├── tr1/ frr.conf startup.sh
130137
│ ├── tr2/ frr.conf startup.sh
131138
│ ├── tr3/ frr.conf startup.sh
@@ -141,7 +148,7 @@ deploy/containerlab/
141148

142149
## Prerequisites
143150

144-
- ContainerLab ≥ 0.54
151+
- ContainerLab >= 0.54
145152
- Docker
146153
- `kind` CLI
147154
- Host kernel with SRv6 support
@@ -165,16 +172,15 @@ task deploy
165172

166173
| Task | Description |
167174
|--------------------|----------------------------------------------------------------|
168-
| `build` | Build all container images (node, cosmos, gobgp, frr) |
175+
| `build` | Build all container images (node, galactic-router, frr) |
169176
| `build:node` | Build the custom `kindest/node:galactic` image |
170-
| `build:cosmos` | Build the Cosmos BGP operator image from source |
171-
| `build:gobgp` | Build the GoBGP container from upstream release binary |
177+
| `build:galactic-router` | Build the galactic-router container from Go source |
172178
| `build:frr` | Build the FRR container from Alpine edge |
173179
| `deploy` | Build images, apply host sysctls, and deploy the lab |
174180
| `deploy:topology` | Deploy the ContainerLab topology (transit routers + clusters) |
175-
| `deploy:images` | Load images into Kind clusters and wait for cosmos rollout |
176-
| `deploy:underlay` | Apply FRR DaemonSets to all three clusters |
177-
| `deploy:overlay` | Apply GoBGP DaemonSets and Cosmos BGP CRs to iad and sjc |
181+
| `deploy:images` | Load container images into Kind clusters |
182+
| `deploy:underlay` | Apply FRR DaemonSets to all clusters |
183+
| `deploy:overlay` | Apply galactic-router DaemonSets and BGP CRs |
178184
| `destroy` | Destroy the lab and remove all Kind clusters |
179185
| `reload` | Full rebuild — destroy then redeploy |
180186
| `inspect` | Show running nodes and management addresses |
@@ -194,35 +200,36 @@ docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast summary"
194200
# Worker SRv6 prefixes should be present on all TR nodes
195201
docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff01::/48"
196202
docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff02::/48"
203+
docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff03::/48"
197204
```
198205

199206
### FRR DaemonSets (eBGP underlay)
200207

201208
```bash
202209
# Check pods are running
203-
docker exec iad-control-plane kubectl get pods -n iad-underlay
204-
docker exec sjc-control-plane kubectl get pods -n sjc-underlay
205-
docker exec infra-control-plane kubectl get pods -n infra-underlay
210+
docker exec dfw-control-plane kubectl get pods -n galactic-system
211+
docker exec iad-control-plane kubectl get pods -n galactic-system
212+
docker exec sjc-control-plane kubectl get pods -n galactic-system
206213

207214
# Run vtysh inside a pod
208-
docker exec iad-control-plane kubectl exec -n iad-underlay ds/iad-underlay \
215+
docker exec iad-control-plane kubectl exec -n galactic-system ds/iad-underlay \
216+
-- vtysh -c "show bgp ipv6 unicast summary"
217+
docker exec iad-control-plane kubectl exec -n galactic-system ds/iad-rr-underlay \
209218
-- vtysh -c "show bgp ipv6 unicast summary"
210219
```
211220

212-
### GoBGP DaemonSets (L3VPN overlay)
221+
### galactic-router DaemonSets (EVPN overlay)
213222

214223
```bash
215224
# Check pods are running
216-
docker exec iad-control-plane kubectl get pods -n iad-overlay
217-
docker exec sjc-control-plane kubectl get pods -n sjc-overlay
218-
219-
# Check iBGP session to infra-control-plane
220-
docker exec iad-control-plane kubectl exec -n iad-overlay ds/iad-overlay -- gobgp neighbor
221-
docker exec sjc-control-plane kubectl exec -n sjc-overlay ds/sjc-overlay -- gobgp neighbor
222-
223-
# Inspect VPN RIB
224-
docker exec iad-control-plane kubectl exec -n iad-overlay ds/iad-overlay -- gobgp global rib -a vpnv6
225-
docker exec sjc-control-plane kubectl exec -n sjc-overlay ds/sjc-overlay -- gobgp global rib -a vpnv6
225+
docker exec dfw-control-plane kubectl get pods -n galactic-system
226+
docker exec iad-control-plane kubectl get pods -n galactic-system
227+
docker exec sjc-control-plane kubectl get pods -n galactic-system
228+
229+
# Check EVPN routes via BGPRouter status
230+
docker exec dfw-control-plane kubectl get bgprouters -A
231+
docker exec iad-control-plane kubectl get bgprouters -A
232+
docker exec sjc-control-plane kubectl get bgprouters -A
226233
```
227234

228235
## Notes
@@ -234,4 +241,4 @@ docker exec sjc-control-plane kubectl exec -n sjc-overlay ds/sjc-overlay -- gobg
234241
configured on worker data-plane interfaces.
235242
- Cilium's iptables rules block BGP by default; the bootstrap script inserts
236243
`ip6tables -I INPUT` rules for TCP/179 before Cilium starts on each worker.
237-
- infra-control-plane peers with tr3 as AS 65000, the same AS used by all three clusters.
244+
- iad-worker-rr peers with tr3 as AS 65000, the same AS used by all three clusters.

‎deploy/containerlab/Taskfile.yaml‎

Lines changed: 8 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@ vars:
77
sh: echo *.clab.yaml
88
FRR_VERSION: "10.6.1"
99
FRR_IMAGE: frr:{{.FRR_VERSION}}
10-
COSMOS_IMAGE: cosmos:latest
1110

1211
tasks:
1312
default:
@@ -18,22 +17,12 @@ tasks:
1817
build:
1918
desc: Build all container images
2019
cmds:
21-
- task: "clone:cosmos"
2220
- task: "build:node"
2321
- task: "build:frr"
2422
- task: "build:galactic-router"
25-
- task: "build:cosmos"
26-
27-
"clone:cosmos":
28-
desc: Clone the cosmos source tree needed for go mod replace and image build
29-
status:
30-
- test -d build/cosmos
31-
cmds:
32-
- git clone --depth=1 https://github.com/milo-os/cosmos build/cosmos
3323

3424
"build:node":
3525
desc: Build the Kind node image with the galactic CNI plugin
36-
deps: ["clone:cosmos"]
3726
cmds:
3827
- docker build --network=host -t kindest/node:galactic -f containers/kindest-node-galactic/Dockerfile ../..
3928

@@ -47,13 +36,7 @@ tasks:
4736
"build:galactic-router":
4837
desc: Build the galactic-router container image
4938
cmds:
50-
- docker build --network=host -t galactic-router:latest -f containers/galactic/Dockerfile ../..
51-
52-
"build:cosmos":
53-
desc: Build the cosmos operator container image from the local clone
54-
deps: ["clone:cosmos"]
55-
cmds:
56-
- docker build --network=host -t {{.COSMOS_IMAGE}} -f build/cosmos/build/Dockerfile build/cosmos
39+
- docker build --network=host -t galactic-router:latest -f containers/galactic-router/Dockerfile ../..
5740

5841
deploy:
5942
desc: Build images and deploy the full lab end-to-end
@@ -113,14 +96,6 @@ tasks:
11396
vars: {IMAGE: galactic-router:latest, NODE: sjc-worker}
11497
- task: load-image
11598
vars: {IMAGE: galactic-router:latest, NODE: dfw-worker}
116-
- task: load-image
117-
vars: {IMAGE: "{{.COSMOS_IMAGE}}", NODE: iad-worker}
118-
- task: load-image
119-
vars: {IMAGE: "{{.COSMOS_IMAGE}}", NODE: iad-worker-rr}
120-
- task: load-image
121-
vars: {IMAGE: "{{.COSMOS_IMAGE}}", NODE: sjc-worker}
122-
- task: load-image
123-
vars: {IMAGE: "{{.COSMOS_IMAGE}}", NODE: dfw-worker}
12499

125100
destroy:
126101
desc: Destroy the lab
@@ -157,7 +132,7 @@ tasks:
157132
- ./scripts/install-underlay.sh
158133

159134
"deploy:overlay":
160-
desc: Install the galactic-router overlay DaemonSets and cosmos operator
135+
desc: Install the galactic-router overlay DaemonSets
161136
cmds:
162137
- ./scripts/install-overlay.sh
163138

@@ -179,8 +154,12 @@ tasks:
179154
done
180155
181156
"test:bgp-underlay":
182-
desc: Verify underlay BGP sessions on iad and sjc workers
157+
desc: Verify underlay BGP sessions on dfw, iad, and sjc workers
183158
cmds:
159+
- |
160+
docker exec dfw-control-plane \
161+
kubectl exec -n galactic-system ds/dfw-underlay \
162+
-- vtysh -c "show bgp ipv6 unicast summary"
184163
- |
185164
docker exec iad-control-plane \
186165
kubectl exec -n galactic-system ds/iad-underlay \
@@ -198,7 +177,7 @@ tasks:
198177
- docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff03::/48"
199178

200179
"test:evpn":
201-
desc: Verify EVPN BGP routes on iad, sjc, and dfw via cosmos BGPRouter status
180+
desc: Verify EVPN BGP routes on iad, sjc, and dfw via BGPRouter status
202181
cmds:
203182
- docker exec iad-control-plane kubectl get bgprouters -A
204183
- docker exec sjc-control-plane kubectl get bgprouters -A
@@ -210,6 +189,5 @@ tasks:
210189
- task: destroy
211190
- docker rmi kindest/node:galactic || true
212191
- docker rmi galactic-router:latest || true
213-
- docker rmi {{.COSMOS_IMAGE}} || true
214192
- docker rmi {{.FRR_IMAGE}} || true
215193
- rm -rf clab-{{.LAB}} build/

‎deploy/containerlab/containers/galactic-agent/Dockerfile‎

Lines changed: 0 additions & 9 deletions
This file was deleted.

0 commit comments

Comments
 (0)