Skip to content

Commit d27cdb3

Browse files
kvapsclaude
andcommitted
fix(stand): per-cluster registry IP + dedicated image-build target
Parallel stands each get their own /24 hashed off NAME (e.g. e2e1 → 10.16.0.0/24, e2e2 → 10.79.0.0/24, ...). The blockstor-controller and blockstor-satellite manifests hardcoded 10.164.0.1:5000 — that was the bridge IP of a single past cluster, so as soon as a second stand came up the VMs couldn't pull the images (ImagePullBackOff, dial tcp 10.164.0.1:5000: i/o timeout). Switch to a placeholder __REGISTRY__ in the deploy YAMLs; install- blockstor.sh now reads any node's InternalIP, derives the bridge gateway (.1 of the /24), and substitutes the placeholder before applying. up.sh's Talos registry-trust patch follows the same pattern so containerd accepts plain HTTP to that mirror. Also add stand/build-images.sh + `make build-images` so the controller / satellite images can be rebuilt and pushed to localhost:5000 in a single step — needed whenever the source moves (e.g. the cmd/controller path change in fab5137 left the registry image with the stale ENTRYPOINT ["/manager"]). Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
1 parent fab5137 commit d27cdb3

6 files changed

Lines changed: 59 additions & 14 deletions

File tree

stand/Makefile

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ KUBECONFIG := $(WORK_DIR)/kubeconfig
1313
export TALOSCONFIG
1414
export KUBECONFIG
1515

16-
.PHONY: stand-help up down reset piraeus oracle blockstor pools smoke smoke-blockstor burnin-blockstor e2e e2e-list use list stand-clean
16+
.PHONY: stand-help up down reset piraeus oracle build-images blockstor pools smoke smoke-blockstor burnin-blockstor e2e e2e-list use list stand-clean
1717

1818
stand-help: ## list dev-stand targets
1919
@echo "stand targets (use NAME=<cluster>):"
@@ -34,6 +34,9 @@ piraeus: ## install piraeus-operator + linstor-csi (NAME=<n>)
3434
oracle: ## install Java LINSTOR controller as in-cluster oracle (NAME=<n>)
3535
@./stand/install-oracle.sh "$(WORK_DIR)"
3636

37+
build-images: ## build + push blockstor/blockstor-satellite to localhost:5000 (shared across all stands)
38+
@./stand/build-images.sh
39+
3740
blockstor: ## install blockstor CRDs + controller + satellites (NAME=<n>)
3841
@./stand/install-blockstor.sh "$(WORK_DIR)"
3942

stand/blockstor-deploy.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ spec:
7070
serviceAccountName: blockstor-controller
7171
containers:
7272
- name: manager
73-
image: 10.164.0.1:5000/blockstor:dev
73+
image: __REGISTRY__/blockstor:dev
7474
imagePullPolicy: Always
7575
args:
7676
- --rest-bind-address=:3370

stand/blockstor-satellite-daemonset.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ spec:
5050
- {key: node-role.kubernetes.io/control-plane, operator: DoesNotExist}
5151
containers:
5252
- name: satellite
53-
image: 10.164.0.1:5000/blockstor-satellite:dev
53+
image: __REGISTRY__/blockstor-satellite:dev
5454
imagePullPolicy: Always
5555
args:
5656
# drbd's /etc/drbd.conf includes /etc/drbd.d/*.res, so the

stand/build-images.sh

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/usr/bin/env bash
2+
#
3+
# usage: build-images.sh [REGISTRY]
4+
#
5+
# Build the controller + satellite container images from the current
6+
# checkout and push them to the host's local Docker registry. The
7+
# blockstor-deploy.yaml / blockstor-satellite-daemonset.yaml on the
8+
# stand pull these via the per-cluster bridge gateway (see
9+
# install-blockstor.sh) but the underlying registry is a single
10+
# host-local one — so we push once and every parallel stand sees it.
11+
12+
set -euo pipefail
13+
14+
REGISTRY=${1:-localhost:5000}
15+
REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
16+
cd "$REPO_ROOT"
17+
18+
echo ">> docker build $REGISTRY/blockstor:dev (controller stage)"
19+
docker build --target controller -t "$REGISTRY/blockstor:dev" .
20+
21+
echo ">> docker build $REGISTRY/blockstor-satellite:dev (satellite stage)"
22+
docker build --target satellite -t "$REGISTRY/blockstor-satellite:dev" .
23+
24+
echo ">> docker push $REGISTRY/blockstor:dev"
25+
docker push "$REGISTRY/blockstor:dev"
26+
27+
echo ">> docker push $REGISTRY/blockstor-satellite:dev"
28+
docker push "$REGISTRY/blockstor-satellite:dev"
29+
30+
echo ">> images ready"

stand/install-blockstor.sh

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,13 @@
1313
# 3. apply blockstor-satellite-daemonset.yaml
1414
# 4. wait for controller + satellites Running
1515
#
16-
# Assumes the host registry on 10.164.0.1:5000 is reachable from the
17-
# cluster (Talos config-patch trusts http for that mirror — see
18-
# stand/up.sh).
16+
# Assumes the host registry is reachable from the cluster on the
17+
# bridge gateway (.1 of this cluster's NET_CIDR). Talos config-patch
18+
# trusts http for that mirror — see stand/up.sh. The deploy manifests
19+
# carry an `__REGISTRY__` placeholder which this script substitutes
20+
# with the actual bridge IP, computed from the first node's
21+
# InternalIP — this is how parallel stands on the same host all see
22+
# the same registry-on-host without colliding on a single IP.
1923

2024
set -euo pipefail
2125

@@ -24,14 +28,20 @@ export KUBECONFIG="$WORK_DIR/kubeconfig"
2428

2529
REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
2630

31+
# Bridge gateway = .1 of the cluster CIDR. Read it off any node's
32+
# InternalIP (Talos VMs all live in the same /24 as the host bridge).
33+
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
34+
REGISTRY="${NODE_IP%.*}.1:5000"
35+
echo ">> using host registry at $REGISTRY"
36+
2737
echo ">> apply CRDs"
2838
kubectl apply -f "$REPO_ROOT/config/crd/bases/" 2>&1 | tail -5
2939

3040
echo ">> apply controller + RBAC"
31-
kubectl apply -f "$REPO_ROOT/stand/blockstor-deploy.yaml" 2>&1 | tail -5
41+
sed "s|__REGISTRY__|$REGISTRY|g" "$REPO_ROOT/stand/blockstor-deploy.yaml" | kubectl apply -f - 2>&1 | tail -5
3242

3343
echo ">> apply satellite DaemonSet"
34-
kubectl apply -f "$REPO_ROOT/stand/blockstor-satellite-daemonset.yaml" 2>&1 | tail -5
44+
sed "s|__REGISTRY__|$REGISTRY|g" "$REPO_ROOT/stand/blockstor-satellite-daemonset.yaml" | kubectl apply -f - 2>&1 | tail -5
3545

3646
echo ">> wait for controller Running"
3747
kubectl -n blockstor-system rollout status deploy/blockstor-controller --timeout=120s

stand/up.sh

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -106,15 +106,17 @@ machine:
106106
- name: dm_thin_pool
107107
- name: dm_snapshot
108108
- name: dm_crypt
109-
# Trust the host-side Docker registry on 10.164.0.1:5000 via plain
110-
# HTTP. The blockstor-controller / blockstor-satellite images we
111-
# build live there; without this patch containerd refuses to pull
112-
# them ("http: server gave HTTP response to HTTPS client").
109+
# Trust the host-side Docker registry on the bridge gateway (.1 of
110+
# this cluster's NET_CIDR) via plain HTTP. The blockstor-controller
111+
# / blockstor-satellite images we build live there; without this
112+
# patch containerd refuses to pull them ("http: server gave HTTP
113+
# response to HTTPS client"). The bridge IP varies per cluster
114+
# because parallel stands each get their own /24, hashed off NAME.
113115
registries:
114116
mirrors:
115-
"10.164.0.1:5000":
117+
"${NET_CIDR%.*}.1:5000":
116118
endpoints:
117-
- "http://10.164.0.1:5000"
119+
- "http://${NET_CIDR%.*}.1:5000"
118120
skipFallback: true
119121
YAML
120122

0 commit comments

Comments
 (0)