Skip to content

Commit 818aaba

Browse files
Madhu-1Madhu-1
authored
Merge pull request #533 from nixpanic/docs/images-configmap
docs: add documentation for custom container images ConfigMap
2 parents baa0679 + 2d7b8ce commit 818aaba

1 file changed

Lines changed: 248 additions & 0 deletions

File tree

‎docs/custom-images.md‎

Lines changed: 248 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,248 @@
1+
# Customizing Container Images
2+
3+
The Ceph-CSI Operator allows you to customize the container images used by the CSI drivers through a ConfigMap-based image set configuration. This is particularly useful for:
4+
5+
- **Disconnected/Air-gapped deployments**: Deploy from a private registry without internet access
6+
- **Custom image builds**: Use your own builds of the CSI components
7+
- **Version pinning**: Lock specific component versions for stability
8+
- **Registry mirroring**: Use a local mirror of public registries
9+
10+
## Available Image Keys
11+
12+
The operator uses the following image keys in the ConfigMap. Each key corresponds to a specific container in the CSI driver deployment:
13+
14+
| Key | Description | Default Image |
15+
|-----|-------------|---------------|
16+
| `provisioner` | CSI external-provisioner sidecar | `registry.k8s.io/sig-storage/csi-provisioner:v6.2.0` |
17+
| `attacher` | CSI external-attacher sidecar | `registry.k8s.io/sig-storage/csi-attacher:v4.12.0` |
18+
| `resizer` | CSI external-resizer sidecar | `registry.k8s.io/sig-storage/csi-resizer:v2.1.0` |
19+
| `snapshotter` | CSI external-snapshotter sidecar | `registry.k8s.io/sig-storage/csi-snapshotter:v8.5.0` |
20+
| `registrar` | CSI node-driver-registrar sidecar | `registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.17.0` |
21+
| `snapshot-metadata` | CSI snapshot-metadata sidecar (RBD only) | `registry.k8s.io/sig-storage/csi-snapshot-metadata:v1.0.0` |
22+
| `plugin` | Ceph-CSI driver plugin | `quay.io/cephcsi/cephcsi:v3.17.0` |
23+
| `addons` | CSI-Addons sidecar | `quay.io/csiaddons/k8s-sidecar:v0.14.0` |
24+
| `ex-snapshotter` | Extended snapshotter for CephFS volume groups (optional) | Not set by default |
25+
26+
> **Note**: You only need to specify the images you want to override. Any keys not provided in your ConfigMap will use the default images. The image references shown in the example ConfigMaps above are examples only; use the latest available or supported images for your environment.
27+
28+
## Configuration Levels
29+
30+
The operator supports image customization at two levels:
31+
32+
1. **Global (OperatorConfig)**: Apply custom images to all drivers managed by the operator
33+
2. **Per-Driver**: Override images for a specific driver instance
34+
35+
When both are configured, the per-driver configuration takes precedence.
36+
37+
## Global Image Configuration
38+
39+
To apply custom images to all drivers, create a ConfigMap and reference it in the `OperatorConfig`:
40+
41+
### Step 1: Create the Image Set ConfigMap
42+
43+
```yaml
44+
apiVersion: v1
45+
kind: ConfigMap
46+
metadata:
47+
name: custom-csi-images
48+
namespace: ceph-csi-operator-system
49+
data:
50+
# Override only the images you need to customize
51+
plugin: "my-registry.example.com/cephcsi/cephcsi:v3.17.0"
52+
provisioner: "my-registry.example.com/sig-storage/csi-provisioner:v6.2.0"
53+
attacher: "my-registry.example.com/sig-storage/csi-attacher:v4.12.0"
54+
# ... add other images as needed
55+
```
56+
57+
### Step 2: Reference in OperatorConfig
58+
59+
```yaml
60+
apiVersion: csi.ceph.io/v1
61+
kind: OperatorConfig
62+
metadata:
63+
name: ceph-csi-operator-config
64+
namespace: ceph-csi-operator-system
65+
spec:
66+
driverSpecDefaults:
67+
imageSet:
68+
name: custom-csi-images
69+
```
70+
71+
## Per-Driver Image Configuration
72+
73+
To customize images for a specific driver, create a ConfigMap in the driver's namespace and reference it in the Driver CR:
74+
75+
### Step 1: Create the Image Set ConfigMap
76+
77+
```yaml
78+
apiVersion: v1
79+
kind: ConfigMap
80+
metadata:
81+
name: rbd-custom-images
82+
namespace: ceph-csi-operator-system
83+
data:
84+
plugin: "my-registry.example.com/cephcsi/cephcsi:v3.17.0-custom"
85+
snapshotter: "my-registry.example.com/sig-storage/csi-snapshotter:v8.5.0"
86+
```
87+
88+
### Step 2: Reference in Driver CR
89+
90+
```yaml
91+
apiVersion: csi.ceph.io/v1
92+
kind: Driver
93+
metadata:
94+
name: rbd.csi.ceph.com
95+
namespace: ceph-csi-operator-system
96+
spec:
97+
imageSet:
98+
name: rbd-custom-images
99+
```
100+
101+
## Example: Disconnected Deployment
102+
103+
For air-gapped or disconnected environments, you need to mirror all required images to your private registry:
104+
105+
### Step 1: Mirror Images to Private Registry
106+
107+
```bash
108+
# Example using skopeo to mirror images
109+
PRIVATE_REGISTRY="registry.internal.example.com"
110+
111+
# Mirror all required images
112+
skopeo copy docker://registry.k8s.io/sig-storage/csi-provisioner:v6.2.0 \
113+
docker://${PRIVATE_REGISTRY}/sig-storage/csi-provisioner:v6.2.0
114+
115+
skopeo copy docker://registry.k8s.io/sig-storage/csi-attacher:v4.12.0 \
116+
docker://${PRIVATE_REGISTRY}/sig-storage/csi-attacher:v4.12.0
117+
118+
skopeo copy docker://registry.k8s.io/sig-storage/csi-resizer:v2.1.0 \
119+
docker://${PRIVATE_REGISTRY}/sig-storage/csi-resizer:v2.1.0
120+
121+
skopeo copy docker://registry.k8s.io/sig-storage/csi-snapshotter:v8.5.0 \
122+
docker://${PRIVATE_REGISTRY}/sig-storage/csi-snapshotter:v8.5.0
123+
124+
skopeo copy docker://registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.17.0 \
125+
docker://${PRIVATE_REGISTRY}/sig-storage/csi-node-driver-registrar:v2.17.0
126+
127+
skopeo copy docker://registry.k8s.io/sig-storage/csi-snapshot-metadata:v1.0.0 \
128+
docker://${PRIVATE_REGISTRY}/sig-storage/csi-snapshot-metadata:v1.0.0
129+
130+
skopeo copy docker://quay.io/cephcsi/cephcsi:v3.17.0 \
131+
docker://${PRIVATE_REGISTRY}/cephcsi/cephcsi:v3.17.0
132+
133+
skopeo copy docker://quay.io/csiaddons/k8s-sidecar:v0.14.0 \
134+
docker://${PRIVATE_REGISTRY}/csiaddons/k8s-sidecar:v0.14.0
135+
```
136+
137+
### Step 2: Create ConfigMap with Private Registry URLs
138+
139+
```yaml
140+
apiVersion: v1
141+
kind: ConfigMap
142+
metadata:
143+
name: disconnected-images
144+
namespace: ceph-csi-operator-system
145+
data:
146+
provisioner: "registry.internal.example.com/sig-storage/csi-provisioner:v6.2.0"
147+
attacher: "registry.internal.example.com/sig-storage/csi-attacher:v4.12.0"
148+
resizer: "registry.internal.example.com/sig-storage/csi-resizer:v2.1.0"
149+
snapshotter: "registry.internal.example.com/sig-storage/csi-snapshotter:v8.5.0"
150+
registrar: "registry.internal.example.com/sig-storage/csi-node-driver-registrar:v2.17.0"
151+
snapshot-metadata: "registry.internal.example.com/sig-storage/csi-snapshot-metadata:v1.0.0"
152+
plugin: "registry.internal.example.com/cephcsi/cephcsi:v3.17.0"
153+
addons: "registry.internal.example.com/csiaddons/k8s-sidecar:v0.14.0"
154+
```
155+
156+
### Step 3: Apply to OperatorConfig
157+
158+
```yaml
159+
apiVersion: csi.ceph.io/v1
160+
kind: OperatorConfig
161+
metadata:
162+
name: ceph-csi-operator-config
163+
namespace: ceph-csi-operator-system
164+
spec:
165+
driverSpecDefaults:
166+
imageSet:
167+
name: disconnected-images
168+
```
169+
170+
## Image Pull Secrets
171+
172+
If your private registry requires authentication, configure image pull secrets:
173+
174+
### Step 1: Create Image Pull Secret
175+
176+
```bash
177+
kubectl create secret docker-registry private-registry-secret \
178+
--docker-server=registry.internal.example.com \
179+
--docker-username=<username> \
180+
--docker-password=<password> \
181+
--docker-email=<email> \
182+
-n ceph-csi-operator-system
183+
```
184+
185+
### Step 2: Reference in Driver CR
186+
187+
```yaml
188+
apiVersion: csi.ceph.io/v1
189+
kind: Driver
190+
metadata:
191+
name: rbd.csi.ceph.com
192+
namespace: ceph-csi-operator-system
193+
spec:
194+
imageSet:
195+
name: disconnected-images
196+
controllerPlugin:
197+
imagePullSecrets:
198+
- name: private-registry-secret
199+
nodePlugin:
200+
imagePullSecrets:
201+
- name: private-registry-secret
202+
```
203+
204+
## Verification
205+
206+
After applying your custom image configuration, verify that the correct images are being used:
207+
208+
```bash
209+
# Check controller plugin deployment
210+
kubectl get deployment -n ceph-csi-operator-system -o yaml | grep image:
211+
212+
# Check node plugin daemonset
213+
kubectl get daemonset -n ceph-csi-operator-system -o yaml | grep image:
214+
```
215+
216+
## Troubleshooting
217+
218+
### Images Not Updating
219+
220+
If your custom images are not being applied:
221+
222+
1. Verify the ConfigMap exists in the correct namespace:
223+
```bash
224+
kubectl get configmap <configmap-name> -n <namespace>
225+
```
226+
227+
2. Check the OperatorConfig or Driver CR references the correct ConfigMap name:
228+
```bash
229+
kubectl get operatorconfig ceph-csi-operator-config -o yaml
230+
kubectl get driver <driver-name> -o yaml
231+
```
232+
233+
3. Check operator logs for image loading errors:
234+
```bash
235+
kubectl logs -n ceph-csi-operator-system deployment/ceph-csi-operator-controller-manager
236+
```
237+
238+
### Image Pull Errors
239+
240+
If pods fail to pull images:
241+
242+
1. Verify the image URLs are correct and accessible from your cluster
243+
2. Check if image pull secrets are properly configured
244+
3. Verify network connectivity to your registry
245+
4. Check pod events for specific error messages:
246+
```bash
247+
kubectl describe pod <pod-name> -n ceph-csi-operator-system
248+
```

0 commit comments

Comments
 (0)