Skip to content

Commit 7eb9c07

Browse files
zdover23zdover23
authored
Merge branch 'main' into docs-2026-05-21-rook-ceph-install-why-rook
2 parents 8f448ee + abd9e78 commit 7eb9c07

3 files changed

Lines changed: 343 additions & 7 deletions

File tree

‎docs/architecture/cloud-storage/rook-ceph.md‎

Lines changed: 326 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,326 @@
1+
---
2+
title: Installing Rook-Ceph on Kubernetes
3+
---
4+
5+
# Installing Rook-Ceph on Kubernetes
6+
7+
## Overview
8+
9+
This guide provides step-by-step instructions for deploying a Ceph storage
10+
cluster using the Rook operator on Kubernetes. Rook automates the deployment,
11+
configuration, and management of Ceph clusters within Kubernetes environments.
12+
13+
The instructions here are meant only as a general guideline. We recommend that
14+
you use the instructions found in the [official Rook
15+
documentation](https://rook.io/docs/rook/latest/) and the [upstream Ceph
16+
documentation](https://docs.ceph.com/).
17+
18+
19+
## Prerequisites
20+
21+
Before beginning the installation, ensure the following requirements are met:
22+
23+
### Kubernetes Cluster Requirements
24+
25+
- Kubernetes v1.25 or higher
26+
- `kubectl` configured to communicate with your cluster
27+
- Administrator access to the Kubernetes cluster
28+
- At least 3 worker nodes for a production cluster (1 node minimum for testing)
29+
- Verify compatibility between your Kubernetes version and the Rook version you
30+
  intend to deploy — see the [Rook releases page](https://github.com/rook/rook/releases)
31+
  for version compatibility information
32+
33+
### Storage Requirements
34+
35+
- Raw block devices available on worker nodes (unformatted, no filesystem)
36+
- Minimum 10 GB of storage per OSD
37+
- Devices should not be mounted or in use by the operating system
38+
39+
### Network Requirements
40+
41+
- Network connectivity between all cluster nodes
42+
- Network access between pods is handled by the Kubernetes network plugin (CNI).
43+
  Ensure your CNI supports the required pod-to-pod communication. If you need
44+
  to open ports for external access to Ceph services, the typical ports are
45+
  6789, 3300, and 6800-7300.
46+
47+
### System Requirements
48+
49+
- Linux kernel 4.5 or higher (5.x recommended)
50+
- LVM2 packages installed on all nodes
51+
- Minimum 2 GB RAM per node (4 GB+ recommended)
52+
- `helm` installed if using Helm-based deployment (optional)
53+
54+
## Configuration Options
55+
56+
### Customizing the Cluster
57+
58+
Edit `cluster.yaml` to customize your deployment before creating the cluster:
59+
60+
#### Storage Configuration
61+
62+
Specify which devices to use for OSDs:
63+
64+
```yaml
65+
storage:
66+
  useAllNodes: true
67+
  useAllDevices: false
68+
  deviceFilter: "^sd[b-z]"  # Use sdb, sdc, etc.
69+
```
70+
71+
Or specify devices explicitly:
72+
73+
```yaml
74+
storage:
75+
  nodes:
76+
  - name: "node1"
77+
    devices:
78+
    - name: "/dev/sdb"
79+
  - name: "node2"
80+
    devices:
81+
    - name: "/dev/sdc"
82+
```
83+
84+
#### Resource Limits
85+
86+
Set resource limits for Ceph daemons:
87+
88+
```yaml
89+
resources:
90+
  mon:
91+
    limits:
92+
      cpu: "2000m"
93+
      memory: "4Gi"
94+
    requests:
95+
      cpu: "1000m"
96+
      memory: "2Gi"
97+
  osd:
98+
    limits:
99+
      cpu: "2000m"
100+
      memory: "4Gi"
101+
    requests:
102+
      cpu: "1000m"
103+
      memory: "2Gi"
104+
```
105+
106+
#### Network Configuration
107+
108+
Configure network settings for client and cluster traffic:
109+
110+
```yaml
111+
network:
112+
  provider: host  # or multus for advanced networking
113+
  # Uncomment for dual network configuration
114+
  # connections:
115+
  #   encryption:
116+
  #     enabled: true
117+
```
118+
119+
### Dashboard Access
120+
121+
Enable and access the Ceph dashboard:
122+
123+
```bash
124+
# The dashboard is enabled by default in cluster.yaml
125+
126+
# Get the dashboard password
127+
kubectl -n rook-ceph get secret rook-ceph-dashboard-password \
128+
  -o jsonpath="{['data']['password']}" | base64 --decode && echo
129+
130+
# Port-forward to access the dashboard
131+
kubectl -n rook-ceph port-forward service/rook-ceph-mgr-dashboard 8443:8443
132+
```
133+
134+
Access the dashboard at: `https://localhost:8443`
135+
136+
Username: `admin`
137+
Password: (from the command above)
138+
139+
## Creating Storage Classes
140+
141+
### Block Storage (RBD)
142+
143+
Create a storage class for block devices:
144+
145+
```bash
146+
kubectl create -f csi/rbd/storageclass.yaml
147+
```
148+
149+
Test the storage class:
150+
151+
```bash
152+
# Create a test PVC
153+
cat <<EOF | kubectl apply -f -
154+
apiVersion: v1
155+
kind: PersistentVolumeClaim
156+
metadata:
157+
  name: rbd-pvc
158+
spec:
159+
  accessModes:
160+
  - ReadWriteOnce
161+
  resources:
162+
    requests:
163+
      storage: 1Gi
164+
  storageClassName: rook-ceph-block
165+
EOF
166+
167+
# Verify PVC is bound
168+
kubectl get pvc rbd-pvc
169+
```
170+
171+
### File Storage (CephFS)
172+
173+
Deploy the CephFS filesystem:
174+
175+
```bash
176+
kubectl create -f filesystem.yaml
177+
```
178+
179+
Create a storage class for shared filesystem:
180+
181+
```bash
182+
kubectl create -f csi/cephfs/storageclass.yaml
183+
```
184+
185+
### Object Storage (RGW)
186+
187+
Deploy the object storage service:
188+
189+
```bash
190+
kubectl create -f object.yaml
191+
```
192+
193+
Wait for the RGW pods to be ready:
194+
195+
```bash
196+
kubectl -n rook-ceph get pods -l app=rook-ceph-rgw
197+
```
198+
199+
## Verification
200+
201+
### Verify All Storage Types
202+
203+
Check that all storage components are operational:
204+
205+
```bash
206+
# Check block storage
207+
kubectl get storageclass rook-ceph-block
208+
209+
# Check filesystem storage
210+
kubectl get storageclass rook-cephfs
211+
212+
# Check object storage
213+
kubectl -n rook-ceph get cephobjectstore
214+
```
215+
216+
### Test Storage Functionality
217+
218+
Create test workloads using each storage type:
219+
220+
```bash
221+
# Test RBD block storage
222+
kubectl create -f csi/rbd/pvc.yaml
223+
kubectl create -f csi/rbd/pod.yaml
224+
225+
# Test CephFS
226+
kubectl create -f csi/cephfs/pvc.yaml
227+
kubectl create -f csi/cephfs/pod.yaml
228+
```
229+
230+
## Troubleshooting
231+
232+
### Common Issues
233+
234+
**Operator not starting:**
235+
236+
```bash
237+
# Check operator logs
238+
kubectl -n rook-ceph logs -l app=rook-ceph-operator
239+
```
240+
241+
**OSDs not starting:**
242+
243+
```bash
244+
# Check OSD prepare logs
245+
kubectl -n rook-ceph logs -l app=rook-ceph-osd-prepare
246+
247+
# Verify devices are available and unused
248+
kubectl -n rook-ceph exec -it deployment/rook-ceph-tools -- ceph-volume inventory
249+
```
250+
251+
**Cluster stuck in HEALTH_WARN:**
252+
253+
```bash
254+
# Check detailed cluster status
255+
kubectl -n rook-ceph exec -it deployment/rook-ceph-tools -- ceph health detail
256+
257+
# Check for common issues
258+
kubectl -n rook-ceph exec -it deployment/rook-ceph-tools -- ceph -s
259+
```
260+
261+
## Cleanup
262+
263+
To remove the Rook-Ceph cluster:
264+
265+
**Note:** Rook uses Kubernetes finalizers to protect resources from accidental
266+
deletion. If `kubectl delete` commands hang, you may need to manually remove
267+
finalizers from the relevant custom resources. See the
268+
[Rook cleanup documentation](https://rook.io/docs/rook/latest/Storage-Configuration/ceph-teardown/)
269+
for details.
270+
271+
```bash
272+
# Delete the cluster
273+
kubectl delete -f cluster.yaml
274+
275+
# Delete object storage (if created)
276+
kubectl delete -f object.yaml
277+
278+
# Delete filesystem (if created)
279+
kubectl delete -f filesystem.yaml
280+
281+
# Delete the operator
282+
kubectl delete -f operator.yaml
283+
kubectl delete -f common.yaml
284+
kubectl delete -f crds.yaml
285+
```
286+
287+
**Cleaning up storage on nodes (CAUTION: This deletes all data):**
288+
289+
Run the following on each node that had OSDs. In addition to removing the Rook
290+
data directory, the raw block devices used by OSDs must be wiped before they
291+
can be reused:
292+
293+
```bash
294+
# Remove Rook data directory
295+
sudo rm -rf /var/lib/rook
296+
297+
# Wipe each OSD device (replace /dev/sdX with the actual device name)
298+
sudo sgdisk --zap-all /dev/sdX
299+
```
300+
301+
## Next Steps
302+
303+
After successful installation:
304+
305+
1. Configure monitoring with Prometheus and Grafana
306+
2. Set up backup and disaster recovery procedures
307+
3. Implement resource quotas and limits
308+
4. Configure advanced networking if required
309+
5. Review and adjust Ceph configuration parameters
310+
6. Set up regular maintenance schedules
311+
312+
## Additional Resources
313+
314+
- Official Rook documentation: https://rook.io/docs/rook/latest/
315+
- Ceph documentation: https://docs.ceph.com/
316+
- Rook GitHub repository: https://github.com/rook/rook
317+
- Rook Slack community: https://rook-io.slack.com/
318+
319+
## Notes
320+
321+
- This guide provides a basic Rook-Ceph deployment. While the prerequisites
322+
  describe a production-grade setup, additional considerations apply for
323+
  production environments, including high availability, performance tuning,
324+
  and security hardening.
325+
- Always test deployment procedures in a non-production environment first.
326+
- Keep Rook and Ceph versions updated for security and stability improvements.

‎docs/architecture/index.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ CobaltCore is built on top of OpenStack and IronCore, leveraging their capabilit
1414
- [**HA Service**](./cluster#ha-service): The high availability service that ensures critical workloads remain operational even in the event of failures.
1515
- [**Cortex**](./cortex): Smart initial placement and scheduling service for compute, storage, and network in cloud-native cloud environments.
1616
- [**Cloud Storage**](./cloud-storage/): Ceph-based distributed storage stack including Rook, Chorus, Arbiter, and Prysm for lifecycle management, replication, quorum, and observability.
17-
- [**Ceph**](.cloud-storage/ceph): An all-in-one storage system that provides object, block, and file storage and delivers extraordinary scalability.
18-
- [**Rook-Ceph Installation**](./rook-ceph-install): A procedure for deploying
19-
the all-in-one storage system that provides object, block, and file storage
20-
and delivers extraordinary scalability.
17+
- [**Ceph**](./cloud-storage/ceph): An all-in-one storage system that provides object, block, and file storage and delivers extraordinary scalability.
18+
- [**Rook-Ceph Installation**](./cloud-storage/rook-ceph.md): A procedure for
19+
deploying the all-in-one storage system that provides object, block, and file
20+
storage and delivers extraordinary scalability.

‎docs/changelog/index.md‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,18 @@ title: Changelog
44

55
# Change Log
66

7-
| Date | Change |
8-
| :--- | :--- |
9-
| 2026-04-28 | Change log created |
7+
| Date | Change | git hash |
8+
| :--- | :--- | |
9+
| 2025-11-26 | Add ceph.md, an overview page that explains the function and purpose of Ceph | a01c4435 |
10+
| 2026-03-03 | Add a page that explains how to use Rook, a tool for deploying Ceph | 499a3a76 |
11+
| 2026-03-04 | Add documentation for Prysm, a CLI tool for real-time monitoring of RGW, Ceph storage clusters, and various hardware components. | deeb3ea8 |
12+
| 2026-03-12 | Add documentation for Arbiter, a tool that deploys external arbiters in Rook-provisioned clusters in an effort to avoid split-brain scenarios (scenarios in which each site thinks that it's the primary) | 26891975 |
13+
| 2026-03-13 | Add documentation for Chorus, data replication software that supports S3 and OpenStack Swift APIs and enables zero-downtime migration between storage systems| 36ed4417 |
14+
| 2026-03-23 | Add Architectural information to ceph.md, including sections on data storage and organization, eliminating centralization, the cluster map, high availability, authentication, security, smart daemons, hyperscale, dynamic cluster management, and client interfaces (RGB, RGW, CephFS). | 4021e59d |
15+
| 2026-03-26 | Add RBD information to the architecture page, including an explanation of blocks, an explanation of the fundamentals of Ceph block devices, differences in the caching behavior of kernel modules and of applications that rely on the librd library, the value of having a unified storage cluster, thin provisioning, resizable Ceph block devices, data striping across multiple OSDs, integration with RADOS, snapshotting capabilities, and the configurable nature of Ceph's replication factor. | f977c63a |
16+
| 2026-03-31 | Add a summary of RGW to the Architecture page, including an explanation of RGW as a RESTful gateway to store objects and metadata in a Ceph cluster, an explanation of the operation of RGW (as a FastCGI or standalone HTTP service sitting atop the Ceph Storage Cluster), an explanation of RGW's dual API compatibility (with both Amazon S3 and OpenStack Swift), an explanation of the operations supported by the S3-compatible API, an explanation of the operations supported by the Swift-compatible API, an explanation of how RGW can allow multiple independent users to share the same Ceph cluster while maintaining isolation, an explanation of the bucket-based model RGW uses to organize data for S3 compatibility, an explanation of RGW's use of RADOS pools to organize different types of data internally, explanations of advanced features such as object versioning and server-side encryption and cross-origin resource sharing, explanations of RGW's support of horizontal scaling and caching of frequently-accessed objects (to reduce latency), explanations of multi-site replication, asynchronous execution of time-consuming tasks such as garbage collection and data synchronization, explanation of byte-range requests (for large-file retrieval), and explanation of monitoring | 628dfd0c |
17+
| 2026-03-31 | Add a summary of CephFS to the Architecture page, explaining data and metadata management, ceph-mds, POSIX-compliance, methods of accessing CephFS (kernel client, FUSE, libcephfs), snapshots, quotas, support for multiple filesystems, access control (ACLs), and path-based access restrictions | fd962649 |
18+
| 2026-04-28 | Change log created | 953b13ff |
19+
1020

1121
![Bundesministerium für Wirtschaft und Energie (BMWE)-EU funding logo](https://apeirora.eu/assets/img/BMWK-EU.png "apeirora_logo")

0 commit comments

Comments
 (0)