docs: add YAML spec and health checks documentation (#105)

teran · web-flow · commit 4e65c6b0862e · 2026-07-14T04:04:10.000+03:00
- Add Declarative configuration format section to README with CephConfig
and CephOSDConfig kind descriptions and field tables
- Add Health checks section documenting all 14 indicators with
thresholds and descriptions
- Update examples/config.yaml with multi-doc example including
CephOSDConfig and inline comments
- Fix copy-paste bug in UncleanPGs GoDoc comment

---------

Signed-off-by: Igor Shishkin &lt;me@teran.dev&gt;
diff --git a/README.md b/README.md
@@ -5,19 +5,20 @@
 [![Go Reference](https://pkg.go.dev/badge/github.com/runityru/cephctl.svg)](https://pkg.go.dev/github.com/runityru/cephctl)
 
 Small utility to control Ceph cluster configuration just like any other declarative
-    configuration
+configuration
 
 ## Main features
 
 * Easy-to-use healthcheck which may contain checks against status & configuration
-    and indicate some some not trivial issues
+  and indicate some some not trivial issues
 * Declarative configuration support which is apply only if needed
 * Diff configuration: check what the difference between currently running configuration
-    and desired or migrated from other cluster
+  and desired or migrated from other cluster
 
 ## Usage
 
 <!-- markdownlint-disable MD013 -->
+
 ```shell
 $ ./cephctl
 usage: cephctl [<flags>] <command> [<args> ...]
@@ -56,6 +57,7 @@ version
     Print version and exit
 
 ```
+
 <!-- markdownlint-enable MD013 -->
 
 ## How it works
@@ -64,20 +66,148 @@ Cephctl uses native Ceph CLIs to work with cluster configuration so it's require
 to have Ceph binaries w/ configured `ceph.conf`. Alternatively it's possible
 to adjust `ceph` binary path to access ceph in container and/or remote machine.
 
+## Declarative configuration format
+
+cephctl uses a declarative YAML specification to describe the desired Ceph cluster
+configuration. The file can contain one or more configuration documents separated
+by `---`.
+
+Each document has two mandatory fields:
+
+* `kind` — defines the type of configuration section (see below)
+* `spec` — the actual configuration payload
+
+### kind: CephConfig
+
+Maps to `ceph config set` commands. The spec is a nested map of configuration
+sections and their key-value pairs:
+
+```yaml
+kind: CephConfig
+spec:
+  <section>:
+    <key>: "<value>"
+```
+
+* **section** — any valid Ceph configuration section (e.g., `global`, `mon`,
+  `osd`, `client.radosgw`, `mgr`, etc.)
+* **key** — any valid Ceph configuration parameter within that section
+* **value** — the value as a string (YAML strings, quoted or unquoted)
+
+Example:
+
+```yaml
+kind: CephConfig
+spec:
+  global:
+    rbd_cache: "true"
+    osd_pool_default_size: 3
+  client.radosgw:
+    rgw_cache_lru_size: 100
+    rgw_crypt_require_ssl: "true"
+  osd:
+    rocksdb_perf: "true"
+```
+
+### kind: CephOSDConfig
+
+Maps to `ceph osd set-*` commands. The spec defines OSD-level operational
+parameters:
+
+```yaml
+kind: CephOSDConfig
+spec:
+  allow_crimson: <bool>
+  backfillfull_ratio: <float>
+  full_ratio: <float>
+  nearfull_ratio: <float>
+  require_min_compat_client: <string>
+```
+
+Available fields and their defaults:
+
+| Field                       | Type   | Default | Description                                                                                              |
+| --------------------------- | ------ | ------- | -------------------------------------------------------------------------------------------------------- |
+| `allow_crimson`             | bool   | `false` | Enable experimental Crimson OSD backend (risky for production)                                           |
+| `backfillfull_ratio`        | float  | `0.9`   | OSD considered full for backfill purposes at this ratio                                                  |
+| `full_ratio`                | float  | `0.95`  | OSD considered full and blocks writes at this ratio                                                      |
+| `nearfull_ratio`            | float  | `0.85`  | OSD considered near-full at this ratio                                                                   |
+| `require_min_compat_client` | string | `reef`  | Minimum allowed client version (`luminous`, `nautilus`, `octopus`, `pacific`, `quincy`, `reef`, `squid`) |
+
+### Multi-document example
+
+Both kinds can be combined in a single file using the `---` separator:
+
+```yaml
+---
+kind: CephConfig
+spec:
+  global:
+    rbd_cache: "true"
+  client.radosgw:
+    rgw_cache_lru_size: 100
+---
+kind: CephOSDConfig
+spec:
+  allow_crimson: false
+  backfillfull_ratio: 0.9
+  full_ratio: 0.95
+  nearfull_ratio: 0.85
+  require_min_compat_client: reef
+```
+
+## Health checks
+
+`cephctl healthcheck` runs the following checks and reports the status for each
+indicator. Each check is reported as **GOOD**, **AT_RISK**, **DANGEROUS**, or
+**UNKNOWN**.
+
+| Indicator                 | Type             | GOOD               | AT_RISK           | DANGEROUS        | Description                                                  |
+| ------------------------- | ---------------- | ------------------ | ----------------- | ---------------- | ------------------------------------------------------------ |
+| `CLUSTER_STATUS`          | overall          | `HEALTH_OK`        | `HEALTH_WARN`     | `HEALTH_ERR`     | Overall cluster health status from `ceph status`             |
+| `QUORUM`                  | monitors         | all mons in quorum | some mons missing | —                | Whether all monitor nodes are participating in quorum        |
+| `MON_DOWN`                | monitors         | 0                  | >0                | —                | Count of monitor nodes that are not up                       |
+| `OSD_DOWN`                | OSD              | 0                  | >0                | —                | Count of OSDs in down state                                  |
+| `OSD_OUT`                 | OSD              | 0                  | >0                | —                | Count of OSDs in out state                                   |
+| `DOWN_PGS`                | placement groups | 0                  | —                 | >0               | PGs stored on down OSDs with no available copy               |
+| `UNCLEAN_PGS`             | placement groups | 0                  | >0                | —                | PGs not in clean state (e.g., recovering, backfilling)       |
+| `INACTIVE_PGS`            | placement groups | 0                  | —                 | >0               | PGs that cannot perform IO (inactive)                        |
+| `IP_COLLISION`            | networking       | no collisions      | —                 | collisions found | Duplicate front or back IP addresses across OSD hosts        |
+| `MUTES_AMOUNT`            | health           | 0                  | >0                | —                | Muted health checks that could mask real issues              |
+| `OSD_METADATA_SIZE`       | storage          | ≤7%                | >15%              | >20%             | OSD metadata (block.db) size as percentage of total capacity |
+| `OSD_NUM_DAEMON_VERSIONS` | versions         | 1 version          | 2 versions        | >2 versions      | Number of distinct OSD daemon versions running               |
+| `ALLOW_CRIMSON`           | OSD              | disabled           | enabled           | —                | Whether experimental Crimson OSD is allowed                  |
+| `DEVICE_HEALTH_WEAROUT`   | hardware         | no worn devices    | wear >50%         | wear >75%        | SSD/NVMe devices with high wear level                        |
+
+### Using health checks for monitoring
+
+You can run health checks against a remote cluster via SSH:
+
+```shell
+cephctl --ceph-binary='ssh user@mon01 ceph' healthcheck
+```
+
+Or by setting the environment variable:
+
+```shell
+export CEPHCTL_CEPH_BINARY='ssh user@mon01 ceph'
+cephctl healthcheck
+```
+
 ## Roadmap
 
-* [X] v0.0.0
-  * [X] Apply declarative configuration for `ceph config`
-  * [X] Dump cluster configuration to CephConfig specification
-  * [X] Diff configuration against running configuration for `ceph config`
-  * [X] Perform healthcheck based on current cluster status
-  * [X] Add healthchecks based on current cluster configuration
-* [X] v0.1.0
-  * [X] Additional healthchecks based on hardware status
-  * [X] FreeBSD support in builds
-  * [X] Remote Ceph cluster access via SSH
-* [X] v0.2.0
-  * [X] Apply/Dump declarative configuration for `ceph osd set-*` stuff
+* [x] v0.0.0
+  * [x] Apply declarative configuration for `ceph config`
+  * [x] Dump cluster configuration to CephConfig specification
+  * [x] Diff configuration against running configuration for `ceph config`
+  * [x] Perform healthcheck based on current cluster status
+  * [x] Add healthchecks based on current cluster configuration
+* [x] v0.1.0
+  * [x] Additional healthchecks based on hardware status
+  * [x] FreeBSD support in builds
+  * [x] Remote Ceph cluster access via SSH
+* [x] v0.2.0
+  * [x] Apply/Dump declarative configuration for `ceph osd set-*` stuff
 * [ ] v0.3.0
   * [ ] Apply/Dump declarative configuration for Ceph Object Gateway (rgw)
 * [ ] v0.4.0
@@ -96,11 +226,11 @@ If you gonna use cephctl as a library for your purposes please feel free to
 but please note a few things:
 
 1. Cephctl doesn't use internal packages to allow you to do whatever you like.
-    Cephctl project doesn't aim to limit your usage.
+   Cephctl project doesn't aim to limit your usage.
 2. Internal program interfaces are not guaranteed to be stable between releases
-    since they're written and serve for internal purposes.
+   since they're written and serve for internal purposes.
 3. CLI interface (until 1.0.x at least) is also not guaranteed to be stable:
-    subcommands and options are subjects to change between versions.
+   subcommands and options are subjects to change between versions.
 
 ## Installation
 
@@ -168,9 +298,11 @@ goreleaser build --snapshot --clean
 or manually via Go compiler
 
 <!-- markdownlint-disable MD013 -->
+
 ```shell
 go build -v -ldflags="-X 'main.appVersion=$(git rev-parse --short HEAD) (trunk build)' -X 'main.buildTimestamp=$(date -u +%Y-%m-%dT%H:%m:%SZ)'" -o dist/cephctl ./cmd/cephctl/...
 ```
+
 <!-- markdownlint-enable MD013 -->
 
 ## Contribution
diff --git a/examples/config.yaml b/examples/config.yaml
@@ -1,10 +1,31 @@
 ---
+# CephConfig: maps to `ceph config set` commands
+# Sections: global, mon, osd, mgr, client.radosgw, etc.
 kind: CephConfig
 spec:
   global:
+    # Enable RBD caching for improved performance
     rbd_cache: "true"
+    # Default replication size
+    osd_pool_default_size: "3"
   client.radosgw:
-    rgw_cache_lru_size: 100
+    rgw_cache_lru_size: "100"
+    # Require SSL for RadosGW encryption operations
     rgw_crypt_require_ssl: "true"
   osd:
     rocksdb_perf: "true"
+---
+# CephOSDConfig: maps to `ceph osd set-*` commands
+kind: CephOSDConfig
+spec:
+  # Crimson is experimental - keep disabled for production
+  allow_crimson: false
+  # OSDs at 90% capacity prevent backfill to this OSD
+  backfillfull_ratio: 0.9
+  # OSDs at 95% capacity block all writes
+  full_ratio: 0.95
+  # OSDs at 85% capacity trigger near-full warning
+  nearfull_ratio: 0.85
+  # Minimum client version allowed to connect (luminous, nautilus,
+  # octopus, pacific, quincy, reef, squid)
+  require_min_compat_client: reef
diff --git a/models/clusterhealth.go b/models/clusterhealth.go
@@ -143,8 +143,9 @@ const (
 
 	// ClusterHealthIndicatorTypeUncleanPGs reflects amount of PGs which are not in clean state
 	//
-	// Description: Inactive PGs indicator shows how many PGs are inactive i.e. can not be
-	// 	used to perform IO operations at the moment.
+	// Description: Unclean PGs indicator shows how many PGs are not in clean state i.e.
+	// 	they are in states like recovering, backfilling, peering, etc. This is normal
+	// 	during maintenance operations but may indicate issues if persistent.
 	//
 	// Ref: https://docs.ceph.com/en/latest/rados/operations/monitoring-osd-pg/#monitoring-pg-states
 	//
