doc: revise io isolation runbook from end-to-end validation

Walked through the runbook on a 2-node CRI-O 1.31 + kubeadm cluster with
OpenEBS LVM-LocalPV and StackGres 1.18.6.

Main correctness fixes from the validation:

- Poll for `fio` binary after `kubectl wait --for=condition=Ready`.
  The Ready signal flips when the container starts, not when the
  in-container `apt-get install fio` finishes; subsequent fio
  invocations otherwise fail with "command not found".

- Callout for the one-shot device glob expansion. CRI-O resolves
  `Devices:` globs once, at startup, against currently-existing block
  devices. On a fresh node with no PVCs, `/dev/dm-*` matches nothing,
  the class loads with empty parameters, and annotated pods get no
  throttle (silently).

- Verification now tells the reader to grep for
  `WARN: device wildcard ... does not match any device nodes` in the
  runtime journal. That message is the canary for the empty-class
  failure above.

- Expected output reflects how `kubectl describe` renders extended
  resources (`150k`, not `150000`); added a `jsonpath` recipe for the
  exact integer.

- cgroup-path discovery rewritten to be runtime/cgroup-driver
  agnostic. The previous fixed path landed at the *outer* systemd scope
  (`crio-<id>.scope/io.max`) which is empty under modern CRI-O + crun + systemd
  cgroup driver (the throttle is on the inner `.../container/io.max`).

- Clarified that `clusterPods` under `spec.metadata.annotations`
  is the StackGres convention that propagates annotations onto the
  cluster pods (this is how the BlockIO class selection reaches the
  runtime).

- Framing callout near the top: StackGres 1.19 (upcoming) will offer
  an in-cluster, no-node-prep alternative that needs host-cgroup
  access from the cluster pods (blocked by OpenShift SCC in many
  environments). Names the trade-off honestly so readers can pick.

- Fixed broken anchor `#433-enable-blockio-in-cri-o`.

New "Operational considerations" section:

- Add a matrix of what triggers a node reboot in day-2 operations and what
  doesn't.

- Behavior under contention from uncapped workloads. The cap is a hard
  upper bound but only a soft lower bound; achieving the cap requires
  the device not to be saturated by I/O the scheduler doesn't know
  about. Recommends dedicating database nodes (label + nodeSelector +
  taint) as the cleanest pattern.

New "Limitations and caveats" section:

- One-shot device glob expansion, with the "seed generously at node
  provisioning" mitigation (pre-provision e.g. 32 placeholder PVCs per
  node, restart the runtime).

- Footgun call-out: pin the placeholder Pod with `nodeSelector`, not
  `nodeName`. `WaitForFirstConsumer` storage classes need the scheduler
  to bind the PVC; a Pod with `spec.nodeName` set directly bypasses
  the scheduler and the PVC stays Pending forever.

- conmon and cgroup-path variations. The path where the throttle lands
  depends on `cgroup_manager` and `monitor_cgroup`; recommends the
  path-agnostic find-based diagnostic.

- Static provisioning as a robust alternative. For workloads where the
  placeholder-seeding approach isn't a fit (unbounded pod count,
  discomfort with kernel dm-minor reuse semantics), pre-create fixed
  LVs and expose them as static PVs. Each LV's dm-N is fixed at
  creation time; the BlockIO class binds deterministically. Trade-off
  is losing dynamic expansion.

- ZFS-backed PVs (moved here from Troubleshooting): write throttling is
  ineffective on ZFS due to buffered-write cgroup attribution; read
  throttling still works.

Capacity-planning guidance:

- Why 4K and not Postgres' 8K page size: `blk-throttle` counts BIOs,
  4K is the kernel's atomic unit and matches vendor IOPS specs; the
  cap is in BIOs/sec regardless of each BIO's size. Optional 8K
  cross-check noted but not required.

- Fio vs pgbench/production replay: fio is the right tool for drive
  characterization (kernel-level BIOs/sec); pgbench (or replay) has a
  different role --validating that the chosen tier comfortably fits
  the actual workload's tps target once the cap is set. Two different
  questions at two different stages.

- Tier-design guidance: 3-5 classes is the practical range; values
  should span an order of magnitude; the highest tier should sit well
  below the node's safe ceiling so multiple high-tier pods can coexist.

Editorial:

- StorageClass in example renamed `topolvm-nvme` -> `local-lvm-nvme`
  with an explicit "substitute the name your provisioner exposes" note
  --the previous name was confusingly TopoLVM-flavored while the
  callout already documents OpenEBS LVM-LocalPV and LVMS as equivalent.

- Removed duplicated ZFS troubleshooting entry now that it lives in
  Limitations.

Author: ahachete