feat(dist_reuse): KV cache sharing across TP/PP/CP + single-node radix match

Squashes 5 commits (20a65d5 + d97db4d + 72632ec + b9230c6 + 7cd04ad)
into a single landed feature.  This is the full dist_reuse stack on
top of PR #165 (RankInfo refactor), validated end-to-end on a 2-machine
GPU setup (gpu-146.56.224.46 master / gpu-129.211.162.213 peer):

  S1 (single-node TP=1)        cached_ratio 99.65%   PASS
  S2 (single-node TP=2)        cached_ratio 99.65%   PASS
  S3 (cross-node TP=1)         master cold->warm 99.63%, peer
                               crosshit 99.63% storage=272 backend=
                               FlexKVConnector  (PEERH2H @ 6.22 GB/s
                               via mooncake/RDMA)   PASS x3
  357/357 unit tests on both nodes                  PASS

== Original commits (in chronological order) ==

[20a65d5] feat(dist_reuse): KV cache sharing across TP/PP/CP + single-
node radix match
  Initial dist_reuse stack: master coordinator, sharing-domain key,
  aggregate radix, redis-meta namespace, multi-node policy, P2P
  transfer types (PEERH2H/H2PEERH/PEERSSD2H/H2PEERSSD), failure
  detector, four S{1..4} sglang+FlexKV e2e benchmark scripts.

[d97db4d] fix(dist_reuse): unblock cross-instance KV cache sharing on
s3_cross_node_tp1
  Three runtime bugs blocked the s3 (master prime / peer crosshit) flow:
    1) GPUCPUTransferWorker._transfer_impl had positional-arg drift
       on the transfer_kv_blocks pybind: C++ added 'start_layer_id'
       between 'chunk_size_in_bytes' and 'num_layers' (transfer.cu
       2025-07-10), which silently mapped is_h2d=False onto
       transfer_num_cta and launched D2H kernels with gridDim(0)
       -> cudaErrorInvalidConfiguration on every D2H.
       Fix: bind every value to the C++ pybind name with kwargs and
       add 'start_layer_id=0' explicitly.
    2) GlobalCacheEngine._maybe_attach_multi_sd_peerh2h_ops carried a
       dead 'layer_num' parameter which the only caller in
       _get_impl_local passed undefined -> NameError on first
       cross-instance reuse hit.  Fix: drop the dead parameter and 6
       call sites in tests/test_d3_filter_and_get_clones.py.
    3) merge_to_batch_graph raised NotImplementedError on PEERH2H /
       H2PEERH / PEERSSD2H / H2PEERSSD as soon as a real cross-instance
       hit produced a P2P op.  Fix: whitelist the four types as P2P
       passthrough (preserves per-block src_block_node_ids and
       per-op target_node_ids from D-3 multi-SD broadcast clones),
       wire dependencies on merged_h2d_op (GET) / merged_d2h_op (PUT).

[72632ec] fix(memory_handle): propagate _import_tensor_handle exceptions
  Previously _import_tensor_handle logged the error and returned
  torch.empty(0) on import failure, which silently dropped the wrapper
  into a 0-element tensor and surfaced as an unrelated IndexError later
  in worker.py::_get_layer_ptrs (layer_blocks[lay_id][0] out of range).
  Now always re-raise, keeping the original traceback so cross-node
  CUDA IPC handle device-id mismatches surface at their source.
  Consistent with _import_cuda_ipc_handle which never swallowed.

[b9230c6] fix(config): move tp_node_idx from ModelConfig to RankInfo
  PR #165 removed tp_rank from ModelConfig but
  ModelConfig.tp_node_idx still referenced self.tp_rank, raising
  AttributeError.  Two pre-existing test_cache_config_batch_b.py
  cases failed because of this.
  Fix: remove ModelConfig.tp_node_idx (replaced with a migration
  comment); add RankInfo.tp_node_idx (tp_rank // tp_size_per_node)
  to complement RankInfo.tp_rank_per_node (tp_rank %
  tp_size_per_node); update the two TP-node-count tests to
  construct a RankInfo for tp_node_idx assertions.

[7cd04ad] docs(monitoring): document the new flexkv_py_dist_reuse_*
metrics
  Added user-facing documentation for the 5 cross-instance reuse
  metrics in docs/monitoring/README_{en,zh}.md (kept in sync):
    * \xa72.3 'Cross-instance Reuse Metrics' table with type, labels,
      severity and KNOWN_ISSUE-derived alert thresholds.
    * 'Instrumentation status' subtable that flags the two metrics
      (lease_meta_nullptr_total / about_to_evict_total) whose Python
      collector hooks are ready but whose C++ master-side trigger
      has not yet landed, with a callout that '0' on these two does
      NOT mean 'system healthy'.
    * \xa71.1 environment variable table now documents
      PROMETHEUS_MULTIPROC_DIR (the sample dir used by
      prometheus_client across sglang TP/PP workers, KVManager
      subprocess and transfer workers).
    * \xa73.5 'Multiprocess Scrape Notes' explaining the
      MultiProcessCollector aggregation path and the recommended
      /dev/shm/flexkv_prom tmpfs override.
    * \xa73.6 'Recommended PromQL alerts' section with 4 ready-to-paste
      Prometheus alert rules:
        - FlexKVDistReuseLeaseMetaNullptr (critical, any positive)
        - FlexKVDistReusePeerReadFailureRate (critical, > 0.1%)
        - FlexKVDistReusePeerReadP99High (warning, > 500ms)
        - FlexKVDistReuseEvictPressure (warning, ratio > 10)
    * The /metrics curl verification snippet now also greps
      flexkv_py_dist_reuse_.

Author: phaedonsun

benchmarks/dist_benchmark/TWONODE_DIRECT_README.md

@@ -0,0 +1,125 @@
+# FlexKV Two-Node Direct-Mode e2e — How to Run
+
+This harness validates **distributed KV cache sharing** across two physical
+hosts (146 ↔ 129) **without touching sglang** — it drives FlexKV directly
+via ``KVManager`` + ``KVTPClient`` inside ``benchmark_dist_direct.py``.
+
+## Why direct mode (not ``benchmark_dist_kvcache.py``)
+
+* No ``KVServer`` subprocess → simpler lifecycle, no residual IPC socket at
+  ``/tmp/flexkv_server`` to clean up.
+* Uses the same ``get_match`` / ``put_async`` main path that §2.1 wires
+  through to ``_sharing_domain_gate_get`` and ``_notify_sd_ready_on_put``,
+  so any breakage here is a real breakage.
+* GPU footprint is tiny (~40 MB / GPU) — safe to run alongside a live
+  sglang serving that already owns most of the memory.
+
+## Conflict isolation with the running sglang process
+
+| Resource | sglang (GLM-5-FP8) | this benchmark |
+|---|---|---|
+| Mooncake engine TCP port | **5555** (sglang Transfer) | **5556 on 146** / **5557 on 129** |
+| Redis logical DB | 0 (mooncake keys ``mooncake/*``) | 0 (mooncake) + **DB 1 for flexkv keys** |
+| Redis key prefixes (DB 1) | — | ``sd:*``, ``instance:*``, ``node:*`` … |
+| GPU VRAM | most of it | one GPU, ~40 MB |
+| IPC sockets | ``/tmp/flexkv_server`` | **none** (direct mode) |
+
+So the only shared resources are the physical Redis server (different DB)
+and the RDMA NICs (different QP). Neither overlaps in state.
+
+## Quick checklist before launching
+
+1. **Redis reachable + password OK**:
+   ```bash
+   redis-cli -h 10.206.0.9 -p 6379 -a 123456 PING        # → PONG
+   ```
+2. **DB 1 is clean (first time only)**:
+   ```bash
+   redis-cli -h 10.206.0.9 -p 6379 -a 123456 -n 1 DBSIZE # → (integer) 0
+   # If non-zero and you know it's our leftover state, wipe it:
+   #   redis-cli -h 10.206.0.9 -p 6379 -a 123456 -n 1 FLUSHDB
+   # Do NOT touch DB 0 — mooncake + sglang live there.
+   ```
+3. **Mooncake ports 5556 / 5557 free** on the respective hosts:
+   ```bash
+   ss -ltn | grep -E ':5556|:5557'                       # should be empty
+   ```
+4. **FlexKV built with FLEXKV_ENABLE_P2P=1** on both hosts:
+   ```bash
+   python3 -c 'from flexkv.cache.redis_meta import dist_available; print(dist_available())'
+   # → True
+   ```
+
+## Run
+
+### Step A — on host **146** (10.206.0.9), start **PUT-only**
+
+```bash
+cd /data1/phaedonsun/flexkv/FlexKV
+
+export PYTHONPATH=/data1/phaedonsun/flexkv/FlexKV
+export LD_LIBRARY_PATH=/data1/phaedonsun/flexkv/FlexKV/build/lib
+export CUDA_VISIBLE_DEVICES=0          # pick any single free GPU
+
+python3 benchmarks/dist_benchmark/benchmark_dist_direct.py \
+    --config benchmarks/dist_benchmark/twonode_direct_146.yml \
+    --mode put-only \
+    --batch-size 1 --sequence-length 256 \
+    --seed 42 \
+    --rebuild-interval-ms 20
+```
+
+Expected final line before the process idles:
+```
+Data published to Redis. Press Enter to shutdown (keep running for other nodes to GET)...
+```
+
+### Step B — on host **129** (10.206.0.13), start **GET-only with same seed**
+
+```bash
+cd /data1/phaedonsun/flexkv/FlexKV
+
+export PYTHONPATH=/data1/phaedonsun/flexkv/FlexKV
+export LD_LIBRARY_PATH=/data1/phaedonsun/flexkv/FlexKV/build/lib
+export CUDA_VISIBLE_DEVICES=0          # different physical GPU, same index is fine
+
+python3 benchmarks/dist_benchmark/benchmark_dist_direct.py \
+    --config benchmarks/dist_benchmark/twonode_direct_129.yml \
+    --mode get-only \
+    --batch-size 1 --sequence-length 256 \
+    --seed 42 \
+    --rebuild-interval-ms 20
+```
+
+## Success criteria
+
+In the 129 log look for:
+
+```
+--- GET Phase ---
+  GET: 256/256 tokens, data_size: 0.000 GB, cache_ratio: 100.00% ...
+```
+
+A non-zero ``cache_ratio`` means the 129 instance:
+1. Found the 146 instance via the shared Redis (``instance:*`` discovery)
+2. Resolved the 146 peer SD's ``node_id`` from the aggregate radix
+3. Issued a Mooncake RDMA read against 146's mooncake engine @ 5556
+4. Received KV data that matches byte-for-byte what 146 PUT
+
+If ``cache_ratio: 0.00%``:
+* Check the 129 log for ``[DistReuse]`` lines — the §2.1 gate ruled it out.
+* Check ``KEYS sd:*`` in Redis DB 1 — the 146 side should have published
+  ``sd:<…>:block:<nid>:<hash>`` keys.
+* Check Mooncake connectivity by running the ``transfer_engine_bench``
+  binary between 146:5556 and 129:5557.
+
+## Teardown
+
+* 146: press Ctrl-C in the PUT-only terminal (the Ctrl-C handler calls
+  ``kvmanager.shutdown()`` which releases Mooncake + Redis state).
+* 129: the GET-only run exits on its own; its ``atexit`` hook tears
+  KVManager down.
+* Optionally wipe Redis DB 1 between runs:
+  ```bash
+  redis-cli -h 10.206.0.9 -p 6379 -a 123456 -n 1 FLUSHDB
+  ```

benchmarks/dist_benchmark/benchmark_dist_direct.py

@@ -132,6 +132,14 @@ def load_dist_direct_config(config_path: str):
         user_config.local_ip = config["local_ip"]
     if "redis_password" in config:
         user_config.redis_password = config["redis_password"]
+    # Optional: pick a non-default Redis logical DB so FlexKV keys don't collide
+    # with other tenants (e.g. Mooncake meta, or another running FlexKV /
+    # sglang instance on the same physical Redis).  The flexkv keys (``sd:*``,
+    # ``instance:*``, ``node:*`` …) all live in the selected DB; the mooncake
+    # backend continues to use whatever DB its ``metadata_server`` URL
+    # implies (default 0).
+    if "flexkv_redis_db" in config:
+        user_config.flexkv_redis_db = int(config["flexkv_redis_db"])
 
     # Auto-generate mooncake config JSON and set MOONCAKE_CONFIG_PATH if P2P is enabled
     if config.get("enable_p2p_cpu", False) or config.get("enable_p2p_ssd", False):