Skip to content

Commit eb0cf40

Browse files
committed
design doc: soften language, use neutral framing
Signed-off-by: Dawei Huang <daweihuang@microsoft.com>
1 parent 4940fab commit eb0cf40

1 file changed

Lines changed: 70 additions & 19 deletions

File tree

doc/gnoi-native-grpc-design.md

Lines changed: 70 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2,23 +2,23 @@
22

33
## TL;DR
44

5-
`gnoi_shutdown_daemon` issues gNOI RPCs by shelling out to `docker exec gnmi gnoi_client`. This is fragile, opaque, and already causing silent failures. Replace with direct Python gRPC calls using vendored proto stubs.
5+
`gnoi_shutdown_daemon` issues gNOI RPCs by shelling out to `docker exec gnmi gnoi_client`. This introduces several layers of indirection that make failures hard to diagnose and completion detection unreliable. This document proposes replacing the subprocess path with direct Python gRPC calls using vendored proto stubs.
66

7-
## 1. Why This Pattern Is Bad
7+
## 1. Limitations of the Current Approach
88

9-
| Problem | Impact |
10-
|---------|--------|
11-
| Requires `gnmi` container running and healthy | DPU shutdown silently fails if gnmi is restarting |
12-
| Subprocess + Docker CLI overhead per RPC | Extra process creation, Docker round-trip, stdout capture |
13-
| Output is unstructured text with a header line | Any format change in `gnoi_client` breaks parsing |
14-
| gRPC status codes are lost | Caller only sees `rc != 0` — no code, no details |
15-
| Errors are Go `panic()` stack traces on stderr | Production diagnosis requires SSH + manual docker exec |
16-
| `suppress_stderr=True` discards those panics | Error output goes to `/dev/null`, logs say only "command failed" |
17-
| String matching for completion detection | `"reboot complete" in out_s.lower()` doesn't match actual output (see §2) |
18-
| Tight coupling to CLI flag interface | `-module System -rpc Reboot -jsonin '{...}'` is a serialization layer we don't need |
19-
| Security surface | Shell-out through Docker CLI is wider than a direct gRPC socket |
9+
| Observation | Consequence |
10+
|-------------|-------------|
11+
| Requires NPU `gnmi` container running and healthy | DPU shutdown depends on an unrelated NPU container's availability, even though the RPC target is the DPU's own gnmi server |
12+
| Subprocess + Docker CLI overhead per RPC | Extra process creation, Docker round-trip, stdout capture on each call |
13+
| Output is unstructured text with a header line | Parsing is coupled to `gnoi_client`'s print format, which has no stability guarantee |
14+
| gRPC status codes are not propagated | Caller only sees `rc != 0` — no status code, no error details |
15+
| Errors surface as Go `panic()` stack traces on stderr | Diagnosing RPC failures requires SSH + manual docker exec |
16+
| `suppress_stderr=True` on the Reboot call | Panic output is discarded; logs show only "command failed" |
17+
| Completion check uses string matching | `"reboot complete" in out_s.lower()` does not match actual output format (see §2) |
18+
| Tight coupling to CLI flag interface | `-module System -rpc Reboot -jsonin '{...}'` adds a serialization layer between caller and protobuf |
19+
| Broader privilege surface | Shell-out through Docker CLI vs. a direct gRPC socket |
2020

21-
## 2. Already Broken: RebootStatus Parsing
21+
## 2. Existing Issue: RebootStatus Completion Detection
2222

2323
The poll loop checks:
2424

@@ -34,7 +34,7 @@ System RebootStatus
3434
{"active":false,"status":{"status":"STATUS_SUCCESS","message":"..."}}
3535
```
3636

37-
`"reboot complete"` never appears → poll **always times out** regardless of DPU state.
37+
`"reboot complete"` does not appear in this output → the poll always exhausts its timeout regardless of DPU state.
3838

3939
## 3. Proposed Change
4040

@@ -59,10 +59,42 @@ if not resp.active and resp.status.status == STATUS_SUCCESS:
5959
```
6060

6161
### What this gives us
62-
- **Structured errors**: `grpc.RpcError` with status code + details instead of opaque exit codes
63-
- **Correct completion check**: inspect `resp.active` and `resp.status.status` directly
64-
- **No container dependency**: gRPC goes straight to the DPU, gnmi container health is irrelevant
65-
- **No parsing**: protobuf deserialization, not string matching on CLI output
62+
63+
**Better error detection** — gRPC errors carry status codes and details natively:
64+
```python
65+
except grpc.RpcError as e:
66+
logger.log_error(f"{dpu_name}: Reboot failed: {e.code()} {e.details()}")
67+
# e.g. "UNAVAILABLE: connection refused" vs today's "command failed"
68+
```
69+
70+
**Better testing** — mocks operate on typed protobuf objects instead of crafting subprocess stdout strings:
71+
```python
72+
# Today: mock must reproduce gnoi_client's exact text output
73+
mock_execute.return_value = (0, "reboot complete", "") # this doesn't even match reality
74+
75+
# After: mock returns a typed response
76+
mock_client.reboot_status.return_value = RebootStatusResponse(
77+
active=False, status=RebootStatus(status=STATUS_SUCCESS))
78+
```
79+
80+
**Correct completion check** — inspect `resp.active` and `resp.status.status` directly instead of string matching.
81+
82+
**Removes unnecessary NPU gnmi container dependency** — the current approach shells into the NPU's `gnmi` container to run `gnoi_client`, but there's no reason the NPU daemon needs the NPU gnmi container as an intermediary. The DPU's own gnmi server is the actual RPC endpoint; direct gRPC connects to it without involving the NPU container.
83+
84+
**Scales to future RPCs** — the same pattern extends to any gNOI or gNMI call without adding more subprocess wrappers:
85+
```python
86+
# Adding a new gNOI RPC is just another method on the client
87+
class GnoiClient:
88+
def reboot(self, ...): ...
89+
def reboot_status(self, ...): ...
90+
def cancel_reboot(self, ...): ... # future
91+
def system_time(self, ...): ... # future
92+
93+
# Or a gNMI client alongside it
94+
with GnmiClient(f"{dpu_ip}:{port}") as client:
95+
client.get(path="/system/state/...")
96+
```
97+
Each new RPC is a typed method with protobuf request/response — no new shell commands, no new output formats to parse.
6698

6799
## 4. Scope
68100

@@ -72,6 +104,25 @@ if not resp.active and resp.status.status == STATUS_SUCCESS:
72104
- Refactor the two RPC call sites in `GnoiRebootHandler`
73105
- Update unit tests
74106

107+
New directory structure:
108+
```
109+
host_modules/gnoi/
110+
├── __init__.py
111+
├── client.py # GnoiClient: reboot(), reboot_status(), context manager
112+
├── system_pb2.py # vendored from openconfig/gnoi system.proto
113+
├── system_pb2_grpc.py
114+
├── types_pb2.py # dependency of system.proto
115+
└── types_pb2_grpc.py
116+
```
117+
118+
The daemon change is essentially replacing `execute_command(["docker", "exec", ...])` with:
119+
```python
120+
with GnoiClient(f"{dpu_ip}:{port}") as client:
121+
client.reboot(method=REBOOT_METHOD_HALT, ...)
122+
# ...
123+
resp = client.reboot_status()
124+
```
125+
75126
### Out of scope
76127
- TLS/mTLS on midplane (future work; midplane is trusted today)
77128
- Main loop, config DB subscription, halt flag handling — unchanged

0 commit comments

Comments
 (0)