Commit 53461b6
authored
xds: skip DiscoveryRequest for unsubscribed types on stream ready (#12782)
## TL;DR
When the bootstrap declares more than one xDS server (e.g. a default
server for LDS/CDS plus an authority-specific EDS-only server),
grpc-java was sending CDS/LDS DiscoveryRequests to the EDS-only server
too. That server replies `UNIMPLEMENTED`, which tears down the stream
and EDS data never arrives. Fix: skip DiscoveryRequests for resource
types we don't actually subscribe to on a given server.
## Problem
Under [A47 — xDS
Federation](https://github.com/grpc/proposal/blob/master/A47-xds-federation.md),
authorities can declare their own `xds_servers` block in the bootstrap.
When an ADS stream is opened to an authority-specific server,
`ControlPlaneClient.adjustResourceSubscription` sends a
`DiscoveryRequest` for every **globally-subscribed** resource type —
even types that have no subscription for *this* server. The empty
request still carries a `type_url`, and an authority-specific server
(e.g. an EDS-only control plane) may reject it with `UNIMPLEMENTED`,
which tears down the entire stream before the legitimate request that
follows can complete.
## Reproduce
Bootstrap declares two servers — call them control-plane-A (handles
LDS/CDS for authority A) and control-plane-B (handles EDS only for
authority B).
A grpc-java channel that resolves through LDS → CDS in authority A and
EDS in authority B opens an ADS stream to control-plane-B. When that
stream becomes ready,
[`sendDiscoveryRequests`](https://github.com/grpc/grpc-java/blob/master/xds/src/main/java/io/grpc/xds/client/ControlPlaneClient.java)
iterates `resourceStore.getSubscribedResourceTypesWithTypeUrl()` — which
returns **all three** types (Listener, Cluster, Endpoint) — and calls
`adjustResourceSubscription` for each. For Listener and Cluster,
`getSubscribedResources(serverInfo, type)` returns null/empty, but the
request is still sent on the wire:
```
io.grpc.StatusRuntimeException: UNAVAILABLE: Error retrieving EDS resource ...:
UNIMPLEMENTED. Details: Watches for type type.googleapis.com/envoy.config.cluster.v3.Cluster
are not supported in this service
```
The Cluster (CDS) request reaches control-plane-B, gets rejected, and
the stream goes into backoff with no EDS data ever delivered. grpc-go on
the same bootstrap works fine against the same server, which pointed at
the asymmetry.
## Fix
In `adjustResourceSubscription`, return early when both:
1. `resources` is `null` (the store reports no subscription for this
type on this server), and
2. No DiscoveryRequest of this type has been sent on the current stream
(`!adsStream.sentTypes.contains(resourceType)`).
Per the `ResourceStore` contract in `XdsClient.java`, a `null` return
means "no subscription", while an empty collection means a **wildcard**
subscription — a real subscription that must still emit the empty
`resource_names` request and start its missing-resource timers.
The "DiscoveryRequest of this type has been sent on the current stream"
discriminator is tracked in a per-stream `sentTypes` set on `AdsStream`,
populated wherever a request is actually transmitted (initial sends,
ACKs, NACKs). Per-stream is the right scope because the server's view of
our subscriptions is per-stream — on stream replacement the set is
cleared implicitly along with the `AdsStream` instance.
Gating on `!versions.containsKey(type)` instead would be incorrect
because `versions` is only populated on ACK. If a watch is canceled
after the initial DiscoveryRequest goes out but before any response is
ACKed, that guard would suppress the empty unsubscribe — leaving the
server with a stale subscription until the stream resets. Tracking
actual sends per-stream closes that window.
The unsubscribe-all path (had-version, now-empty/null) is preserved: we
still send the empty request and clear the version, telling the server
to drop our subscription for that type.
## Mirrors grpc-go
This brings grpc-java in line with grpc-go's behavior. The Go equivalent
is
[`adsStreamImpl.sendExisting`](https://github.com/grpc/grpc-go/blob/v1.80.0/internal/xds/clients/xdsclient/ads_stream.go#L335-L368):
```go
for typ, state := range s.resourceTypeState {
state.nonce = ""
if len(state.subscribedResources) == 0 {
continue // <-- explicit skip
}
names := resourceNames(state.subscribedResources)
if err := s.sendMessageLocked(stream, names, typ.TypeURL, state.version, state.nonce, nil); err != nil {
return err
}
s.startWatchTimersLocked(typ, names)
}
```
Two structural reasons grpc-go avoids this bug today:
1. The iteration domain is **per-stream**
([`s.resourceTypeState`](https://github.com/grpc/grpc-go/blob/v1.80.0/internal/xds/clients/xdsclient/ads_stream.go#L143)),
populated only when
[`subscribe`](https://github.com/grpc/grpc-go/blob/v1.80.0/internal/xds/clients/xdsclient/ads_stream.go#L167-L193)
is called for a resource on this stream — so a Cluster type never even
appears in the iteration of an EDS-only stream.
2. Even within that per-stream iteration, the explicit `if
len(state.subscribedResources) == 0 { continue }` covers the case where
the type has no subscription on this stream.
The grpc-java fix is the equivalent of (2). The
`!adsStream.sentTypes.contains` guard is needed because Java's iteration
domain is global (`getSubscribedResourceTypesWithTypeUrl` is
xds-client-wide), so we may see types we never subscribed to on this
stream.
Note that grpc-go physically separates two paths:
[`sendNewLocked`](https://github.com/grpc/grpc-go/blob/v1.80.0/internal/xds/clients/xdsclient/ads_stream.go#L308-L318)
handles runtime sub/unsub and sends every queued request unconditionally
(so an empty unsubscribe always goes on the wire, ACK or no ACK), while
`sendExisting` handles stream re-establishment and applies the `len ==
0` skip. grpc-java has a single `adjustResourceSubscription` function
that serves both paths — the per-stream `sentTypes` set is what lets the
same guard distinguish them: on a fresh stream `sentTypes` is empty so
the guard reduces to "no subscription → skip" (mirroring
`sendExisting`), while at runtime after the initial request
`sentTypes.contains(type)` is true so the guard does not trigger and the
empty unsubscribe is sent (mirroring `sendNewLocked`).
## Test plan
Unit tests in `ControlPlaneClientTest`:
- `streamReady_skipsEmptyDiscoveryRequestForUnsubscribedType` — the
federation case, asserts CDS request is suppressed and EDS still goes
through
- `streamReady_sendsRequestForAllTypesWhenAllHaveResources` — guards
against over-eager skip
- `streamReady_skipsTypeWithNoSubscription` — `null` return skips
- `streamReady_sendsWildcardRequestAndStartsTimers` — empty collection
(wildcard) still sends and starts timers
- `cancelBeforeAck_sendsEmptyUnsubscribe` — cancel-before-ACK timing
window still emits the unsubscribe
## Spec note
xDS SoTW spec ([Envoy xDS
protocol](https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol))
treats an empty `resource_names` for LDS/CDS as a wildcard subscription
("send me everything of this type"). The previous grpc-java behavior
would unintentionally trigger wildcard CDS subscriptions on every
authority-specific stream — which an EDS-only server is right to refuse.
Skipping when no subscription exists side-steps that misinterpretation;
legitimate wildcard subscriptions (empty collection from a real
subscription) still go on the wire as intended, and the existing
unsubscribe-all path (with a prior version) continues to work.1 parent 0ddad69 commit 53461b6
2 files changed
Lines changed: 310 additions & 0 deletions
File tree
- xds/src
- main/java/io/grpc/xds/client
- test/java/io/grpc/xds/client
Lines changed: 31 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
160 | 160 | | |
161 | 161 | | |
162 | 162 | | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
163 | 188 | | |
164 | 189 | | |
165 | 190 | | |
| |||
319 | 344 | | |
320 | 345 | | |
321 | 346 | | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
322 | 352 | | |
323 | 353 | | |
324 | 354 | | |
| |||
358 | 388 | | |
359 | 389 | | |
360 | 390 | | |
| 391 | + | |
361 | 392 | | |
362 | 393 | | |
363 | 394 | | |
| |||
Lines changed: 279 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
0 commit comments