Commit 9188043
committed
Native allocator follow-up: dynamic settings, query/datafusion pools, plugin nodeStats
Builds on #21703 to add the pieces a real production deployment needs:
* Make pool min/max settings dynamic and grouped-validate cross-setting
invariants on every cluster-state update (sum of pool mins <= root,
per-pool min <= max). FLIGHT_MIN/INGEST_MIN defaults are 0L (the prior
Long.MAX_VALUE defaults caused the new validator to reject any non-MAX
root). setPoolMin now updates the live BufferAllocator.setLimit so the
Dynamic property has observable effect even when the rebalancer is off.
* Derive ROOT_LIMIT_SETTING default from the AC node-native-memory budget
(limit minus buffer-percent) so the framework respects the same budget
AC throttles on, with a Long.MAX_VALUE fallback when AC is unconfigured.
* Add QUERY and DATAFUSION pools alongside FLIGHT and INGEST. Pools init
at min when the rebalancer is enabled; otherwise at max so non-rebalanced
nodes can still allocate.
* Wire arrow-flight-rpc to the FLIGHT pool, parquet-data-format to the
INGEST pool, and analytics-engine via the framework's allocator service.
Hard-fail if the framework plugin is missing — silently skipping the
wire-up is the silent-misconfiguration class of bug we want to prevent.
* Cleanup ad-hoc allocator fallbacks in parquet-data-format / analytics-engine
so all Arrow consumers go through the unified pool hierarchy.
* Rebalancer now distributes headroom across all pools (not only those with
current allocation > 0). Avoids the dead-pool corner case where a pool
with min=0 starts at limit=0, can never make a first allocation, and
never receives a bonus.
Plugin _nodes/stats integration:
* Add Plugin#nodeStats() hook + PluginNodeStats interface in server.
* Wire NodeStats to carry Map<String, PluginNodeStats> with version-gated
ser/deser at V_3_7_0 and top-level rendering under nodes.<id>.<name>.
* Each entry is wire-framed as (name, length-prefixed bytes); the receiver
wraps the inner payload with NamedWriteableAwareStreamInput and drops
entries whose subtype is not registered locally. This makes mixed-version
rolling upgrades safe — a coordinator that lacks the plugin a data node
is running keeps decoding the rest of NodeStats instead of failing the
whole response.
* NativeAllocatorPluginStats adapter wraps NativeAllocatorPoolStats so the
framework contributes to _nodes/stats. The dedicated _native_allocator/stats
REST endpoint is gone — one observability surface, not two.
* Plugin stats are emitted on every _nodes/stats request regardless of the
?metric= filter; matches RemoteStoreNodeStats precedent. A future PR can
add a Metric.PLUGIN_STATS gate without breaking the wire protocol.
DataFusion spill memory limit:
* Promote datafusion.spill_memory_limit_bytes to Setting.Property.Dynamic.
* Wire addSettingsUpdateConsumer that branches on
NativeBridge.isSpillLimitDynamic(): mirrors live when df_set_spill_limit
is exported, otherwise warns and waits for next node restart.
API hygiene:
* parquet.max_per_vsr_allocation_ratio is a divisor (limit/N), not a ratio.
Renamed to parquet.max_per_vsr_allocation_divisor with a hard upper
bound of 100 to reject fat-finger PUTs that would starve every VSR.
Real regressions caught during self-audit:
* Existing arrow-flight-rpc internalClusterTests and the sandbox coordinator
ITs were not declaring the framework plugin in nodePlugins(). After the
flight transport got wired to the FLIGHT pool, those ITs fail at node
startup with "ArrowNativeAllocator not initialized". Each IT now installs
the framework plugin and lists it as an extendedPlugin.
Signed-off-by: Gaurav Singh <snghsvn@amazon.com>1 parent ac2b2fd commit 9188043
42 files changed
Lines changed: 1567 additions & 126 deletions
File tree
- libs/arrow-spi/src
- main/java/org/opensearch/arrow/spi
- test/java/org/opensearch/arrow/spi
- plugins
- arrow-base/src
- main/java/org/opensearch/arrow/allocator
- test/java/org/opensearch/arrow/allocator
- arrow-flight-rpc
- src
- main/java/org/opensearch/arrow/flight/transport
- test/java/org/opensearch/arrow/flight/transport
- sandbox
- plugins
- analytics-backend-datafusion
- src
- main/java/org/opensearch/be/datafusion
- nativelib
- test/java/org/opensearch/be/datafusion
- analytics-engine
- src/main/java/org/opensearch/analytics/exec
- composite-engine/src/internalClusterTest/java/org/opensearch/composite
- parquet-data-format/src
- main/java/org/opensearch/parquet
- engine
- memory
- test/java/org/opensearch/parquet
- engine
- memory
- vsr
- writer
- qa/analytics-engine-coordinator
- server/src
- main/java/org/opensearch
- action/admin/cluster/node/stats
- node
- plugins
- test/java/org/opensearch
- action/admin/cluster
- node/stats
- stats
- cluster
- test/framework/src/main/java/org/opensearch/cluster
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 18 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
58 | 76 | | |
59 | 77 | | |
60 | 78 | | |
| |||
Lines changed: 39 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
Lines changed: 12 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
33 | 37 | | |
34 | 38 | | |
35 | 39 | | |
| |||
42 | 46 | | |
43 | 47 | | |
44 | 48 | | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
45 | 57 | | |
46 | 58 | | |
47 | 59 | | |
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
18 | 20 | | |
19 | 21 | | |
20 | 22 | | |
21 | 23 | | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
25 | 31 | | |
26 | 32 | | |
27 | 33 | | |
| |||
Lines changed: 177 additions & 21 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
| 22 | + | |
21 | 23 | | |
22 | 24 | | |
23 | 25 | | |
| 26 | + | |
24 | 27 | | |
25 | 28 | | |
| 29 | + | |
26 | 30 | | |
27 | 31 | | |
28 | 32 | | |
| |||
50 | 54 | | |
51 | 55 | | |
52 | 56 | | |
53 | | - | |
54 | | - | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
55 | 67 | | |
56 | | - | |
57 | | - | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
58 | 78 | | |
59 | 79 | | |
60 | 80 | | |
61 | 81 | | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
62 | 96 | | |
63 | 97 | | |
64 | 98 | | |
65 | | - | |
| 99 | + | |
66 | 100 | | |
67 | 101 | | |
68 | 102 | | |
| |||
80 | 114 | | |
81 | 115 | | |
82 | 116 | | |
83 | | - | |
| 117 | + | |
84 | 118 | | |
85 | 119 | | |
86 | 120 | | |
| |||
95 | 129 | | |
96 | 130 | | |
97 | 131 | | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
98 | 192 | | |
99 | 193 | | |
100 | 194 | | |
| |||
127 | 221 | | |
128 | 222 | | |
129 | 223 | | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
134 | 227 | | |
135 | | - | |
136 | | - | |
137 | | - | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
138 | 236 | | |
139 | | - | |
140 | | - | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
141 | 248 | | |
142 | | - | |
143 | | - | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
144 | 252 | | |
145 | 253 | | |
146 | 254 | | |
| |||
161 | 269 | | |
162 | 270 | | |
163 | 271 | | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
164 | 301 | | |
165 | 302 | | |
166 | 303 | | |
| |||
169 | 306 | | |
170 | 307 | | |
171 | 308 | | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
172 | 313 | | |
173 | 314 | | |
174 | 315 | | |
175 | 316 | | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
176 | 332 | | |
177 | 333 | | |
178 | 334 | | |
| |||
185 | 341 | | |
186 | 342 | | |
187 | 343 | | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
192 | 348 | | |
193 | 349 | | |
194 | 350 | | |
| |||
0 commit comments