Commit 47efc0d
authored
Migrate BookkeeperProtocol from protobuf-java to LightProto (apache#4780)
* Fix LedgerHandle.batchReadUnconfirmedAsync: use slog log instead of LOG
The batchReadUnconfirmedAsync method added in apache#4739 calls LOG.error(...),
but LedgerHandle was migrated to slog and only has a lowercase `log`
field. Master fails to compile.
Convert the call to the slog builder style used elsewhere in the file.
* Migrate DataFormats and DbLedgerStorageDataFormats from protobuf-java to LightProto
Replace Google's protobuf-java with StreamNative LightProto for the storage and
metadata formats in `bookkeeper-proto`. The wire protocol (`BookkeeperProtocol`)
remains on protobuf-java for now.
LightProto generates mutable, reusable, ByteBuf-aware classes with built-in
proto2 TextFormat (de)serialization (via `generateTextFormat=true`), so the
existing TextFormat-based znode payloads (cookies, auditor votes, lock data,
underreplication entries, layout) round-trip byte-identically.
Notable behavior changes:
- `BookKeeper.DigestType.toProtoDigestType` now returns the LightProto-generated
enum (same constants, different package).
- v3 ledger metadata uses a hand-rolled length-prefixed delimited writer/reader
matching protobuf's `writeDelimitedTo`/`mergeDelimitedFrom`.
* Add SpotBugs excludes for lightproto-generated classes
The existing exclude `~org.apache.bookkeeper.proto.DataFormats.*` matched
protobuf's nested `DataFormats$LedgerMetadataFormat` etc. LightProto generates
the same messages as flat top-level classes (`LedgerMetadataFormat` directly
in `org.apache.bookkeeper.proto`), so those weren't excluded and triggered
12 bugs in the generated code (bit-twiddling, exposed internal byte arrays,
etc.) that aren't actionable.
Replace the obsolete `DataFormats.*` exclude with explicit per-message
patterns covering both packages and `LightProtoCodec` (also generated
per-package).
* Fix checkstyle: remove redundant same-package import in MockBookies
* Migrate BookkeeperProtocol from protobuf-java to LightProto
Migrates the BookkeeperProtocol.proto wire protocol to use LightProto for
serialization. Combined with the prior migrations of DataFormats and
DbLedgerStorageDataFormats, this drops the protobuf-java runtime dependency
from bookkeeper-proto entirely.
LightProto produces wire-compatible output with protoc for the same .proto,
so on-the-wire bookie/client compatibility is preserved.
Notes on lifecycle handling:
- LightProto messages parsed from a ByteBuf hold lazy references into that
buffer for field access. The decoders now call materialize() on parsed
Request/Response/AuthMessage instances so they survive after the source
buffer is released.
- Server response paths that put entry payloads into ReadLacResponse or
ReadResponse now copy the bytes via ByteBufUtil.getBytes(...), matching
the previous ByteString.copyFrom semantics.
Drive-by fix: processWriteLacRequestV3/processReadLacRequestV3 were
ordering work on r.getAddRequest().getLedgerId() instead of the matching
WriteLac/ReadLac request. With protobuf this returned a default 0 for the
unset field; with LightProto it throws IllegalStateException.
* Fix checkstyle: remove redundant same-package imports
* Fix shaded-jar tests after protobuf-java removal
protobuf-java is no longer pulled in transitively by bookkeeper-server,
so the shaded jars no longer contain (shaded) protobuf classes, and the
flat lightproto-generated classes have replaced the BookkeeperProtocol
outer class.
- BookKeeperServerShadedJarTest / DistributedLogCoreShadedJarTest:
drop the now-irrelevant testProtobufShadedPath checks and switch the
BookkeeperProtocol presence check to a real lightproto class
(AddRequest).
- Drop the dead com.google.protobuf:protobuf-java <include> from the
three shade plugin configs (bookkeeper-server-shaded,
bookkeeper-server-tests-shaded, distributedlog-core-shaded).
* Use ByteBufList.toByteBuf to avoid copying request bodies
Replace ByteBufList.coalesce(...) with a new ByteBufList#toByteBuf method
on the WriteLac and AddEntry request paths. coalesce allocates a new
buffer and copies all the bytes; toByteBuf wraps the existing buffers in
a CompositeByteBuf (or returns the single buffer directly when the list
has one entry, or Unpooled.EMPTY_BUFFER when empty), transferring
ownership to the caller and releasing the source ByteBufList.
Adds unit tests covering the empty / single / multi-buffer paths,
including ref-count behaviour for both the list and the underlying
buffers.
* Fix ByteBufList.toByteBuf to not release the source list
The previous implementation released the source ByteBufList when
producing the wrapping ByteBuf, but callers in PerChannelBookieClient
only wrap a buffer they don't own — the ByteBufList's lifecycle is
managed by the upstream PendingAddOp, which shares the same list across
multiple bookies in a quorum write. Releasing in toByteBuf produced a
double-release / use-after-free that surfaced as
IllegalReferenceCountException in tests like BookieStickyReadsTest.
Change toByteBuf to leave the source list's ref count untouched and
return a wrapper that holds its own retains (a CompositeByteBuf for
multiple buffers, a retainedDuplicate for the single-buffer fast path,
or Unpooled.EMPTY_BUFFER for empty). This matches the original
ByteBufList.coalesce semantics from the caller's point of view, while
still avoiding the byte copy.
Tests updated to assert the new ownership semantics.
* Don't read required ledgerId/entryId from error responses
Four completion handlers (AddCompletion, ForceLedgerCompletion,
ReadCompletion, WriteLacCompletion) were always reading the inner
*Response's ledgerId/entryId fields, even when the outer Response
carried an error status. On error responses (e.g. EUA from a rejected
SASL handshake) those required fields are not populated. Under
protobuf-java they returned the default 0; under LightProto they throw
IllegalStateException("Field 'ledgerId' is not set"), which surfaced as
GSSAPIBookKeeperTest.testNotAllowedClientId blowing up on the client
side after the server rejected the auth.
Read the inner response only when status is EOK and hasXxxResponse() is
true. Otherwise fall back to the request's ledgerId/entryId, which the
CompletionValue base class already records.
* Read response ledgerId/entryId on long-poll reads in ReadCompletion
Long-poll reads send entryId=LAST_ADD_CONFIRMED and the bookie fills in
the actual entry id (and ledgerId) on the response when an entry is
returned. The previous fix in ReadCompletion always used the request's
recorded entryId, which made the long-poll path look like an empty
piggy-back response on the client side and dropped the entry buffer.
Read ledgerId/entryId from the response when status is EOK and the
inner ReadResponse is present; fall back to the request's recorded
values only on error envelopes (where the inner response may be
missing or unpopulated). Fixes
TestReadLastConfirmedAndEntry.testRaceOnLastAddConfirmed.1 parent 9633bb7 commit 47efc0d
57 files changed
Lines changed: 769 additions & 1022 deletions
File tree
- bookkeeper-proto
- bookkeeper-server/src
- main/java/org/apache/bookkeeper
- client
- proto
- util
- test/java/org/apache/bookkeeper
- client
- proto
- test
- util
- buildtools/src/main/resources/bookkeeper
- microbenchmarks/src/main/java/org/apache/bookkeeper/proto
- shaded
- bookkeeper-server-shaded
- bookkeeper-server-tests-shaded
- distributedlog-core-shaded
- tests/shaded
- bookkeeper-server-shaded-test/src/test/java/org/apache/bookkeeper/tests/shaded
- distributedlog-core-shaded-test/src/test/java/org/apache/bookkeeper/tests/shaded
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | 28 | | |
33 | 29 | | |
34 | 30 | | |
| |||
42 | 38 | | |
43 | 39 | | |
44 | 40 | | |
45 | | - | |
46 | 41 | | |
47 | 42 | | |
48 | 43 | | |
49 | 44 | | |
50 | 45 | | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | 46 | | |
71 | 47 | | |
72 | 48 | | |
73 | 49 | | |
74 | 50 | | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | 51 | | |
80 | 52 | | |
81 | 53 | | |
| |||
Lines changed: 7 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | | - | |
| 38 | + | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
50 | | - | |
| 49 | + | |
| 50 | + | |
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| |||
329 | 329 | | |
330 | 330 | | |
331 | 331 | | |
332 | | - | |
333 | | - | |
| 332 | + | |
| 333 | + | |
334 | 334 | | |
335 | 335 | | |
336 | 336 | | |
| |||
413 | 413 | | |
414 | 414 | | |
415 | 415 | | |
416 | | - | |
417 | | - | |
| 416 | + | |
| 417 | + | |
418 | 418 | | |
419 | 419 | | |
420 | 420 | | |
| |||
Lines changed: 13 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
117 | 117 | | |
118 | 118 | | |
119 | 119 | | |
120 | | - | |
| 120 | + | |
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
125 | 125 | | |
126 | 126 | | |
127 | 127 | | |
128 | | - | |
| 128 | + | |
129 | 129 | | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
135 | 140 | | |
136 | 141 | | |
137 | 142 | | |
138 | | - | |
| 143 | + | |
139 | 144 | | |
140 | 145 | | |
141 | 146 | | |
| |||
Lines changed: 46 additions & 55 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
26 | 25 | | |
27 | 26 | | |
28 | 27 | | |
| |||
40 | 39 | | |
41 | 40 | | |
42 | 41 | | |
43 | | - | |
44 | 42 | | |
45 | 43 | | |
46 | 44 | | |
| |||
90 | 88 | | |
91 | 89 | | |
92 | 90 | | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
| 91 | + | |
97 | 92 | | |
98 | 93 | | |
99 | 94 | | |
| |||
114 | 109 | | |
115 | 110 | | |
116 | 111 | | |
117 | | - | |
118 | | - | |
119 | | - | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
120 | 115 | | |
121 | 116 | | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
| 117 | + | |
126 | 118 | | |
127 | 119 | | |
128 | | - | |
| 120 | + | |
129 | 121 | | |
130 | 122 | | |
131 | 123 | | |
132 | | - | |
133 | | - | |
134 | | - | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
135 | 127 | | |
136 | | - | |
| 128 | + | |
137 | 129 | | |
138 | 130 | | |
139 | 131 | | |
| |||
171 | 163 | | |
172 | 164 | | |
173 | 165 | | |
174 | | - | |
175 | | - | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
176 | 169 | | |
177 | 170 | | |
178 | 171 | | |
179 | 172 | | |
180 | 173 | | |
181 | 174 | | |
182 | 175 | | |
183 | | - | |
| 176 | + | |
184 | 177 | | |
185 | 178 | | |
186 | 179 | | |
187 | | - | |
| 180 | + | |
188 | 181 | | |
189 | 182 | | |
190 | 183 | | |
191 | 184 | | |
192 | 185 | | |
193 | 186 | | |
194 | 187 | | |
195 | | - | |
196 | | - | |
| 188 | + | |
| 189 | + | |
197 | 190 | | |
198 | 191 | | |
199 | 192 | | |
200 | 193 | | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
| 194 | + | |
| 195 | + | |
205 | 196 | | |
206 | 197 | | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
213 | 203 | | |
214 | 204 | | |
215 | 205 | | |
| |||
272 | 262 | | |
273 | 263 | | |
274 | 264 | | |
275 | | - | |
276 | | - | |
| 265 | + | |
| 266 | + | |
277 | 267 | | |
278 | 268 | | |
279 | 269 | | |
| |||
286 | 276 | | |
287 | 277 | | |
288 | 278 | | |
289 | | - | |
290 | | - | |
| 279 | + | |
| 280 | + | |
291 | 281 | | |
292 | 282 | | |
293 | | - | |
| 283 | + | |
294 | 284 | | |
295 | 285 | | |
296 | 286 | | |
| |||
299 | 289 | | |
300 | 290 | | |
301 | 291 | | |
302 | | - | |
| 292 | + | |
303 | 293 | | |
304 | 294 | | |
305 | 295 | | |
| |||
321 | 311 | | |
322 | 312 | | |
323 | 313 | | |
324 | | - | |
| 314 | + | |
325 | 315 | | |
326 | 316 | | |
327 | 317 | | |
| |||
330 | 320 | | |
331 | 321 | | |
332 | 322 | | |
333 | | - | |
| 323 | + | |
334 | 324 | | |
335 | 325 | | |
336 | 326 | | |
| |||
353 | 343 | | |
354 | 344 | | |
355 | 345 | | |
356 | | - | |
| 346 | + | |
357 | 347 | | |
358 | | - | |
| 348 | + | |
359 | 349 | | |
360 | | - | |
361 | | - | |
| 350 | + | |
| 351 | + | |
362 | 352 | | |
363 | 353 | | |
364 | 354 | | |
| |||
421 | 411 | | |
422 | 412 | | |
423 | 413 | | |
424 | | - | |
425 | | - | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
426 | 417 | | |
427 | 418 | | |
428 | 419 | | |
429 | 420 | | |
430 | 421 | | |
431 | 422 | | |
432 | 423 | | |
433 | | - | |
434 | | - | |
435 | | - | |
436 | | - | |
437 | | - | |
438 | | - | |
439 | | - | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
440 | 431 | | |
441 | 432 | | |
442 | 433 | | |
| |||
0 commit comments