Skip to content

Pre-serialize pub/sub signals once across fan-out (flag-gated, default off)#2485

Open
thjaeckle wants to merge 5 commits into
eclipse-ditto:masterfrom
beyonnex-io:feature/preserialize-pubsub-fanout
Open

Pre-serialize pub/sub signals once across fan-out (flag-gated, default off)#2485
thjaeckle wants to merge 5 commits into
eclipse-ditto:masterfrom
beyonnex-io:feature/preserialize-pubsub-fanout

Conversation

@thjaeckle

@thjaeckle thjaeckle commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What

Introduce an opt-in optimization that serializes a published signal once and reuses those bytes across all
remote fan-out destinations, instead of letting Pekko Artery re-serialize the identical payload once per
destination. Flag-gated behind ditto.pubsub.pre-serialize-fanout-enabled and disabled by default.

Why

Under high fan-out load, cluster-internal serialization of published signals is a dominant CPU cost on the
things service (a top hotspot, ~25% of CPU, in production profiles). When a single published signal fans out to
N subscriber nodes, Artery serializes the identical payload N times.

How

  • New PreSerializedPublishSignal transport envelope carrying a shared, lazily-computed SignalBytesHolder. The
    first Artery encoder thread serializes the payload once; every subsequent remote destination reuses the cached
    bytes. Only the small per-destination routing information differs between envelopes.
  • A dedicated PreSerializedPublishSignalSerializer (own serializer id) reconstructs a plain PublishSignal on
    the receiving side, so the Subscriber actor is unchanged.
  • Publisher only uses the envelope when the fan-out reaches at least one remote node; purely node-local
    fan-out keeps using the in-memory signal (nothing is serialized there).
  • Exposed via Helm on the publishing services (things, policies, connectivity). Subscriber-only services
    (gateway, thingssearch) do not read the flag but must run this version so they can deserialize incoming
    envelopes.

Safety / rollout

The envelope relies on a new serializer; a node that does not have it registered would drop the fan-out message.
It is therefore only safe to enable after the whole cluster runs this version. A two-phase rollout is
documented in the (draft) 3.9.4 release notes: upgrade the entire fleet with the flag off, then enable it on the
publishing services. Because the default is off, upgrading is a no-op until the flag is explicitly turned on.

Robustness

  • All wire length prefixes are validated against the remaining bytes before allocating, so a truncated / corrupt /
    misrouted frame fails cleanly instead of raising OutOfMemoryError / NegativeArraySizeException on the Artery
    inbound thread.
  • An unknown inner manifest (e.g. a newer signal type reaching an older node) propagates the inner serializer's
    NotSerializableException — consistent with the regular PublishSignal path — instead of throwing on the
    decoder thread.
  • Strings are UTF-8-encoded exactly once on the serialization path.

Validation

  • The full system-test suite passes with the flag enabled — delivery correctness and cluster stability
    verified under real multi-node, multi-connection load (in addition to the default flag-off run).
  • JMH benchmark (SignalFanoutSerializationJmh): the pre-serialized path stays flat as fan-out grows and is
    up to ~11× faster than per-destination serialization at higher fan-out (e.g. fan-out 10: ~23 µs vs ~263 µs per
    publication), with lower heap allocation. Deserialization cost is unchanged.

Tests

Serializer round-trip (both entry points), holder thread-safety / serialize-once, serialization-extension
registration, end-to-end fan-out across two clustered actor systems, and malformed / truncated / unknown-manifest
failure cases.

thjaeckle and others added 3 commits July 8, 2026 08:51
@thjaeckle @claude
Pre-serialize pub/sub events once across fan-out (flag-gated, default…
2e1129d 
… off)

Pekko Artery re-serializes each published signal once per subscriber node
(prod: ~4.74x fan-out; the identical DittoHeaders dominate toBinary CPU, which
is ~21% of things-service CPU). Add an opt-in path that serializes a published
signal once and reuses the bytes across all remote destinations:

- SignalBytesHolder: serialize-once memoizer shared across a publication's
  fan-out envelopes (single volatile record, benign race).
- PreSerializedPublishSignal: wire-only transport envelope (shared holder +
  per-destination groups); never delivered to a Subscriber as-is.
- PreSerializedPublishSignalSerializer: custom length-prefixed frame reusing
  CborJsonifiableSerializer for the signal payload; fromBinary rebuilds a plain
  PublishSignal so the Subscriber is unchanged. Registered in
  ditto-pekko-config.conf (id 656329406).
- Publisher builds one shared holder per publication and sends the envelope when
  enabled and the signal fans out to more than one destination; AbstractSubscriber
  converts a locally-delivered envelope back to a PublishSignal.
- New config flag ditto.pubsub.pre-serialize-fanout-enabled (default false,
  env DITTO_PUBSUB_PRE_SERIALIZE_FANOUT_ENABLED).

Micro-benchmark (fan-out 5, 50 read-granted subjects): 131us -> 25us per
publication (5.18x less serialization work). Default off; requires a two-step
decode-first rollout before enabling fleet-wide (documented in reference.conf).
When enabled, fan-out destinations share one serialization span (trade-off of
serializing once) -- also noted in reference.conf.

Tests: unit round-trip (both paths + memoization + headers/readGrantedSubjects
survival + concurrent holder access), SerializationExtension registration +
round-trip, 2-node end-to-end with the flag on (remote + local delivery), and
PubSubFactoryTest unchanged with the flag off. Full reactor build (compile +
checkstyle + license + japicmp) green.

Reviewed with the workflow code-review; fixed the one confirmed correctness
issue (guard the deserialized-payload cast against a non-Signal result instead
of an opaque ClassCastException) and added the fan-out>1 threshold.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@thjaeckle @claude
helm: expose pre-serialize-fanout pub/sub flag + bump chart to 4.3.1
d99b678 
Expose the ditto.pubsub.pre-serialize-fanout-enabled setting (default false) for
the three services that run a pub/sub Publisher (things, policies, connectivity)
via DITTO_PUBSUB_PRE_SERIALIZE_FANOUT_ENABLED, mirroring the existing
policiesEnforcer settings. Lets operators toggle the optimization per the
required decode-first, enable-fleet-wide rollout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@thjaeckle @claude
Harden pre-serialize pub/sub fan-out + add 3.9.4 activation guide
ea1553f 
Address code-review findings on the flag-gated pre-serialize fan-out:

- PreSerializedPublishSignalSerializer: pin BIG_ENDIAN byte order so the
  Artery (little-endian) and byte[] (big-endian) paths interoperate;
  validate every wire length prefix against remaining bytes before
  allocating (no OOM/NegativeArraySize on a truncated/corrupt/misrouted
  frame); propagate the inner serializer's NotSerializableException for an
  unknown inner manifest instead of throwing IllegalArgumentException on the
  Artery decoder thread; wrap parse failures as a clean NotSerializableException;
  UTF-8-encode each string exactly once (encode-once EncodedFrame) instead of
  re-encoding in both size and write passes.
- Publisher: only build the shared holder / envelope when the fan-out reaches
  at least one remote node, so purely-local (single-node/dev) fan-out no longer
  allocates a holder + envelopes for payloads that are never serialized.
- Tests: cover unknown inner manifest, truncated frame and corrupt length prefix.
- Docs: add a 3.9.4 release-notes draft with a two-phase safe-activation guide
  (upgrade whole fleet with flag off, then enable on the publishing services).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@thjaeckle thjaeckle self-assigned this Jul 9, 2026
@thjaeckle thjaeckle added this to the 3.9.4 milestone Jul 9, 2026
@thjaeckle

thjaeckle commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

System tests:

@thjaeckle thjaeckle marked this pull request as draft July 9, 2026 16:04
@thjaeckle @claude
Fix Artery buffer-order corruption in pre-serialize fan-out serializer
0198278 
The endianness handling added earlier called buf.order(BIG_ENDIAN) on the
ByteBuffer that Artery passes to a ByteBufferSerializer. For Artery that buffer
is a pooled, reused EnvelopeBuffer allocated once as little-endian and never
reset; ByteBuffer.order() is sticky, so the flip persisted and the frame headers
(lengths, serializer id, compression indices) of the NEXT messages reusing that
buffer were written/read in the wrong order. That corrupted unrelated Artery
frames -> "Decoder - Dropping message due to BufferUnderflowException",
compression-table desync, dropped ddata gossip -> chronic Replicator$Update
25s timeouts -> pub/sub subscriptions never propagating -> cluster-wide 503/408.

Reproduced in the system tests (flag on: ~1000 failures + ~3000 ddata Update
timeouts; flag-off baseline: 0 failures, 4 timeouts). The serializer is only
invoked when the flag is on, which is why the baseline stayed clean; JMH missed
it because it serialized into its own buffers, not Artery's reused pool.

Fix: never change the order of a buffer we don't own. Standardize on
little-endian (Artery's native order): the (Object, ByteBuffer) and
(ByteBuffer, manifest) paths use the buffer's native order without mutating it;
the byte[] paths pin little-endian on their own allocated/wrapped buffers so both
entry points stay interoperable. Tests updated to mirror Artery (little-endian
buffers).

Also add SignalFanoutSerializationJmh: a JMH benchmark that models the flag-off
baseline via Artery's zero-copy ByteBuffer path (the pre-existing hand-rolled
benchmark compared the wrong path). It shows the pre-serialized path is flat and
up to ~11x faster than per-destination serialization with lower heap allocation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@thjaeckle

thjaeckle commented Jul 10, 2026

Copy link
Copy Markdown
Member Author

System tests:

@thjaeckle thjaeckle marked this pull request as ready for review July 10, 2026 08:03
@thjaeckle @claude
Remove misleading hand-rolled pre-serialize benchmark
3ea1253 
PreSerializeFanoutBenchmark serialized BOTH the baseline and the optimized path
via the byte[] entry point, so it never compared against Artery's zero-copy
ByteBuffer path (the actual flag-off path) and could not measure the real
difference. SignalFanoutSerializationJmh (proper JMH, models the Artery path)
supersedes it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@thjaeckle

thjaeckle commented Jul 10, 2026

Copy link
Copy Markdown
Member Author

@alstanchev this has the potential to greatly improve throughput and reduce CPU resources when you have multiple subscribers of events (e.g. connections, websockets, SSEs) running in the cluster.
However activation of it - as already added to the draft release notes - has to be carefully planned:
It's a 2-step thing, first deploy 3.9.4 and afterwards activate the flag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alstanchev alstanchev alstanchev approved these changes

Assignees

@thjaeckle thjaeckle

Labels

None yet

Projects

Ditto Planning
Status: No status

Milestone

3.9.4

Development

Successfully merging this pull request may close these issues.

2 participants

@thjaeckle @alstanchev