feat(hydro_lang): add an embedded serialization type for network channels

[shadaj]

Fixes #2999

Adds TCP...embedded() as an alternative to .bincode() for network channels.
An embedded channel leaves message serialization to code outside of Hydro: the
raw element type flows across the channel instead of serialized bytes, so the
developer can perform custom serialization for that channel. It is only supported
by the embedded deployment backend and the simulator; other backends panic at
instantiation time with a clear message.

Networking API (hydro_lang/src/networking/mod.rs):

• New Embedded marker SerKind and NetworkingConfig::embedded() builder
  method (mirrors .bincode()), plus an is_external() hook on SerKind /
  NetworkFor.

IR representation (hydro_lang/src/compile/ir/mod.rs):

• Per the maintainer's guidance, replaced the Network node's
  serialize_fn / deserialize_fn (Option<DebugExpr>) with two structured
  half-enums: NetworkSend and NetworkRecv. Each is either Custom (a stored
  serialize/deserialize expr, how bincode is lowered today) or Embedded
  (structured info: the optional routing MemberId tag on that side plus the raw
  element type). This sets up moving serialization codegen into a post-IR pass.
• NetworkSend/NetworkRecv::pipeline() synthesizes the codegen expression: for
  Embedded it only converts the MemberId tag to/from TaglessMemberId and
  passes the raw payload through (identity when untagged).
• VersionedNetworkFork/VersionedNetwork (multi-version sim) now carry
  NetworkSend/NetworkRecv for each half instead of raw exprs.
• Threaded the raw element type through instantiate_network, the Deploy
  trait sink/source methods, and the DfirBuilder create_network /
  create_versioned_network* methods. Added
  Deploy::SUPPORTS_EXTERNAL_SERIALIZATION (default false) so instantiate_network
  panics on unsupported backends.

Backends:

• Embedded backend (compile/embedded.rs): sets SUPPORTS_EXTERNAL_SERIALIZATION,
  records the raw element type per channel, and generates FnMut(T) /
  Stream<Item = Result<T, _>> network struct fields (instead of Bytes) for
  embedded channels.
• Simulator (sim/graph.rs, sim/builder.rs): sets the const and carries the raw
  payload type through its in-memory channels instead of Bytes.
• Deploy graph / containerized / ecs / maelstrom: signature-only updates; they
  reject embedded serialization via the default const.

Tests:

• hydro_test/hydro_test_embedded: new echo_network_embedded example + a test
  showing the generated channel exposes raw String payloads.
• hydro_lang sim: new sim_embedded_send test.
• Regenerated the IR debug snapshots (stable + nightly) for the reshaped
  Network node.

Co-authored-by: Infinity 🤖 infinity@hydro.run

[cloudflare-workers-and-pages]

Deploying hydro with    Cloudflare Pages

Latest commit:	0f8d5cc
Status:	✅  Deploy successful!
Preview URL:	https://cddb4321.hydroflow.pages.dev
Branch Preview URL:	https://sandbox-f2bec072-3efe-449b-8.hydroflow.pages.dev

View logs

[Copilot]

Pull request overview

Adds a new “embedded” serialization mode for Hydro network channels, allowing raw element types to traverse channels without Hydro-managed serialization (supported only by the embedded backend and the simulator), and refactors network IR serialization representation to support that capability.

Changes:

• Introduces TCP...embedded() and an “external serialization” hook (is_external) to bypass serialize/deserialize thunk generation.
• Refactors HydroNode::Network IR from serialize_fn/deserialize_fn into structured NetworkSend/NetworkRecv halves and threads raw element type through instantiation/codegen paths.
• Updates embedded backend + simulator to carry raw payload types, and adds tests/examples plus regenerated IR snapshots.

Reviewed changes

Copilot reviewed 39 out of 39 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
hydro_test/src/embedded/mod.rs	Exposes new embedded-network example module.
hydro_test/src/embedded/echo_network_embedded.rs	New example flow using .embedded() to send raw String.
hydro_test/src/distributed/snapshots/first_ten_distributed_ir.snap	Snapshot regen for NetworkSend/Recv IR shape.
hydro_test/src/distributed/snapshots-nightly/first_ten_distributed_ir.snap	Nightly snapshot regen for NetworkSend/Recv IR shape.
hydro_test/src/cluster/snapshots/two_pc_ir@coordinator_mermaid.snap	Snapshot regen reflecting updated generated stageleft quote locations/IR.
hydro_test/src/cluster/snapshots/two_pc_ir.snap	Snapshot regen for new network IR serialization structure.
hydro_test/src/cluster/snapshots/simple_cluster_ir.snap	Snapshot regen for new network IR serialization structure.
hydro_test/src/cluster/snapshots/paxos_ir@proposer_mermaid.snap	Snapshot regen reflecting updated generated quote locations/IR.
hydro_test/src/cluster/snapshots/paxos_ir.snap	Snapshot regen for new network IR serialization structure.
hydro_test/src/cluster/snapshots/map_reduce_ir@surface_graph_loc1v1.snap	Snapshot regen reflecting updated generated quote locations/IR.
hydro_test/src/cluster/snapshots/map_reduce_ir.snap	Snapshot regen for new network IR serialization structure.
hydro_test/src/cluster/snapshots/many_to_many_ir.snap	Snapshot regen for new network IR serialization structure.
hydro_test/src/cluster/snapshots/compute_pi_ir.snap	Snapshot regen for new network IR serialization structure.
hydro_test/src/cluster/snapshots-nightly/two_pc_ir@coordinator_mermaid.snap	Nightly snapshot regen reflecting updated quote locations/IR.
hydro_test/src/cluster/snapshots-nightly/two_pc_ir.snap	Nightly snapshot regen for new network IR serialization structure.
hydro_test/src/cluster/snapshots-nightly/simple_cluster_ir.snap	Nightly snapshot regen for new network IR serialization structure.
hydro_test/src/cluster/snapshots-nightly/paxos_ir@proposer_mermaid.snap	Nightly snapshot regen reflecting updated quote locations/IR.
hydro_test/src/cluster/snapshots-nightly/map_reduce_ir@surface_graph_loc1v1.snap	Nightly snapshot regen reflecting updated quote locations/IR.
hydro_test/src/cluster/snapshots-nightly/map_reduce_ir.snap	Nightly snapshot regen for new network IR serialization structure.
hydro_test/src/cluster/snapshots-nightly/many_to_many_ir.snap	Nightly snapshot regen for new network IR serialization structure.
hydro_test/src/cluster/snapshots-nightly/compute_pi_ir.snap	Nightly snapshot regen for new network IR serialization structure.
hydro_test_embedded/src/lib.rs	Adds generated module wiring + test asserting raw String channel types.
hydro_test_embedded/build.rs	Generates embedded code for the new embedded-network example.
hydro_lang/src/viz/render.rs	Updates network node labeling logic for new NetworkSend/Recv representation.
hydro_lang/src/sim/versioned_network.rs	Updates versioned-network splice logic to carry NetworkSend/Recv.
hydro_lang/src/sim/tests/mod.rs	Adds simulator test covering .embedded() send/recv semantics.
hydro_lang/src/sim/graph.rs	Declares simulator support for external serialization + threads external type metadata.
hydro_lang/src/sim/builder.rs	Carries raw payload types through sim channels and adjusts Result-wrapping behavior.
hydro_lang/src/networking/mod.rs	Adds Embedded ser kind, .embedded() builder, and is_external() hook.
hydro_lang/src/live_collections/stream/networking.rs	Lowers networking nodes to NetworkSend/Recv (embedded vs custom).
hydro_lang/src/live_collections/keyed_stream/networking.rs	Same as above for keyed networking ops (including demux/tagging).
hydro_lang/src/deploy/maelstrom/deploy_maelstrom.rs	Signature update to accept external-type metadata (still rejects processes).
hydro_lang/src/deploy/deploy_graph.rs	Signature-only updates to thread external-type metadata.
hydro_lang/src/deploy/deploy_graph_containerized.rs	Signature-only updates to thread external-type metadata.
hydro_lang/src/deploy/deploy_graph_containerized_ecs.rs	Signature-only updates to thread external-type metadata.
hydro_lang/src/compile/ir/mod.rs	Introduces NetworkSend/Recv, pipelines, and external-type threading + backend gating.
hydro_lang/src/compile/embedded.rs	Embedded backend support for external serialization + raw-type network field generation.
hydro_lang/src/compile/deploy_provider.rs	Adds SUPPORTS_EXTERNAL_SERIALIZATION and threads external-type metadata through Deploy APIs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.