Skip to content

Leios prototype recreation onto current main (chunked)#5370

Open
ch1bo wants to merge 976 commits into
leios-prototypefrom
leios-prototype-remake
Open

Leios prototype recreation onto current main (chunked)#5370
ch1bo wants to merge 976 commits into
leios-prototypefrom
leios-prototype-remake

Conversation

@ch1bo
Copy link
Copy Markdown

@ch1bo ch1bo commented May 15, 2026

Note

The base branch here (leios-prototype) is the state before the recreation;
this PR is intended to merge by fast-forwarding leios-prototype to the head
of leios-prototype-remake. The diff is huge because it includes the
snapshot-merge to upstream main. For focused review:

  • Recreation against current main: main...leios-prototype-remake
  • Per-commit first-parent view: git log --first-parent ebc8eac0a8..leios-prototype-remake

Important

Cross-repo: this branch is consumed by ouroboros-consensus's
leios-prototype-remake (PR IntersectMBO/ouroboros-consensus#2041), where
the Leios threadnet test is currently green on 100/100 random seeds with
this branch pinned via SRP.

Done

  • Snapshot-merge onto upstream main (ebc8eac0a8) — the
    leios-prototype-remake ref is reset to upstream main's tree while
    keeping the prototype's historic commits reachable via the merge
    commit's first parent.
  • Demo-tuning: BearerBytes class replaces dataSize (55c16cf687)
    Drop the dataSize :: bytes -> Word field from ProtocolSizeLimits;
    replace with a class BearerBytes bytes where bearerBytesSize :: bytes -> Word and default instances for BS.ByteString, BL.ByteString,
    [Char], AnyMessage msg. The seven byteLimits* codec helpers no
    longer take a (bytes -> Word) argument; call sites drop the
    (fromIntegral . LBS.length) boilerplate. Driver.Limits gains a
    BearerBytes bytes constraint on the public entry points
    (runPeerWithLimits and friends).
  • Demo-tuning: arrival-time Reception on recv + TraceRecvMsg
    (319743abd0) — bearer-side recv now returns Maybe (Reception a)
    where Reception a = MkReception !(IntMap Time) !a records per-chunk
    arrival times keyed by starting-byte offset, and that arrival time is
    threaded all the way through the decoder to surface on TraceRecvMsg :: Maybe Time -> AnyMessage ps -> TraceSendRecv ps (with TraceSendMsg :: Time -> AnyMessage ps -> ... for symmetry). The mux demuxer pairs each
    SDU's read time with its offset into the ingress queue
    (IngressQueueVal Int64 (IntMap Time) Builder). The driver's decoder
    threads (IntMap Time, Word rsz, Word sz) so the next-message arrival
    time isn't lost when a message boundary lands inside a chunk
    (runDecoderWithChannel_DecodeDone helper, mbEq <|> mbTm fallback).
    Reception is re-exported from Network.Mux and Ouroboros.Network.Mux;
    MonadMonotonicTime m propagates to a handful of test entry points.
    Squashes the prototype's original Reception drop together with the
    subsequent TraceRecvMsg consumer-side wiring + the four runDecoder*
    bugfixes that followed.
  • Demo-tuning: field names on Reception (3afd70f505) — name the
    MkReception fields receivalTimes / received for ergonomic access.

Still TODO

  • TCP_NOTSENT_LOWAT + TCP_NODELAY on socket bearer set on the
    mw/leios-prototype branch — can be rebased.
  • Re-sign all commits if required for merge.
  • Open question: can these add-ons be PRed straight upstream into
    ouroboros-network master, eliminating the need for a fork branch?

Intentional differences from the original prototype

The Leios mini-protocols (LeiosNotify, LeiosFetch) live in ouroboros-consensus
rather than here — easier iteration in one repo. That's why this branch carries
only the bearer-layer demo-tuning (arrival times, BearerBytes) but no Leios
protocol content.

Merge plan

Fast-forward leios-prototype to the head of this branch (after the
TODOs above are resolved).

coot and others added 30 commits January 28, 2026 12:03
* leios block is 1000 MuxSDUs
* praos block is 10 MuxSDUs

Define `EGRESS_THROUPUT`, `INGRESS_DELAY` and `RTS_OPTIONS` for easier
configuration.
Label the thread just once by the `Job` when it is scheduled by
`JobPool`.
Let the server only respond to first inbound connection, then exit.
This makes it easier to get eventlog output from the server.
* build with chap: use versions from the git repo rather than the latest
  version in CHaP, this allows to build older releases
* release to CHaP:
  * better branch name
  * use `add-from-github.sh` once, to avoid GitHub rate limits
typed-protocols-1.2 integration
Some times we need to release breaking changes in already released
versions.  It is impossible to do if a major versions `0.x.y.z` and
`0.x+1.y.z` are already published.  Hence, we are going to increase `x`
in `x.y.z.w` whenever we release from `main` branch.
coot and others added 12 commits May 12, 2026 13:47
This avoids re-serialisation cost to get the size, increases allocation
by storing bytes of the original `tx`.
WithBytes retains tx bytes for whole tx life time (e.g. until the
inbound side exits), avoiding it we can GC the bytes after decoding is
done and we extract the length of a tx.
./network-mux/demo/ns-setup.sh is a generic script for running client
/ server applications using traffic shaping.  We can reuse it for
`tx-submission` demo.
This makes it easier to analyse different runs, since all of them use
the same connection four tuples.
Co-authored-by: Karl Knutsson <karl.fb.knutsson@gmail.com>
Using `hsubparser (command ... <|> command ... <|> ..)` rather
`hsubparser (command ..) <|> hsubparser (command ...) <|> ...`

The former can introspect `hsubparser` and give better usage message over
the latter which cannot introspect different `hsubparser`s.
@github-project-automation github-project-automation Bot moved this to In Progress in Ouroboros Network May 15, 2026
@ch1bo ch1bo changed the title WIP: Leios prototype recreation onto current main (snapshot-merge only) WIP: Leios prototype recreation onto current main May 15, 2026
Snapshot-merge: parents are (old leios-prototype tip, origin/main tip)
but tree equals origin/main byte-for-byte. The leios-prototype branch
ref moves forward without losing the historic prototype commits — they
remain reachable as ancestors via the merge's first parent.

The chunks of the recreation land as ordinary commits on top of this
one.
@ch1bo ch1bo force-pushed the leios-prototype-remake branch from 16ff37f to ebc8eac Compare May 15, 2026 06:51
@ch1bo ch1bo added the leios Issues / PRs related to Leios label May 15, 2026
@ch1bo ch1bo requested a review from nfrisby May 15, 2026 11:52
nfrisby and others added 3 commits May 15, 2026 14:38
Replace the `dataSize :: bytes -> Word` field of `ProtocolSizeLimits`
with a `BearerBytes` class:

    class BearerBytes bytes where
        bearerBytesSize :: bytes -> Word

Default instances cover `BS.ByteString`, `BL.ByteString`, `[Char]` and
`AnyMessage msg` (for tests).  Driver/Limits gains a `BearerBytes bytes`
constraint on the public entry points (`runPeerWithLimits` and friends).
The seven `byteLimits*` codec helpers no longer take a
`(bytes -> Word)` argument — call sites drop the `(fromIntegral .
LBS.length)` boilerplate.  Two test predicates that previously used
`dataSize` now call `bearerBytesSize` directly.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wrap `Channel m a` recv values in a new `Reception a = MkReception
!(IntMap Time) !a`, recording per-chunk arrival times.  In the mux
demuxer the chunk's arrival time is paired with its starting byte
offset; codec/driver layers either strip the time map (the codec sees
only bytes) or propagate it through `wrapMiniProtocolTrailing` at the
mini-protocol callback boundary.  Tests/demos/benches use `IntMap.empty`
when synthesising trailing.

This is the network half of the November Leios demo's arrival-time
plumbing; consumers (e.g. Praos block-arrival tracing on the consensus
side) will be wired up separately.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the positional definition of `Reception` with named fields
`receivalTimes` and `received`:

    data Reception a = MkReception
      { receivalTimes :: !(IntMap Time)
      , received      :: !a
      }

Positional pattern matches continue to work; the fields just make
record-style access available where it reads better.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@ch1bo ch1bo force-pushed the leios-prototype-remake branch from 92f0475 to 3afd70f Compare May 15, 2026 12:41
@ch1bo ch1bo marked this pull request as ready for review May 15, 2026 13:40
@ch1bo ch1bo requested review from a team as code owners May 15, 2026 13:40
@ch1bo ch1bo changed the title WIP: Leios prototype recreation onto current main Leios prototype recreation onto current main May 15, 2026
@ch1bo ch1bo linked an issue May 15, 2026 that may be closed by this pull request
3 tasks
@ch1bo ch1bo changed the title Leios prototype recreation onto current main Leios prototype recreation onto current main (chunked) May 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

leios Issues / PRs related to Leios

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

Update prototype to Dijsktra