Leios prototype recreation onto current main (chunked)#5370
Open
ch1bo wants to merge 976 commits into
Open
Conversation
* leios block is 1000 MuxSDUs * praos block is 10 MuxSDUs Define `EGRESS_THROUPUT`, `INGRESS_DELAY` and `RTS_OPTIONS` for easier configuration.
ouroboros churn fix
Label the thread just once by the `Job` when it is scheduled by `JobPool`.
Let the server only respond to first inbound connection, then exit. This makes it easier to get eventlog output from the server.
mux leios simulation
* build with chap: use versions from the git repo rather than the latest version in CHaP, this allows to build older releases * release to CHaP: * better branch name * use `add-from-github.sh` once, to avoid GitHub rate limits
release scripts
The label is too verbose.
typed-protocols-1.2 integration
Some times we need to release breaking changes in already released versions. It is impossible to do if a major versions `0.x.y.z` and `0.x+1.y.z` are already published. Hence, we are going to increase `x` in `x.y.z.w` whenever we release from `main` branch.
Updated index-state, removed an srp
StandardHash with NFData constraint
This avoids re-serialisation cost to get the size, increases allocation by storing bytes of the original `tx`.
WithBytes retains tx bytes for whole tx life time (e.g. until the inbound side exits), avoiding it we can GC the bytes after decoding is done and we extract the length of a tx.
./network-mux/demo/ns-setup.sh is a generic script for running client / server applications using traffic shaping. We can reuse it for `tx-submission` demo.
This makes it easier to analyse different runs, since all of them use the same connection four tuples.
Co-authored-by: Karl Knutsson <karl.fb.knutsson@gmail.com>
Using `hsubparser (command ... <|> command ... <|> ..)` rather `hsubparser (command ..) <|> hsubparser (command ...) <|> ...` The former can introspect `hsubparser` and give better usage message over the latter which cannot introspect different `hsubparser`s.
tx-submission demo
nix: build all components
Snapshot-merge: parents are (old leios-prototype tip, origin/main tip) but tree equals origin/main byte-for-byte. The leios-prototype branch ref moves forward without losing the historic prototype commits — they remain reachable as ancestors via the merge's first parent. The chunks of the recreation land as ordinary commits on top of this one.
16ff37f to
ebc8eac
Compare
Replace the `dataSize :: bytes -> Word` field of `ProtocolSizeLimits`
with a `BearerBytes` class:
class BearerBytes bytes where
bearerBytesSize :: bytes -> Word
Default instances cover `BS.ByteString`, `BL.ByteString`, `[Char]` and
`AnyMessage msg` (for tests). Driver/Limits gains a `BearerBytes bytes`
constraint on the public entry points (`runPeerWithLimits` and friends).
The seven `byteLimits*` codec helpers no longer take a
`(bytes -> Word)` argument — call sites drop the `(fromIntegral .
LBS.length)` boilerplate. Two test predicates that previously used
`dataSize` now call `bearerBytesSize` directly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wrap `Channel m a` recv values in a new `Reception a = MkReception !(IntMap Time) !a`, recording per-chunk arrival times. In the mux demuxer the chunk's arrival time is paired with its starting byte offset; codec/driver layers either strip the time map (the codec sees only bytes) or propagate it through `wrapMiniProtocolTrailing` at the mini-protocol callback boundary. Tests/demos/benches use `IntMap.empty` when synthesising trailing. This is the network half of the November Leios demo's arrival-time plumbing; consumers (e.g. Praos block-arrival tracing on the consensus side) will be wired up separately. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the positional definition of `Reception` with named fields
`receivalTimes` and `received`:
data Reception a = MkReception
{ receivalTimes :: !(IntMap Time)
, received :: !a
}
Positional pattern matches continue to work; the fields just make
record-style access available where it reads better.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
92f0475 to
3afd70f
Compare
3 tasks
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
The base branch here (
leios-prototype) is the state before the recreation;this PR is intended to merge by fast-forwarding
leios-prototypeto the headof
leios-prototype-remake. The diff is huge because it includes thesnapshot-merge to upstream
main. For focused review:git log --first-parent ebc8eac0a8..leios-prototype-remakeImportant
Cross-repo: this branch is consumed by
ouroboros-consensus'sleios-prototype-remake(PR IntersectMBO/ouroboros-consensus#2041), wherethe Leios threadnet test is currently green on 100/100 random seeds with
this branch pinned via SRP.
Done
main(ebc8eac0a8) — theleios-prototype-remakeref is reset to upstreammain's tree whilekeeping the prototype's historic commits reachable via the merge
commit's first parent.
BearerBytesclass replacesdataSize(55c16cf687)Drop the
dataSize :: bytes -> Wordfield fromProtocolSizeLimits;replace with a
class BearerBytes bytes where bearerBytesSize :: bytes -> Wordand default instances forBS.ByteString,BL.ByteString,[Char],AnyMessage msg. The sevenbyteLimits*codec helpers nolonger take a
(bytes -> Word)argument; call sites drop the(fromIntegral . LBS.length)boilerplate.Driver.Limitsgains aBearerBytes bytesconstraint on the public entry points(
runPeerWithLimitsand friends).Receptiononrecv+TraceRecvMsg(
319743abd0) — bearer-side recv now returnsMaybe (Reception a)where
Reception a = MkReception !(IntMap Time) !arecords per-chunkarrival times keyed by starting-byte offset, and that arrival time is
threaded all the way through the decoder to surface on
TraceRecvMsg :: Maybe Time -> AnyMessage ps -> TraceSendRecv ps(withTraceSendMsg :: Time -> AnyMessage ps -> ...for symmetry). The mux demuxer pairs eachSDU's read time with its offset into the ingress queue
(
IngressQueueVal Int64 (IntMap Time) Builder). The driver's decoderthreads
(IntMap Time, Word rsz, Word sz)so the next-message arrivaltime isn't lost when a message boundary lands inside a chunk
(
runDecoderWithChannel_DecodeDonehelper,mbEq <|> mbTmfallback).Receptionis re-exported fromNetwork.MuxandOuroboros.Network.Mux;MonadMonotonicTime mpropagates to a handful of test entry points.Squashes the prototype's original
Receptiondrop together with thesubsequent
TraceRecvMsgconsumer-side wiring + the fourrunDecoder*bugfixes that followed.
Reception(3afd70f505) — name theMkReceptionfieldsreceivalTimes/receivedfor ergonomic access.Still TODO
TCP_NOTSENT_LOWAT+TCP_NODELAYon socket bearer set on themw/leios-prototypebranch — can be rebased.ouroboros-networkmaster, eliminating the need for a fork branch?Intentional differences from the original prototype
The Leios mini-protocols (
LeiosNotify,LeiosFetch) live inouroboros-consensusrather than here — easier iteration in one repo. That's why this branch carries
only the bearer-layer demo-tuning (arrival times,
BearerBytes) but no Leiosprotocol content.
Merge plan
Fast-forward
leios-prototypeto the head of this branch (after theTODOs above are resolved).