window-stats: efficient rolling window statistics by crocodile-dentist · Pull Request #5384 · IntersectMBO/ouroboros-network

crocodile-dentist · 2026-06-12T12:17:15Z

Description

Efficient sliding-window statistics backed by a finger tree, with a cached user-supplied measure that updates incrementally as the window slides.

This library is a first step towards exposing connection metrics to our applications.

Checklist

Quality

Commit sequence makes sense and have useful messages, see ref.
New tests are added and existing tests are updated.
Self-reviewed the PR.

Maintenance

Linked an issue or added the PR to the current sprint of ouroboros-network project.
Added labels.
Updated changelog files.
The documentation has been properly updated, see ref.

These modules provide a fingertree-backed sliding window with fixed element count. The fingertree backend caches a user-supplied monoidal measure `v` which updates incrementally as elements are added and evicted, which allows for constant time lookup of its value regardless of window size. The Window.Count module exposes a public API, and the private internal implementation is exposed by the Internal.Count module without any guarantees that it stays compatible between any releases. The Measures module exposes some common and practical prebuilt measures.

These modules keep samples within a configured duration of the newest sample's timestamp. These variants abstract over the timestamp type via the 'TimeLike' class, which have two implementations defined: - `UTCTime` / `NominalDiffTime` (wall-clock, from `time`) - `Time` / `DiffTime` (monotonic, from `io-classes:si-timers`) Monotonic is preferred when sliding-window correctness must not be perturbed by NTP corrections or wall-clock jumps; wall-clock fits data that already carries `UTCTime` timestamps.

These modules expose sliding windows backed by an approximate t-digest measure, which permit efficient computations of quantiles in bounded memory.

These expose some combinators which allow for summarising rolling statistics across a long stream in constant memory.

coot · 2026-07-17T08:05:39Z

+  , tmV     :: v
+    -- ^ user supplied measure


Consider making it strict, as all the other fields. A similar tsValue of TimedSample is strict.

coot · 2026-07-17T08:09:00Z

+-- | Configured maximum duration of the window.
+windowMaxDuration :: TimedWindow t v a -> Dur t
+windowMaxDuration = twDuration
+{-# INLINE windowMaxDuration #-}


Move this below TimedWindow.

Btw, the INLINE pragma is not needed, as GHC will do that automatically for small functions.

coot · 2026-07-17T08:44:17Z

+data WelfordMeasure a = WelfordMeasure
+    { welfordN    :: {-# UNPACK #-} !Int
+    , welfordMean :: !a
+    , welfordM2   :: !a


welfordM2 is variance, isn't it? Could you add a haddock.

coot · 2026-07-17T08:48:54Z

+                m2    = m2A + m2B + delta * delta
+                            * fromIntegral nA
+                            * fromIntegral nB
+                            / fromIntegral n


The indentation is misleading, * should be aligned with the second +.

coot · 2026-07-17T08:53:54Z

+--
+class (Ord t, Ord (Dur t), Num (Dur t)) => TimeLike t where
+  -- | The duration type associated with @t@.
+  type Dur t = d | d -> t


Duration?

coot · 2026-07-17T09:38:51Z

+-- for correct behaviour (__note__: this is opposite from 'insertMany' and 'fromFoldable')
+-- \(O(\w log w)\)
+--
+fromListN :: (TimeLike t, FT.Measured v b, Coercible a b)


Do we need this if we have fromFoldable?

The implementation could just be fromListN = fromFoldable and we can let GHC to specialise it for a list.

coot · 2026-07-17T10:01:50Z

+--
+data WindowMeasure v = WindowMeasure
+  { wmCount :: {-# UNPACK #-} !Int
+  , wmV     :: v


Why not making it strict in wmV?

coot · 2026-07-17T10:02:44Z

+takeUntil p win@Window { windowTree } = win { windowTree = windowTree' }
+  where
+    windowTree' = FT.takeUntil p windowTree
+    !_ = wmV $ FT.measure windowTree'


if WindowMeasure would be strict in wmV, then we won't need the strict pattern match here.

coot · 2026-07-17T10:08:32Z

+
+-- | Constructs a window with capacity of @windowMaxSize@ containing a single sample
+--
+singleton :: (FT.Measured v b, Coercible a b) => Int -> a -> Window v b


Why do we need the Coercible in the API, can't the user coerce the data by him self between the types (which is less surprising)?

coot · 2026-07-17T10:21:57Z

+                         FT.EmptyR      -> error "impossible"
+                         prefix FT.:> _ -> prefix
+                  else ftree
+    !_ = FT.measure ftree'


This is not needed, FingerTree is strict in the cached measure.

coot

A next batch of modules reviewed, this time the DigestTimeBatched.

coot · 2026-07-17T12:12:35Z

+  deriving (Generic, NFData, Show)
+
+instance Foldable (Window v) where
+  foldMap f Window { windowTree } = foldMap (f . sampleValue) windowTree


Add null, it will be more efficient than the default one.

coot · 2026-07-17T12:13:54Z

+instance Foldable (TimedWindow t v) where
+  foldMap f TimedWindow { twTree } =
+    foldMap (f . tsValue) twTree


Add null implementation via FT.null . twTree

coot · 2026-07-17T14:59:36Z

+                                     , obDigest = TD.insert a obDigest
+                                     , obCount  = succ obCount
+                                     })
+               else (tdwBucket, newBucket)


it would be easier to read if we'd return the sealed backed - then a type signature fillBucket :: (Maybe SealedBucket, OpenBucket) would explain what's going on.

coot · 2026-07-17T15:06:37Z

+      Nothing -> Nothing
+      Just bucket -> if obFinish bucket > cutoff
+                       then tdwBucket win
+                       else assert (FT.null tdwTree') Nothing


🤔 we need to make sure evictBefore is never called when the data from open bucket is not moved to the fingertree while it has entries older than cutoff. For insert to force this invariant, the tdwBucketDuration must be smaller than tdwDuration. This is true unless r passed to empty is positive.

For this to be true it is essential that we have dt < tdwBucketDuration on line 160 above in fillBucket, rather than dt <= tdwBucketDuration.

coot · 2026-07-17T15:15:27Z

+empty tdwBucketDuration r =
+  TimedDigestWindow
+    { tdwDuration = fromIntegral r * tdwBucketDuration
+    , tdwBucketDuration
+    , tdwBucket = Nothing
+    , tdwTree = FT.empty
+    , tdwCacheDigest = Nothing
+    }


Suggested change

empty tdwBucketDuration r =

TimedDigestWindow

{ tdwDuration = fromIntegral r * tdwBucketDuration

, tdwBucketDuration

, tdwBucket = Nothing

, tdwTree = FT.empty

, tdwCacheDigest = Nothing

}

empty _ r | r <= 0 = error "empty: invariant violation, multiplier must be positive"

empty tdwBucketDuration r =

TimedDigestWindow

{ tdwDuration = fromIntegral r * tdwBucketDuration

, tdwBucketDuration

, tdwBucket = Nothing

, tdwTree = FT.empty

, tdwCacheDigest = Nothing

}

See a comment below.

coot · 2026-07-17T15:28:43Z

+    do
+    start  <- start'
+    finish <- obFinish <$> tdwBucket
+    pure $! finish `diffT` start


Why not use an applicative:

Suggested change

do

start <- start'

finish <- obFinish <$> tdwBucket

pure $! finish `diffT` start

diffT <$> (obFinish <$> tdwBucket)

<*> start'

crocodile-dentist self-assigned this Jun 12, 2026

crocodile-dentist requested a review from a team as a code owner June 12, 2026 12:17

github-project-automation Bot added this to Ouroboros Network Jun 12, 2026

github-project-automation Bot moved this to In Progress in Ouroboros Network Jun 12, 2026

crocodile-dentist marked this pull request as draft June 12, 2026 12:17

crocodile-dentist force-pushed the mw/window-stats branch 5 times, most recently from b83890a to 2a8b584 Compare June 17, 2026 20:36

crocodile-dentist marked this pull request as ready for review June 17, 2026 20:37

crocodile-dentist force-pushed the mw/window-stats branch from 2a8b584 to 2f5b77b Compare June 19, 2026 11:46

crocodile-dentist force-pushed the mw/window-stats branch 4 times, most recently from ab338cb to 21db2bc Compare July 7, 2026 12:38

crocodile-dentist force-pushed the mw/window-stats branch from 21db2bc to 84ce2ac Compare July 8, 2026 09:02

window-stats: approximate-quantile backend via tdigest

c674eb2

These modules expose sliding windows backed by an approximate t-digest measure, which permit efficient computations of quantiles in bounded memory.

crocodile-dentist force-pushed the mw/window-stats branch 2 times, most recently from 8f8d92d to 25ee034 Compare July 8, 2026 11:39

crocodile-dentist added 5 commits July 9, 2026 13:01

window-stats: Add cache to TimedDigestWindow

dc435a4

window-stats: expose integration with the foldl package

81557f6

These expose some combinators which allow for summarising rolling statistics across a long stream in constant memory.

window-stats: add property tests

5d6fec3

window-stats: add benchmarks

b29bdd3

window-stats: meta data

a45580f

crocodile-dentist force-pushed the mw/window-stats branch from 25ee034 to a45580f Compare July 9, 2026 11:01

coot reviewed Jul 17, 2026

View reviewed changes

Uh oh!

Conversation

crocodile-dentist commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Quality

Maintenance

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

crocodile-dentist commented Jun 12, 2026 •

edited

Loading