You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(hydro_lang): restructure reference docs around sliced blocks and mutable singleton references
Reorganize the "Core Framework" reference docs into a new hierarchy and
rewrite content to reflect current Hydro APIs (`sliced!` blocks,
`by_ref`/`by_mut` reference handles, `BoundedValue` keyed singletons)
instead of the legacy `.tick()`-era APIs.
New structure under `docs/docs/hydro/reference/`:
- `introduction/`: Introduction (keeps `/docs/hydro/reference/` URL via
slug), new "Live Collections" concept page (placed first, as the most
core concept), and "Dataflow Programming" (refreshed to emphasize that
Hydro looks like analytics frameworks but models request/response
services)
- `correctness/` ("Safety and Correctness"): section index; "Bounded and
Unbounded Types" rewritten to decouple boundedness from ticks (Bounded =
complete, contents immediately available; unbounded→bounded conversion
via `sliced!` hooks); "Eventual Determinism" trimmed to the semantic
model, with the guarantee correctly scoped to safe APIs ("Hydro's type
system is focused on eventual determinism"); new flagship
"Non-Determinism and `nondet!`" page covering guard mechanics,
locally-resolved vs forwarded guards, multiple `NonDet` parameters as
independently observable forms of non-determinism (Paxos example),
proof-style comment guidance, `nondet_` naming and `# Non-Determinism`
rustdoc conventions, simulator tie-in, and user-defined-function caveats
- `streaming-data/` ("Streaming Data"): section index (request/response
framing); "Streams" (refreshed); "Keyed Streams" (hidden doctest
harnesses migrated to `sliced!`); new "Keyed Singletons" page focused on
the `BoundedValue` variant (one immutable value per request key;
`first()`, `fold_early_stop`, map-based request→response, slicing as a
batch of new entries), with a boundedness table covering all five bound
variants (including MonotonicValue/MonotonicKeys, and Unbounded keys
being added *and removed*) and a new request/response animation
- `state-management/` ("State Management"): new section index laying out
the state philosophy (declarative aggregations → sliced+`by_ref` reads →
`use::state`+`by_mut` for multi-input imperative state); "Singletons and
Optionals" written from placeholder; new "Keyed State" page (Unbounded
`KeyedSingleton`, per-key aggregations, join/lookup patterns); "Slice
Blocks" refreshed (leads with the `by_ref` snapshot-reading idiom); new
"Slice Hooks" reference (`use` / `use::atomic` / `use::state` /
`use::state_null` semantics per collection type, guarantees, unslicing);
new "References and Mutations" page (`by_ref`/`by_mut` handles,
boundedness requirement, code-order access groups, multi-input
shared-state pattern)
- `atomic-collections.mdx`: "Atomic Collections" promoted to a top-level
doc, de-tick-ified wording, modern sim test API in examples
Also update all internal links (`learn/`, `dfir/`, `locations/`),
regenerate category files and sidebar positions, update hydro.run doc
URLs in `hydro_lang` rustdoc comments to the new paths, and align the
single-counter tutorial's eventual-determinism phrasing with the
reference docs.
Validation: full docusaurus build passes (no broken links); all mdtest
doctests pass (62 passing, remainder intentionally no_run/ignore); no
legacy APIs (`.tick()`/`.batch(&tick)`/`.snapshot(&tick)`/`.all_ticks()`/
`.latest()`) appear in any visible example code.
Co-authored-by: Infinity 🤖 <infinity@hydro.run>
PR: #3018
Copy file name to clipboardExpand all lines: docs/docs/dfir/ecosystem.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,7 @@ A rough picture of the Hydro stack is below:
10
10
11
11
Working down from the top:
12
12
13
-
-[*Hydro*](../hydro/reference/index.mdx) is an end-user-facing high-level [location-oriented](https://en.wikipedia.org/wiki/Choreographic_programming)[dataflow](https://en.wikipedia.org/wiki/Dataflow_programming) framework. Hydro is a *global* framework for programming a fleet of processes. Programmers author dataflow pipelines that start with streams of events and data, and span boundaries across multiple `process` and (scalable) `cluster` specifications. [Hydro Deploy](../hydro/reference/deploy/index.mdx) is a service for launching Hydro programs on a variety of platforms.
13
+
-[*Hydro*](../hydro/reference/introduction/index.mdx) is an end-user-facing high-level [location-oriented](https://en.wikipedia.org/wiki/Choreographic_programming)[dataflow](https://en.wikipedia.org/wiki/Dataflow_programming) framework. Hydro is a *global* framework for programming a fleet of processes. Programmers author dataflow pipelines that start with streams of events and data, and span boundaries across multiple `process` and (scalable) `cluster` specifications. [Hydro Deploy](../hydro/reference/deploy/index.mdx) is a service for launching Hydro programs on a variety of platforms.
14
14
15
15
-*Hydrolysis* is a compiler that translates a global Hydro spec to multiple single-threaded DFIR programs, which collectively implement the global spec.
16
16
This compilation phase is currently a part of the Hydro codebase, but will evolve into a standalone optimizing compiler inspired by database query optimizers and [e-graphs](https://en.wikipedia.org/wiki/E-graph).
Copy file name to clipboardExpand all lines: docs/docs/dfir/quickstart/index.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,6 +3,6 @@ This section will get you up and running with Rust and DFIR, and work you throug
3
3
4
4
:::tip
5
5
6
-
DFIR is the low-level IR that forms the foundation of the Hydro stack. If you're looking for a higher-level API, check out [Hydro](../../hydro/reference/index.mdx).
6
+
DFIR is the low-level IR that forms the foundation of the Hydro stack. If you're looking for a higher-level API, check out [Hydro](../../hydro/reference/introduction/index.mdx).
Copy file name to clipboardExpand all lines: docs/docs/hydro/learn/quickstart/concurrent-clients.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -30,7 +30,7 @@ Create a new file `src/concurrent_clients.rs` and add `mod concurrent_clients;`
30
30
31
31
:::
32
32
33
-
Hydro provides [`KeyedStream`](../../reference/live-collections/keyed-streams.mdx) for exactly this purpose. A `KeyedStream<K, V>` is like a `Stream<V>`, but each element is associated with a key of type `K`. The key maintains independent ordering per key while allowing different keys to interleave. The key will be the client ID (type `u32`):
33
+
Hydro provides [`KeyedStream`](../../reference/streaming-data/keyed-streams.mdx) for exactly this purpose. A `KeyedStream<K, V>` is like a `Stream<V>`, but each element is associated with a key of type `K`. The key maintains independent ordering per key while allowing different keys to interleave. The key will be the client ID (type `u32`):
Copy file name to clipboardExpand all lines: docs/docs/hydro/learn/quickstart/index.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -73,7 +73,7 @@ pub fn echo_capitalize<'a>(
73
73
) -> Stream<String, Process<'a, EchoServer>> {
74
74
`)}</CodeBlock>
75
75
76
-
Let's dive into the parameters of this function. The input is a [`Stream`](../../reference/live-collections/streams.md), a [**Live Collection**](../../reference/live-collections/index.md) where new incoming requests arrive over time. The first type parameter, `String`, tells us that each incoming request will appear as a `String` element.
76
+
Let's dive into the parameters of this function. The input is a [`Stream`](../../reference/streaming-data/streams.md), a [**Live Collection**](../../reference/introduction/live-collections.md) where new incoming requests arrive over time. The first type parameter, `String`, tells us that each incoming request will appear as a `String` element.
77
77
78
78
The second type parameter identifies the [**`Location`**](../../reference/locations/index.md) for the live collection. Hydro provides two types of locations: [`Process`](../../reference/locations/processes.md) (a single machine) or [`Cluster`](../../reference/locations/clusters.md) (a set of machines). All locations in a Hydro program have the same lifetime `'a`, which is used to enforce borrowing constraints. By writing code that involves several locations, you can create a distributed system!
Copy file name to clipboardExpand all lines: docs/docs/hydro/learn/quickstart/partitioned-counter.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -194,4 +194,4 @@ Run the test with `cargo test -- partitioned_counter` to verify the implementati
194
194
195
195
You've completed the quickstart tutorial and built a distributed counter service that scales horizontally. Starting from a single counter, you learned to manage keyed state, regroup streams, and work with distributed properties like ordering guarantees. You then abstracted the logic to work with any location type and partitioned state across a cluster. Throughout, you saw how Hydro's type system enforces correctness properties like atomicity and ordering, even as your service scales from a single process to a distributed cluster.
196
196
197
-
For more advanced topics and detailed API documentation, check out the [framework reference](../../reference/index.mdx).
197
+
For more advanced topics and detailed API documentation, check out the [framework reference](../../reference/introduction/index.mdx).
To track the counter value, you need to accumulate state across incoming requests. Hydro provides the [`Singleton`](../../reference/live-collections/singletons-optionals.md) live collection for storing a single value that can be updated over time.
53
+
To track the counter value, you need to accumulate state across incoming requests. Hydro provides the [`Singleton`](../../reference/state-management/singletons-optionals.md) live collection for storing a single value that can be updated over time.
54
54
55
55
You can create a singleton by counting the number of increment requests:
56
56
@@ -90,7 +90,7 @@ The `sliced!` macro takes multiple `use` statements, using syntax inspired by Re
90
90
-`count_snapshot` is the current value of the counter at the time this batch is processed (a `Singleton<usize>`)
91
91
92
92
:::info
93
-
You'll notice the `nondet!` macro in each `use` statement. Hydro guarantees [**eventual determinism**](../../reference/live-collections/determinism.md): given the same inputs, your program will eventually produce the same outputs. However, some operations involve inherent non-determinism, such the boundaries of a batch or which snapshot of asynchronous state is observed.
93
+
You'll notice the `nondet!` macro in each `use` statement. Hydro's type system is focused on [**eventual determinism**](../../reference/correctness/determinism.md): all safe APIs guarantee that given the same inputs, your program will eventually produce the same outputs. However, some operations involve inherent non-determinism, such the boundaries of a batch or which snapshot of asynchronous state is observed.
94
94
95
95
The `nondet!` macro serves two purposes:
96
96
1. It marks points where non-determinism occurs, making them explicit in your code
@@ -9,11 +9,11 @@ import NonAtomicBugAnimation from './NonAtomicBugAnimation';
9
9
10
10
When building stateful services, you often need to ensure **consistency** between different outputs. For example, in a counter service, when a client sends an increment request and receives an acknowledgement, a subsequent get request from that client should observe the incremented count.
11
11
12
-
Without special handling, this guarantee doesn't hold automatically. The acknowledgement might be sent before the count update is visible to get requests, leading to stale reads.
12
+
Without special handling, this guarantee doesn't hold automatically. By default, all live collections in Hydro are transformed **asynchronously**, which means there may be arbitrary delays between when a live collection is updated and when downstream transformations see the updates. The acknowledgement might be sent before the count update is visible to get requests, leading to stale reads.
13
13
14
14
## The Problem: Asynchronous State Updates
15
15
16
-
Consider a simple counter service that handles increment and get requests:
16
+
Consider a simple counter service that handles increment and get requests, with the get requests processed in a [slice block](./state-management/slices.mdx):
17
17
18
18
```rust,ignore
19
19
let current_count = increment_requests.clone().count();
@@ -38,8 +38,8 @@ Hydro provides **atomic collections** to establish consistency relationships bet
38
38
39
39
Atomic collections work by establishing a **happens-before** relationship:
40
40
41
-
1. When you call `atomic()` on a live collection, all downstream computations are associated with an internal tick
42
-
2. When you call `end_atomic()`, the outputs are held until all computations in the tick complete
41
+
1. When you call `atomic()` on a live collection, all downstream computations are associated with an internal synchronization context
42
+
2. When you call `end_atomic()`, the outputs are held until all computations in that context complete
43
43
3. When you use `use::atomic` in a `sliced!` block, the snapshot is guaranteed to reflect all updates from associated `end_atomic`
44
44
45
45
This means:
@@ -53,7 +53,7 @@ let increment_ack = increment_request_processing.end_atomic();
53
53
```
54
54
55
55
The key changes are:
56
-
1.**`atomic()`**: Marks the live collection as atomic, associating it with an internal tick
56
+
1.**`atomic()`**: Marks the live collection as atomic, associating it with an internal synchronization context
57
57
2.**`end_atomic()`**: Releases the acknowledgements and establishes a consistency boundary
58
58
59
59
Now, when you snapshot the count with `use::atomic`, you're guaranteed to observe a version that's consistent with the released acknowledgements:
@@ -91,26 +91,26 @@ Use atomic snapshots when:
91
91
92
92
## Testing Atomicity
93
93
94
-
The [Hydro simulator](../simulation/index.mdx) is essential for testing atomic code. It explores different execution orderings to find bugs where atomicity guarantees are violated:
94
+
The [Hydro simulator](./simulation/index.mdx) is essential for testing atomic code. It explores different execution orderings to find bugs where atomicity guarantees are violated:
95
95
96
96
```rust,ignore
97
97
#[test]
98
98
fn test_read_after_write() {
99
99
let mut flow = FlowBuilder::new();
100
100
let process = flow.process::<()>();
101
-
101
+
102
102
// ... setup code ...
103
-
104
-
flow.sim().exhaustive(|sim| async move {
103
+
104
+
flow.sim().exhaustive(async || {
105
105
// Send increment
106
-
increment_input.send(()).await;
107
-
106
+
increment_input.send(());
107
+
108
108
// Wait for acknowledgement
109
-
increment_ack_output.assert_yields(()).await;
110
-
109
+
increment_ack_output.assert_yields([()]).await;
110
+
111
111
// Get should observe the increment
112
-
get_input.send(()).await;
113
-
get_output.assert_yields(1).await;
112
+
get_input.send(());
113
+
get_output.assert_yields_only([1]).await;
114
114
});
115
115
}
116
116
```
@@ -124,6 +124,6 @@ Atomic collections have important limitations that require special care:
124
124
125
125
1.**Single location only**: Atomic guarantees only apply within a single process or cluster member. They cannot span multiple locations.
126
126
127
-
2.**No distributed atomicity**: If you need atomicity across multiple machines, you must implement a distributed coordination protocol (like two-phase commit or Paxos).
127
+
2.**No distributed atomicity**: If you need atomicity across multiple machines, you must implement a distributed coordination protocol (like two-phase commit or Paxos). Such protocols can be built in Hydro, but have significant performance implications and will not be introduced without explicit intent.
128
128
129
129
3.**Performance implications**: Atomic collections introduce synchronization points that can affect throughput. Use them only when consistency is required.
0 commit comments