Skip to content

Commit e5f4fbf

Browse files
docs(hydro_lang): restructure reference docs around sliced blocks and mutable singleton references
Reorganize the "Core Framework" reference docs into a new hierarchy and rewrite content to reflect current Hydro APIs (`sliced!` blocks, `by_ref`/`by_mut` reference handles, `BoundedValue` keyed singletons) instead of the legacy `.tick()`-era APIs. New structure under `docs/docs/hydro/reference/`: - `introduction/`: Introduction (keeps `/docs/hydro/reference/` URL via slug), new "Live Collections" concept page (placed first, as the most core concept), and "Dataflow Programming" (refreshed to emphasize that Hydro looks like analytics frameworks but models request/response services) - `correctness/` ("Safety and Correctness"): section index; "Bounded and Unbounded Types" rewritten to decouple boundedness from ticks (Bounded = complete, contents immediately available; unbounded→bounded conversion via `sliced!` hooks); "Eventual Determinism" trimmed to the semantic model, with the guarantee correctly scoped to safe APIs ("Hydro's type system is focused on eventual determinism"); new flagship "Non-Determinism and `nondet!`" page covering guard mechanics, locally-resolved vs forwarded guards, multiple `NonDet` parameters as independently observable forms of non-determinism (Paxos example), proof-style comment guidance, `nondet_` naming and `# Non-Determinism` rustdoc conventions, simulator tie-in, and user-defined-function caveats - `streaming-data/` ("Streaming Data"): section index (request/response framing); "Streams" (refreshed); "Keyed Streams" (hidden doctest harnesses migrated to `sliced!`); new "Keyed Singletons" page focused on the `BoundedValue` variant (one immutable value per request key; `first()`, `fold_early_stop`, map-based request→response, slicing as a batch of new entries), with a boundedness table covering all five bound variants (including MonotonicValue/MonotonicKeys, and Unbounded keys being added *and removed*) and a new request/response animation - `state-management/` ("State Management"): new section index laying out the state philosophy (declarative aggregations → sliced+`by_ref` reads → `use::state`+`by_mut` for multi-input imperative state); "Singletons and Optionals" written from placeholder; new "Keyed State" page (Unbounded `KeyedSingleton`, per-key aggregations, join/lookup patterns); "Slice Blocks" refreshed (leads with the `by_ref` snapshot-reading idiom); new "Slice Hooks" reference (`use` / `use::atomic` / `use::state` / `use::state_null` semantics per collection type, guarantees, unslicing); new "References and Mutations" page (`by_ref`/`by_mut` handles, boundedness requirement, code-order access groups, multi-input shared-state pattern) - `atomic-collections.mdx`: "Atomic Collections" promoted to a top-level doc, de-tick-ified wording, modern sim test API in examples Also update all internal links (`learn/`, `dfir/`, `locations/`), regenerate category files and sidebar positions, update hydro.run doc URLs in `hydro_lang` rustdoc comments to the new paths, and align the single-counter tutorial's eventual-determinism phrasing with the reference docs. Validation: full docusaurus build passes (no broken links); all mdtest doctests pass (62 passing, remainder intentionally no_run/ignore); no legacy APIs (`.tick()`/`.batch(&tick)`/`.snapshot(&tick)`/`.all_ticks()`/ `.latest()`) appear in any visible example code. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #3018
1 parent 36830af commit e5f4fbf

48 files changed

Lines changed: 1538 additions & 333 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/docs/dfir/ecosystem.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ A rough picture of the Hydro stack is below:
1010

1111
Working down from the top:
1212

13-
- [*Hydro*](../hydro/reference/index.mdx) is an end-user-facing high-level [location-oriented](https://en.wikipedia.org/wiki/Choreographic_programming) [dataflow](https://en.wikipedia.org/wiki/Dataflow_programming) framework. Hydro is a *global* framework for programming a fleet of processes. Programmers author dataflow pipelines that start with streams of events and data, and span boundaries across multiple `process` and (scalable) `cluster` specifications. [Hydro Deploy](../hydro/reference/deploy/index.mdx) is a service for launching Hydro programs on a variety of platforms.
13+
- [*Hydro*](../hydro/reference/introduction/index.mdx) is an end-user-facing high-level [location-oriented](https://en.wikipedia.org/wiki/Choreographic_programming) [dataflow](https://en.wikipedia.org/wiki/Dataflow_programming) framework. Hydro is a *global* framework for programming a fleet of processes. Programmers author dataflow pipelines that start with streams of events and data, and span boundaries across multiple `process` and (scalable) `cluster` specifications. [Hydro Deploy](../hydro/reference/deploy/index.mdx) is a service for launching Hydro programs on a variety of platforms.
1414

1515
- *Hydrolysis* is a compiler that translates a global Hydro spec to multiple single-threaded DFIR programs, which collectively implement the global spec.
1616
This compilation phase is currently a part of the Hydro codebase, but will evolve into a standalone optimizing compiler inspired by database query optimizers and [e-graphs](https://en.wikipedia.org/wiki/E-graph).

docs/docs/dfir/quickstart/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,6 @@ This section will get you up and running with Rust and DFIR, and work you throug
33

44
:::tip
55

6-
DFIR is the low-level IR that forms the foundation of the Hydro stack. If you're looking for a higher-level API, check out [Hydro](../../hydro/reference/index.mdx).
6+
DFIR is the low-level IR that forms the foundation of the Hydro stack. If you're looking for a higher-level API, check out [Hydro](../../hydro/reference/introduction/index.mdx).
77

88
:::

docs/docs/hydro/learn/quickstart/concurrent-clients.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ Create a new file `src/concurrent_clients.rs` and add `mod concurrent_clients;`
3030

3131
:::
3232

33-
Hydro provides [`KeyedStream`](../../reference/live-collections/keyed-streams.mdx) for exactly this purpose. A `KeyedStream<K, V>` is like a `Stream<V>`, but each element is associated with a key of type `K`. The key maintains independent ordering per key while allowing different keys to interleave. The key will be the client ID (type `u32`):
33+
Hydro provides [`KeyedStream`](../../reference/streaming-data/keyed-streams.mdx) for exactly this purpose. A `KeyedStream<K, V>` is like a `Stream<V>`, but each element is associated with a key of type `K`. The key maintains independent ordering per key while allowing different keys to interleave. The key will be the client ID (type `u32`):
3434

3535
<CodeBlock language="rust" title={
3636
<Link href="https://github.com/hydro-project/hydro/tree/main/hydro_test/src/tutorials/concurrent_clients.rs">src/concurrent_clients.rs</Link>

docs/docs/hydro/learn/quickstart/index.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ pub fn echo_capitalize<'a>(
7373
) -> Stream<String, Process<'a, EchoServer>> {
7474
`)}</CodeBlock>
7575

76-
Let's dive into the parameters of this function. The input is a [`Stream`](../../reference/live-collections/streams.md), a [**Live Collection**](../../reference/live-collections/index.md) where new incoming requests arrive over time. The first type parameter, `String`, tells us that each incoming request will appear as a `String` element.
76+
Let's dive into the parameters of this function. The input is a [`Stream`](../../reference/streaming-data/streams.md), a [**Live Collection**](../../reference/introduction/live-collections.md) where new incoming requests arrive over time. The first type parameter, `String`, tells us that each incoming request will appear as a `String` element.
7777

7878
The second type parameter identifies the [**`Location`**](../../reference/locations/index.md) for the live collection. Hydro provides two types of locations: [`Process`](../../reference/locations/processes.md) (a single machine) or [`Cluster`](../../reference/locations/clusters.md) (a set of machines). All locations in a Hydro program have the same lifetime `'a`, which is used to enforce borrowing constraints. By writing code that involves several locations, you can create a distributed system!
7979

docs/docs/hydro/learn/quickstart/partitioned-counter.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -194,4 +194,4 @@ Run the test with `cargo test -- partitioned_counter` to verify the implementati
194194

195195
You've completed the quickstart tutorial and built a distributed counter service that scales horizontally. Starting from a single counter, you learned to manage keyed state, regroup streams, and work with distributed properties like ordering guarantees. You then abstracted the logic to work with any location type and partitioned state across a cluster. Throughout, you saw how Hydro's type system enforces correctness properties like atomicity and ordering, even as your service scales from a single process to a distributed cluster.
196196

197-
For more advanced topics and detailed API documentation, check out the [framework reference](../../reference/index.mdx).
197+
For more advanced topics and detailed API documentation, check out the [framework reference](../../reference/introduction/index.mdx).

docs/docs/hydro/learn/quickstart/single-counter.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ pub fn single_client_counter_service<'a>(
5050

5151
## Maintaining State with Singleton
5252

53-
To track the counter value, you need to accumulate state across incoming requests. Hydro provides the [`Singleton`](../../reference/live-collections/singletons-optionals.md) live collection for storing a single value that can be updated over time.
53+
To track the counter value, you need to accumulate state across incoming requests. Hydro provides the [`Singleton`](../../reference/state-management/singletons-optionals.md) live collection for storing a single value that can be updated over time.
5454

5555
You can create a singleton by counting the number of increment requests:
5656

@@ -90,7 +90,7 @@ The `sliced!` macro takes multiple `use` statements, using syntax inspired by Re
9090
- `count_snapshot` is the current value of the counter at the time this batch is processed (a `Singleton<usize>`)
9191

9292
:::info
93-
You'll notice the `nondet!` macro in each `use` statement. Hydro guarantees [**eventual determinism**](../../reference/live-collections/determinism.md): given the same inputs, your program will eventually produce the same outputs. However, some operations involve inherent non-determinism, such the boundaries of a batch or which snapshot of asynchronous state is observed.
93+
You'll notice the `nondet!` macro in each `use` statement. Hydro's type system is focused on [**eventual determinism**](../../reference/correctness/determinism.md): all safe APIs guarantee that given the same inputs, your program will eventually produce the same outputs. However, some operations involve inherent non-determinism, such the boundaries of a batch or which snapshot of asynchronous state is observed.
9494

9595
The `nondet!` macro serves two purposes:
9696
1. It marks points where non-determinism occurs, making them explicit in your code

docs/docs/hydro/reference/slices-atomicity/AtomicConsistencyAnimation.js renamed to docs/docs/hydro/reference/AtomicConsistencyAnimation.js

File renamed without changes.

docs/docs/hydro/reference/slices-atomicity/NonAtomicBugAnimation.js renamed to docs/docs/hydro/reference/NonAtomicBugAnimation.js

File renamed without changes.

docs/docs/hydro/reference/slices-atomicity/atomicity.mdx renamed to docs/docs/hydro/reference/atomic-collections.mdx

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
sidebar_position: 1
2+
sidebar_position: 4
33
---
44

55
import AtomicConsistencyAnimation from './AtomicConsistencyAnimation';
@@ -9,11 +9,11 @@ import NonAtomicBugAnimation from './NonAtomicBugAnimation';
99

1010
When building stateful services, you often need to ensure **consistency** between different outputs. For example, in a counter service, when a client sends an increment request and receives an acknowledgement, a subsequent get request from that client should observe the incremented count.
1111

12-
Without special handling, this guarantee doesn't hold automatically. The acknowledgement might be sent before the count update is visible to get requests, leading to stale reads.
12+
Without special handling, this guarantee doesn't hold automatically. By default, all live collections in Hydro are transformed **asynchronously**, which means there may be arbitrary delays between when a live collection is updated and when downstream transformations see the updates. The acknowledgement might be sent before the count update is visible to get requests, leading to stale reads.
1313

1414
## The Problem: Asynchronous State Updates
1515

16-
Consider a simple counter service that handles increment and get requests:
16+
Consider a simple counter service that handles increment and get requests, with the get requests processed in a [slice block](./state-management/slices.mdx):
1717

1818
```rust,ignore
1919
let current_count = increment_requests.clone().count();
@@ -38,8 +38,8 @@ Hydro provides **atomic collections** to establish consistency relationships bet
3838

3939
Atomic collections work by establishing a **happens-before** relationship:
4040

41-
1. When you call `atomic()` on a live collection, all downstream computations are associated with an internal tick
42-
2. When you call `end_atomic()`, the outputs are held until all computations in the tick complete
41+
1. When you call `atomic()` on a live collection, all downstream computations are associated with an internal synchronization context
42+
2. When you call `end_atomic()`, the outputs are held until all computations in that context complete
4343
3. When you use `use::atomic` in a `sliced!` block, the snapshot is guaranteed to reflect all updates from associated `end_atomic`
4444

4545
This means:
@@ -53,7 +53,7 @@ let increment_ack = increment_request_processing.end_atomic();
5353
```
5454

5555
The key changes are:
56-
1. **`atomic()`**: Marks the live collection as atomic, associating it with an internal tick
56+
1. **`atomic()`**: Marks the live collection as atomic, associating it with an internal synchronization context
5757
2. **`end_atomic()`**: Releases the acknowledgements and establishes a consistency boundary
5858

5959
Now, when you snapshot the count with `use::atomic`, you're guaranteed to observe a version that's consistent with the released acknowledgements:
@@ -91,26 +91,26 @@ Use atomic snapshots when:
9191

9292
## Testing Atomicity
9393

94-
The [Hydro simulator](../simulation/index.mdx) is essential for testing atomic code. It explores different execution orderings to find bugs where atomicity guarantees are violated:
94+
The [Hydro simulator](./simulation/index.mdx) is essential for testing atomic code. It explores different execution orderings to find bugs where atomicity guarantees are violated:
9595

9696
```rust,ignore
9797
#[test]
9898
fn test_read_after_write() {
9999
let mut flow = FlowBuilder::new();
100100
let process = flow.process::<()>();
101-
101+
102102
// ... setup code ...
103-
104-
flow.sim().exhaustive(|sim| async move {
103+
104+
flow.sim().exhaustive(async || {
105105
// Send increment
106-
increment_input.send(()).await;
107-
106+
increment_input.send(());
107+
108108
// Wait for acknowledgement
109-
increment_ack_output.assert_yields(()).await;
110-
109+
increment_ack_output.assert_yields([()]).await;
110+
111111
// Get should observe the increment
112-
get_input.send(()).await;
113-
get_output.assert_yields(1).await;
112+
get_input.send(());
113+
get_output.assert_yields_only([1]).await;
114114
});
115115
}
116116
```
@@ -124,6 +124,6 @@ Atomic collections have important limitations that require special care:
124124

125125
1. **Single location only**: Atomic guarantees only apply within a single process or cluster member. They cannot span multiple locations.
126126

127-
2. **No distributed atomicity**: If you need atomicity across multiple machines, you must implement a distributed coordination protocol (like two-phase commit or Paxos).
127+
2. **No distributed atomicity**: If you need atomicity across multiple machines, you must implement a distributed coordination protocol (like two-phase commit or Paxos). Such protocols can be built in Hydro, but have significant performance implications and will not be introduced without explicit intent.
128128

129129
3. **Performance implications**: Atomic collections introduce synchronization points that can affect throughput. Use them only when consistency is required.
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"label": "Safety and Correctness",
3+
"position": 1,
4+
"link": {
5+
"type": "doc",
6+
"id": "hydro/reference/correctness/index"
7+
}
8+
}

0 commit comments

Comments
 (0)