Skip to content

Commit 2342038

Browse files
edwardaleeedwardalee
committed
Many updates. See email
1 parent bee6caf commit 2342038

12 files changed

Lines changed: 265 additions & 234 deletions

‎03-timestamps.md‎

Lines changed: 63 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ See [`src/DistibutedPowerGrid3_TimeStamped.lf`](src/DistibutedPowerGrid3_TimeSta
5151

5252
Key differences from Step 2:
5353
1. Connections use `->` instead of `~>`
54-
2. `GridManager` has an `STA` parameter
54+
2. `GridManager` has a `maxwait` attribute
5555
3. No other changes to the reactor logic — timestamp ordering is handled by the LF runtime
5656

5757
---
@@ -80,40 +80,90 @@ Because the connection wiring is symmetric (California always feeds `in1`, New Y
8080
## The Cost: Waiting
8181

8282
To process a message at timestamp `t`, the grid manager must be sure it has received **all** messages with timestamp ≤ `t`. Otherwise, a late-arriving message (from the other node) could violate timestamp order.
83+
Such a late-arriving message is said to be **tardy**.
8384

84-
This creates an unavoidable wait. The LF **decentralized coordinator** manages this via the **STA (Safe To Advance)** parameter:
85+
This creates an unavoidable wait. The LF **decentralized coordinator** manages this via the **maxwait** attribute:
8586

8687
```lf
87-
reactor GridManager(STA: time = 100 ms) {
88-
// ...
89-
}
88+
@maxwait(100 ms)
89+
gm1 = new GridManager()
9090
```
9191

92-
A grid manager with `STA = 100 ms` waits until its local physical clock reads `T ≥ t + 100 ms` before processing a message at logical time `t`. This ensures that any message from the remote node with timestamp less than `t` has had at least 100 ms to arrive.
92+
A grid manager with `maxwait = 100 ms` advances to a logical time `t` when either:
93+
1. It has received inputs with timestamps `t` or more on both inputs, or
94+
2. Its local physical clock reads `T ≥ t + 100 ms`.
95+
96+
This ensures that any message from the remote node with timestamp less than `t` has had at least 100 ms to arrive.
9397

9498
This is correct as long as:
9599

96-
> **clock sync error + network latency ≤ STA**
100+
> **clock sync error + network latency ≤ maxwait**
101+
102+
For two nodes in California and New York (cross-continental latency ~60–80 ms), a `maxwait` of 100 ms provides a modest safety margin with NTP synchronization (~10–50 ms error). Google Spanner achieves tighter bounds using GPS, the precision time protocol (PTP), and dedicated fiber trunks.
103+
104+
---
105+
106+
## Handling Faults
107+
108+
Tardy events, if not handled, result in messages like this:
109+
110+
```
111+
Fed 1 (op2_main): ERROR: STP violation occurred in a trigger to reaction 3, and there is no handler.
112+
**** Invoking reaction at the wrong tag!
113+
```
114+
115+
Tardy events may be handled with a **tardy handler**.
116+
For example, we can add the following to the `GridManager` reaction:
117+
118+
```lf
119+
reaction(in1, in2) -> out {=
120+
...
121+
=} tardy {=
122+
tag_t intended_tag;
123+
if (in1->is_present) {
124+
intended_tag = in1->intended_tag;
125+
} else {
126+
intended_tag = in2->intended_tag;
127+
}
128+
lf_print_warning("[ts=%lld] Tardy message with intended timestamp %lld received at physical time %lld",
129+
lf_time_logical_elapsed(), intended_tag.time - lf_time_start(), lf_time_physical_elapsed());
130+
=}
131+
```
132+
133+
This handler will be invoked _instead of_ the normal reaction when a tardy message arrives.
134+
This example shows how to extract the **intended tag**, which is a timestamp, microstep pair.
135+
For this grid application, merely printing a warning like this is probably not the right thing to do.
136+
What could you do better?
97137

98-
For two nodes in California and New York (cross-continental latency ~60–80 ms), an STA of 100 ms provides a modest safety margin with NTP synchronization (~10–50 ms error). Google Spanner achieves tighter bounds using GPS and dedicated fiber.
138+
139+
In some cases, it is actually OK to handle tardy messages just like ordinary messages.
140+
The `GridInterface` reactor used to generate test cases is an example of this.
141+
Its `status` input is used to report the balance as viewed by the local GridManager.
142+
If it is OK for these reports to be made late, then we can annotate the reaction with an empty `tardy` handler, as follows:
143+
144+
```
145+
reaction(status) {=
146+
lf_print("%s balance: %d MW", self->node_name, status->value);
147+
=} tardy // Handle tardy messages like any other message.
148+
```
99149

100150
---
101151

102-
## The CAL Theorem Preview
152+
## The CAL Theorem
103153

104-
The waiting introduced by timestamps is not a bug — it is **fundamental**. The **CAL theorem** (Lee et al., 2023) states:
154+
The waiting introduced by timestamps is not a bug — it is **fundamental**. The **CAL theorem** ([Lee et al., 2023](https://doi.org/10.1145/3609119)) states:
105155

106156
> It is impossible to achieve consistency without paying a price in **availability**, where the price is proportional to the latencies in the system.
107157
108-
"Availability" here means: how long must an operator wait before their dispatch command takes effect? The longer the STA, the more consistent the system — and the longer operators wait. We'll return to this in Step 6.
158+
"Availability" here means: how long must an operator wait before their dispatch command takes effect? The longer the `maxwait`, the more consistent the system — and the longer operators wait. We'll return to this in Step 6.
109159

110160
---
111161

112162
## Exercises
113163

114-
1. With `STA = 100 ms` and cross-continental latency of 75 ms, what is the maximum clock synchronization error you can tolerate and still guarantee correct ordering?
164+
1. With `maxwait = 100 ms` and cross-continental latency of 75 ms, what is the maximum clock synchronization error you can tolerate and still guarantee correct ordering?
115165

116-
2. If you reduced STA to 20 ms to improve responsiveness, what would happen if a message arrived 30 ms late? What would the LF runtime do (hint: see the `fault handler` concept)?
166+
2. If you reduced maxwait to 20 ms to improve responsiveness, what would happen if a message arrived 30 ms late? What would the LF runtime do?
117167

118168
3. Revisit the inconsistency scenario from Step 2 (balance = −150 MW, simultaneous curtail and dispatch). Trace through the execution with timestamps. Do both grid managers reach the same balance?
119169

‎04-conservative.md‎

Lines changed: 48 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,61 +1,80 @@
1-
# Step 4: Conservative Coordination — Chandy-Misra Null Messages
1+
# Step 4: Conservative Coordination — Chandy-Misra with Null Messages
22

3-
## The Problem with Finite STA
3+
## The Problem with Finite maxwait
44

5-
In Step 3, we set `STA = 100 ms`. This means:
5+
In Step 3, we set `maxwait = 100 ms`. This means:
66

7-
- The grid manager waits 100 ms after logical time `t` before processing a message at `t`.
8-
- This is safe *if and only if* every remote message arrives within 100 ms of its timestamp.
7+
- The grid manager may wait 100 ms after logical time `t` before processing a message at `t`.
8+
- This is safe (avoids tardy messages) if every remote message arrives within 100 ms of its timestamp.
99

10-
But what if the link between California and New York goes down for 5 minutes? Or a router becomes congested and introduces 500 ms of latency? The `STA = 100 ms` assumption is violated. The LF runtime will invoke the **fault handler** — but we need to have designed one.
10+
But what if the link between California and New York goes down for 5 minutes? Or a router becomes congested and introduces 500 ms of latency? The `maxwait = 100 ms` assumption is violated.
11+
If a local event has caused logical time to advance, then when the message finally arrives, it will be tardy,
12+
and the LF runtime will invoke the **tardy handler**.
1113

12-
A more principled approach avoids making assumptions about latency altogether.
14+
An alternative approach avoids making assumptions about latency altogether.
1315

1416
---
1517

1618
## Conservative Coordination: Wait Until You Know
1719

1820
Instead of guessing that all messages will arrive within some time budget, the **conservative approach** says:
1921

20-
> A node must not process a message at logical time `t` until it has proof that no earlier message is coming from any connected node.
22+
> A node must not process a message at tag `g` until it has proof that no message with tag `g` or less will later arrive.
2123
22-
This is the Chandy-Misra algorithm (1979), originally developed for distributed simulation. LF supports it via `STA = forever`:
24+
In LF, messages on a connection from one federate to another are delivered reliably in order.
25+
Moreover, all messages on such a connection have strictly increasing tags, where a **tag** is a timestamp, microstep pair, `g = (t, m)`.
26+
Hence, when a message has arrived on an input port with some tag `g`, the receiving federate knows it has already received all messages with earlier tags on this input port.
27+
28+
The [Chandy-Misra approach (1979)](https://doi.org/10.1109/TSE.1979.230182), originally developed for distributed simulation, waits to advance to a tag `g` until a message has been received on **every** input port with tag at least `g`.
29+
LF supports this method via `maxwait = forever`:
2330

2431
```lf
25-
reactor GridManager(STA: time = forever) {
26-
// ...
27-
}
32+
@maxwait(forever)
33+
gm1 = new GridManager()
2834
```
2935

30-
With `STA = forever`, the grid manager will wait **indefinitely** — until it receives positive evidence from the remote node that no message with timestamp ≤ `t` is coming.
36+
With `maxwait = forever`, the grid manager will wait **indefinitely** — until it receives positive evidence from the remote node that no message with tag ≤ `g` is coming.
3137

38+
Of course, waiting forever sounds like a bad idea.
39+
What if the remote `GridInterface`
3240
How does that evidence arrive? That's where **null messages** come in.
3341

3442
---
3543

3644
## Null Messages
3745

38-
In our grid, commands are sent only when operators issue dispatches or curtailments. If California issues no commands for 10 minutes, New York's manager has no evidence about California's timeline — and blocks forever under `STA = forever`.
46+
In our grid, commands are sent only when operators issue dispatches or curtailments. If California issues no commands for 10 minutes, New York's manager has no evidence about California's timeline — and blocks forever under `maxwait = forever`.
3947

4048
The fix: California's node sends **null messages** periodically. A null message says:
4149

4250
> "I have no real command at this timestamp, but here is my current timestamp. You may safely advance past it."
4351
44-
In the grid context, a null message with value `0` means "no change in dispatch this QuickDispatchsecond." It is a heartbeat that lets the remote manager advance logical time even during quiet periods.
52+
In the grid context, a null message with value `0` can be used as "no change in dispatch." It is a heartbeat that lets the remote manager advance logical time even during quiet periods.
53+
We create a `GridServer` that wraps the `OperatorConsole` and sends null messages periodically when the console has nothing to say:
4554

4655
```lf
4756
reactor GridServer(null_message_period: time = 1 s) {
48-
input in: int
49-
output received: int
50-
timer t(0, null_message_period)
51-
52-
reaction(w.received, t) -> received {=
53-
if (w.received->is_present) {
54-
lf_set(received, w.received->value); // real command
55-
} else {
56-
lf_set(received, 0); // null message: no dispatch this tick
57-
}
58-
=}
57+
input balance_in: int // balance feedback from local GridManager
58+
output command_out: int // real or null command, forwarded to GridManagers
59+
60+
console = new OperatorConsole()
61+
62+
timer heartbeat(0, null_message_period) // Null-message heartbeat timer: fires every null_message_period
63+
64+
// On each heartbeat: forward real command if present, else null message
65+
reaction(console.command, heartbeat) -> command_out {=
66+
if (console.command->is_present) {
67+
lf_set(command_out, console.command->value);
68+
} else {
69+
// Null message: advances remote logical time, no grid effect
70+
lf_set(command_out, 0);
71+
}
72+
=}
73+
74+
// Print balance to operator display
75+
reaction(balance_in) {=
76+
lf_print("Grid balance = %d MW", balance_in->value);
77+
=} tardy // Handle tardy messages like any other message.
5978
}
6079
```
6180

@@ -66,7 +85,7 @@ The timer fires every second. If no real command has arrived, a `0` is forwarded
6685
## Guarantees and Costs
6786

6887
**What you get:**
69-
- **Strong consistency**: both grid managers agree on the balance at every logical timestamp. No STA violations, ever.
88+
- **Strong consistency**: both grid managers agree on the balance at every logical timestamp. No tardy violations, ever.
7089
- **Bounded wait**: the maximum wait is ~1 second (the null message period) plus network latency.
7190

7291
**What you give up:**
@@ -87,7 +106,7 @@ And here is what our system looks like:
87106

88107
Key changes from Step 3:
89108
- `GridInterface` is wrapped in `GridServer`, which adds a null-message timer.
90-
- `GridManager` has `STA = forever`.
109+
- `GridManager` has `maxwait = forever`.
91110
- Logical connections (`->`) are retained.
92111

93112
---
@@ -98,7 +117,7 @@ Key changes from Step 3:
98117

99118
2. If we reduce the null message period from 1 s to 100 ms, how does this affect (a) wait time, (b) network overhead, and (c) resilience to node failure?
100119

101-
3. Could we use `STA = forever` with *no* null messages at all? Under what circumstances would the system make progress?
120+
3. Could we use `maxwait = forever` with *no* null messages at all? Under what circumstances would the system make progress?
102121

103122
---
104123

0 commit comments

Comments
 (0)