|
5 | 5 | - 2025-03-24: Initial draft |
6 | 6 | - 2025-04-23: Renumbered from ADR-018 to ADR-019 to maintain chronological order. |
7 | 7 | - 2025-11-10: Updated to reflect actual implementation |
| 8 | +- 2026-02-23: Added sequencer catch-up mode documentation |
8 | 9 |
|
9 | 10 | ## Context |
10 | 11 |
|
@@ -445,6 +446,58 @@ if errors.Is(err, coreda.ErrHeightFromFuture) { |
445 | 446 | } |
446 | 447 | ``` |
447 | 448 |
|
| 449 | +#### Sequencer Catch-Up Mode |
| 450 | + |
| 451 | +When a single sequencer comes back online after downtime spanning multiple DA epochs, it enters **catch-up mode** to ensure consistency with base sequencing behavior. |
| 452 | + |
| 453 | +**Problem**: If the sequencer was offline for several DA epochs, it missed mempool transactions that were submitted during that time. However, forced inclusion transactions were still being posted to DA and processed by full nodes running in base sequencing mode. When the sequencer restarts, it must produce blocks that match what base sequencing would have produced during the downtime. |
| 454 | + |
| 455 | +**Solution**: The sequencer detects if it has fallen more than one epoch behind the DA head and enters catch-up mode: |
| 456 | + |
| 457 | +1. **Detection**: On the first epoch fetch after startup, query `GetLatestDAHeight()` to determine the gap |
| 458 | +2. **Catch-Up Mode**: If more than one epoch behind, enter catch-up mode: |
| 459 | + - Only produce blocks with forced inclusion transactions (no mempool) |
| 460 | + - Use DA epoch end timestamps for block timestamps (to match base sequencing) |
| 461 | +3. **Exit**: When `ErrHeightFromFuture` is encountered (reached DA head), exit catch-up mode and resume normal operation |
| 462 | + |
| 463 | +**Key Behaviors During Catch-Up**: |
| 464 | + |
| 465 | +- **No Mempool Transactions**: Only forced inclusion transactions are included in blocks |
| 466 | +- **Matching Timestamps**: Block timestamps are derived from DA epoch end times to match base sequencing |
| 467 | +- **Checkpoint Persistence**: Progress is tracked via checkpoint to handle crashes during catch-up |
| 468 | +- **Single Check**: The `GetLatestDAHeight()` query is performed only once per sequencer lifecycle |
| 469 | + |
| 470 | +**Example**: |
| 471 | + |
| 472 | +``` |
| 473 | +Sequencer offline during epochs 100-150 (5 epochs of 10 blocks each) |
| 474 | +Full nodes (base sequencing) produced blocks with forced txs only |
| 475 | +
|
| 476 | +Sequencer restarts at epoch 160: |
| 477 | +1. Checkpoint DA height: 100 |
| 478 | +2. Latest DA height: 160 |
| 479 | +3. Missed epochs: 6 (more than 1) |
| 480 | +4. Enter catch-up mode |
| 481 | +
|
| 482 | +Catch-up process: |
| 483 | +- Epoch 101-110: Produce blocks with forced txs only, use epoch timestamps |
| 484 | +- Epoch 111-120: Continue catch-up... |
| 485 | +- ... |
| 486 | +- Epoch 151-160: Still catching up |
| 487 | +- Epoch 161: ErrHeightFromFuture -> exit catch-up mode |
| 488 | +
|
| 489 | +Normal operation resumes: |
| 490 | +- Include both forced txs and mempool txs |
| 491 | +- Use current timestamps |
| 492 | +``` |
| 493 | + |
| 494 | +**Benefits**: |
| 495 | + |
| 496 | +- Ensures sequencer produces identical blocks to what base sequencing would have produced |
| 497 | +- Maintains consistency across the network regardless of sequencer downtime |
| 498 | +- Automatic detection and recovery without operator intervention |
| 499 | +- Safe restart after crashes (checkpoint tracks progress) |
| 500 | + |
448 | 501 | #### Grace Period for Forced Inclusion |
449 | 502 |
|
450 | 503 | The grace period mechanism provides tolerance for chain congestion while maintaining censorship resistance: |
@@ -760,6 +813,7 @@ Accepted and Implemented |
760 | 813 | 9. **Transaction Preservation**: All valid transactions are preserved in queues, nothing is lost |
761 | 814 | 10. **Strict MaxBytes Compliance**: Batches never exceed limits, preventing DA submission failures |
762 | 815 | 11. **DA Fault Tolerance**: Grace period prevents false positives during temporary chain congestion |
| 816 | +12. **Automatic Recovery**: Sequencer catch-up mode ensures consistency after downtime without operator intervention |
763 | 817 |
|
764 | 818 | ### Negative |
765 | 819 |
|
|
0 commit comments