Skip to content

Commit f4bc2f9

Browse files
dylankilkennydylankilkenny
authored
feat: Fast delayed resubmit for Stellar insufficient-fee and TRY_AGAIN_LATER (#822)
* feat: Stellar fast resubmit on fee/TAL rejection Insufficient-fee and TRY_AGAIN_LATER rejections now schedule their own delayed resubmission instead of waiting for the status checker. - submit_core schedules a delayed resubmit (5s × attempt) on txInsufficientFee (cap raised 2→6) and the first 3 TRY_AGAIN_LATER rejections; the status-checker ladder remains the fallback rescue - enqueue failures are logged and swallowed — submission outcome unchanged - transaction_submission_queue moved to the high-frequency Redis profile: its 20s enqueue_scheduled sweep quantized the delays and let the status checker double-fire (this feature is the first user of scheduled_on there) - e2e harness: chained WireMock rejection mappings + runner with a 5-scenario timing matrix, channels prod-parity profile (SQS standard/FIFO via LocalStack), and closed-loop soak with residual-based scoring Signed-off-by: Dylan Kilkenny <dylankilkenny95@gmail.com> * test: Slim e2e fast-resubmit harness Keep the repeatable regression core: the 5-scenario timing matrix with measured-gap assertions. Drop the one-off prod-parity and soak tooling used during development (results documented on the PR). Replace the 9 near-duplicate chained mapping files with dynamic stub registration against the WireMock admin API at arm time. The chains decrement into the existing one-shot armed state, so any rejection count works without a hardcoded ceiling. Runner: 2146 -> 565 lines. Matrix re-verified 5/5 after the slim. Signed-off-by: Dylan Kilkenny <dylankilkenny95@gmail.com> * fix: Address PR review findings - Recheck the insufficient-fee cap on the post-increment count before scheduling a fast resubmit. Racing submit jobs for the same transaction could both pass the pre-check; the recheck closes that window and fails a doomed transaction one cycle earlier. New unit test covers it. - Tighten the delay-escalation test to assert the exact 5s-per-attempt formula instead of a 10-20s window. - Bootstrap the e2e harness keystore when missing (generate and fund via friendbot), so the harness runs from a fresh clone. Verified by moving the local keystore aside and rerunning the matrix. Signed-off-by: Dylan Kilkenny <dylankilkenny95@gmail.com> * fix: Skip fast resubmit on post-record cap race When two submit jobs race, the loser's post-record count exceeds the cap. Failing there would clobber the winner's already-scheduled retry, creating a failure mode the passive path does not have. Skip the enqueue instead: the cap stays airtight and the next rejection's pre-record check terminates the transaction normally. Signed-off-by: Dylan Kilkenny <dylankilkenny95@gmail.com> * fix: Final-state guard; harness scenario check - If a transaction reached a final state between the RPC rejection and the atomic retry record (racing job finalized it), the repo returns it unchanged. Both rejection arms now skip scheduling in that case instead of enqueueing a pointless resubmit. Unit tests for both arms. - The e2e runner now fails fast (exit 64) on an invalid scenario name instead of booting the stack and exiting green with zero scenarios; scenario validation happens before any infrastructure starts. Signed-off-by: Dylan Kilkenny <dylankilkenny95@gmail.com> --------- Signed-off-by: Dylan Kilkenny <dylankilkenny95@gmail.com>
1 parent 0b9d236 commit f4bc2f9

7 files changed

Lines changed: 1425 additions & 39 deletions

File tree

‎src/constants/stellar_transaction.rs‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,16 @@ pub const STELLAR_MAX_STUCK_TRANSACTION_LIFETIME_MINUTES: i64 = 15;
7272
pub const STELLAR_RESUBMIT_BASE_INTERVAL_SECONDS: i64 = 10;
7373

7474
/// Maximum number of times a Stellar submission may be retried after an insufficient-fee error.
75-
pub const STELLAR_INSUFFICIENT_FEE_MAX_RETRIES: u32 = 2;
75+
/// Raised to cover multiple fast resubmits across short-lived fee spikes.
76+
pub const STELLAR_INSUFFICIENT_FEE_MAX_RETRIES: u32 = 6;
77+
78+
/// Base delay for fast Stellar resubmits, in seconds.
79+
/// Five seconds is approximately one Stellar ledger close time (`STELLAR_LEDGER_TIME_SECONDS`).
80+
pub const STELLAR_FAST_RESUBMIT_BASE_DELAY_SECONDS: i64 = 5;
81+
82+
/// Maximum TRY_AGAIN_LATER rejections that use the fast-resubmit window.
83+
/// After the first three fast attempts, the status-checker backoff ladder owns rescue.
84+
pub const STELLAR_TRY_AGAIN_LATER_FAST_RETRIES: u32 = 3;
7685

7786
/// Maximum resubmit interval (seconds) to cap the resubmission backoff.
7887
/// Prevents excessively long gaps between resubmissions.

‎src/domain/transaction/stellar/submit.rs‎

Lines changed: 620 additions & 35 deletions
Large diffs are not rendered by default.

‎src/queues/redis/queue.rs‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -227,8 +227,8 @@ impl Queue {
227227
.unwrap_or_default();
228228

229229
// Queue configurations:
230-
// - High-frequency: transaction_status (critical path)
231-
// - Low-frequency: request, submission, notifications, health checks, swaps
230+
// - High-frequency: transaction_submission, transaction_status (critical path)
231+
// - Low-frequency: request, notifications, health checks, swaps
232232
let high_frequency = QueueConfig::high_frequency();
233233
let low_frequency = QueueConfig::low_frequency();
234234

@@ -241,7 +241,9 @@ impl Queue {
241241
transaction_submission_queue: Self::storage(
242242
&format!("{redis_key_prefix}transaction_submission_queue"),
243243
conn_tx_submit,
244-
low_frequency.clone(), // scheduling not used
244+
// Stellar fast resubmit schedules 5s*attempt delayed submit jobs here.
245+
// A 20s sweep quantizes those retries and lets the status checker double-fire.
246+
high_frequency.clone(),
245247
),
246248
transaction_status_queue: Self::storage(
247249
&format!("{redis_key_prefix}transaction_status_queue"),

‎testing/wiremock/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
e2e/env.*
2+
e2e/*.keystore
3+
e2e/**/*.keystore
4+
e2e/generated-relayers/

‎testing/wiremock/README.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,9 +81,89 @@ All scenarios fire **exactly once**, then revert to proxying real traffic.
8181
|--------|-------------|
8282
| `trigger-try-again-later.sh` | Arm the TRY_AGAIN_LATER scenario |
8383
| `trigger-insufficient-fee.sh` | Arm the insufficient fee scenario |
84+
| `e2e-fast-resubmit.sh` | Run the Stellar fast-resubmit E2E matrix |
8485
| `check-scenarios.sh` | Show scenario states and recent requests |
8586
| `reset-all.sh` | Reset all scenarios and clear request log |
8687

88+
## E2E fast-resubmit verification
89+
90+
Run the Stellar fast-resubmit matrix from the repo root:
91+
92+
```bash
93+
./testing/wiremock/scripts/e2e-fast-resubmit.sh all
94+
./testing/wiremock/scripts/e2e-fast-resubmit.sh -o /tmp/fast-resubmit-artifacts all
95+
```
96+
97+
Run one scenario by name:
98+
99+
```bash
100+
./testing/wiremock/scripts/e2e-fast-resubmit.sh if-once
101+
./testing/wiremock/scripts/e2e-fast-resubmit.sh if-escalation
102+
./testing/wiremock/scripts/e2e-fast-resubmit.sh if-overcap
103+
./testing/wiremock/scripts/e2e-fast-resubmit.sh tal-window
104+
./testing/wiremock/scripts/e2e-fast-resubmit.sh tal-window-closes
105+
```
106+
107+
The harness verifies the durable regression cases for Stellar fast resubmit on testnet:
108+
109+
| Scenario | Injected responses | Expected behavior |
110+
|----------|--------------------|-------------------|
111+
| `if-once` | 1 insufficient-fee response | One fast retry after about 5s, then submission proceeds |
112+
| `if-escalation` | 3 insufficient-fee responses | Fast retries after about 5s, 10s, and 15s |
113+
| `if-overcap` | 7 insufficient-fee responses | Six fast retries, then failure at the retry cap |
114+
| `tal-window` | 3 TRY_AGAIN_LATER responses | Fast retries after about 5s, 10s, and 15s |
115+
| `tal-window-closes` | 4 TRY_AGAIN_LATER responses | Three fast retries, then status-checker handoff |
116+
117+
The one-shot WireMock mappings remain static. Multi-response chains are registered dynamically by
118+
the runner at scenario arm time through `POST /__admin/mappings`, then deleted before the next
119+
scenario. This keeps the fixture set small while preserving the same Stellar RPC response bodies.
120+
121+
Prerequisites:
122+
123+
- Docker with Compose support.
124+
- Rust/Cargo for `cargo build --bin openzeppelin-relayer`.
125+
- `curl` and `jq`.
126+
- Network access to Stellar testnet, Horizon testnet, and friendbot.
127+
- The local signer keystore is generated automatically at
128+
`testing/wiremock/e2e/local-signer.keystore` if missing, using
129+
`KEYSTORE_PASSPHRASE`, and the generated testnet account is funded through friendbot before
130+
the relayer starts.
131+
132+
The runner starts WireMock, starts a disposable Redis container for the Redis queue backend, builds
133+
and starts `target/debug/openzeppelin-relayer`, funds the configured Stellar testnet account if
134+
needed, runs the selected scenario set, writes artifacts, and tears the stack down.
135+
136+
`if-overcap` runs last in the full matrix because it intentionally never lands on-chain and can
137+
leave the relayer-side local sequence cache ahead of chain state. `tal-window-closes` sends an
138+
explicit 10-minute `valid_until` because it waits for status-checker handoff after the fast retry
139+
window closes.
140+
141+
Artifacts default to `/tmp/oz-relayer-fast-resubmit` and can be overridden with `-o` or
142+
`ARTIFACT_DIR`. The runner writes:
143+
144+
- `verdicts.json`: JSON Lines, one verdict per scenario.
145+
- `relayer.log`: relayer stdout/stderr.
146+
- `harness.log`: stack setup and harness diagnostics.
147+
- `report.md`: line-count summary, deleted/kept scope, and matrix verdicts.
148+
149+
Each verdict line has this schema:
150+
151+
```json
152+
{
153+
"scenario": "if-once",
154+
"send_calls": 2,
155+
"gaps_s": [5],
156+
"final_status": "confirmed",
157+
"fast_log_seen": true,
158+
"checker_log_seen": false,
159+
"pass": true,
160+
"notes": ""
161+
}
162+
```
163+
164+
Gap checks use a tolerance of expected minus 1 second through expected plus 8 seconds.
165+
`tal-window-closes` treats its final checker-rescue gap as `>= 10s`.
166+
87167
## How It Works
88168

89169
WireMock runs as a transparent proxy (`--proxy-all`) to the target RPC endpoint (default: `https://soroban-testnet.stellar.org`). Pre-loaded stub mappings use **scenarios** (state machines) to intercept specific requests:

‎testing/wiremock/e2e/fast-resubmit-config.json‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
{
2+
"relayers": [
3+
{
4+
"id": "stellar-fast-resubmit",
5+
"name": "Stellar Fast Resubmit E2E",
6+
"network": "testnet",
7+
"paused": false,
8+
"network_type": "stellar",
9+
"signer_id": "local-signer",
10+
"policies": {
11+
"fee_payment_strategy": "relayer",
12+
"min_balance": 0
13+
}
14+
}
15+
],
16+
"notifications": [],
17+
"signers": [
18+
{
19+
"id": "local-signer",
20+
"type": "local",
21+
"config": {
22+
"path": "testing/wiremock/e2e/local-signer.keystore",
23+
"passphrase": {
24+
"type": "env",
25+
"value": "KEYSTORE_PASSPHRASE"
26+
}
27+
}
28+
}
29+
],
30+
"networks": [
31+
{
32+
"type": "stellar",
33+
"network": "testnet",
34+
"rpc_urls": [
35+
"http://localhost:9090"
36+
],
37+
"explorer_urls": [
38+
"https://stellar.expert/explorer/testnet"
39+
],
40+
"average_blocktime_ms": 5000,
41+
"is_testnet": true,
42+
"passphrase": "Test SDF Network ; September 2015",
43+
"horizon_url": "https://horizon-testnet.stellar.org"
44+
}
45+
],
46+
"plugins": []
47+
}

0 commit comments

Comments
 (0)