feat: Add insufficient fee and TRY_AGAIN_LATER outcome metrics for Stellar transactions (#702)

* feat: Add additional stellar metrics

* chore: Test

* feat: Add wiremock setup

* chore: PR feedback

Author: zeljkoX

.claude/skills/wiremock-test/SKILL.md

@@ -0,0 +1,122 @@
+---
+name: wiremock-test
+description: Manage WireMock proxy for RPC testing. Starts the proxy, arms scenarios to simulate transient errors (timeouts, 500s, connection resets, and network-specific errors like Stellar TRY_AGAIN_LATER), and checks metrics. Use to verify relayer behavior under failure conditions.
+allowed-tools: Bash, Read, Grep, Glob, Agent
+argument-hint: "<scenario> | setup <network> [rpc-url] | teardown | status | check-metrics"
+---
+
+# WireMock RPC Test Proxy
+
+You manage a WireMock proxy that sits between the relayer and an RPC endpoint. It forwards traffic normally, but can inject mock error responses on demand.
+
+## Project paths
+
+- Docker compose: `testing/wiremock/docker-compose.yaml`
+- Generic mappings: `testing/wiremock/mappings/generic/` (work with any JSON-RPC network)
+- Stellar mappings: `testing/wiremock/mappings/stellar/`
+- Future EVM mappings: `testing/wiremock/mappings/evm/` (not yet created)
+- Helper scripts: `testing/wiremock/scripts/`
+
+## Available scenarios
+
+### Generic (any network)
+
+| Scenario name | Shorthand | What it does |
+|---|---|---|
+| `rpc-timeout` | `timeout` | 30s delay on next RPC call (any method) |
+| `rpc-500` | `500` | HTTP 500 on next RPC call |
+| `rpc-connection-reset` | `reset` | TCP connection reset on next RPC call |
+
+### Stellar-specific
+
+| Scenario name | Shorthand | What it does |
+|---|---|---|
+| `stellar-try-again-later` | `tal`, `try-again-later` | Returns `TRY_AGAIN_LATER` on next `sendTransaction` |
+| `stellar-insufficient-fee` | `fee`, `insufficient-fee` | Returns `ERROR` with `TxInsufficientFee` on next `sendTransaction` |
+| `stellar-get-transaction-not-found` | `not-found`, `get-tx-not-found` | Returns `NOT_FOUND` for next `getTransaction` |
+
+All scenarios fire **exactly once** then revert to proxying real traffic.
+
+## How to handle user arguments ($ARGUMENTS)
+
+Based on the argument, execute the matching flow:
+
+### `setup` — requires a network argument
+**Syntax:** `setup <network> [rpc-url]`
+
+A bare `setup` with no network is **not allowed** — ask the user which network they want (stellar or evm).
+
+1. Determine target RPC:
+   - `setup stellar` → default: `https://soroban-testnet.stellar.org`
+   - `setup evm` or `setup anvil` → default: `http://host.docker.internal:8545`
+   - `setup stellar <url>` or `setup evm <url>` → use the provided URL as `WIREMOCK_PROXY_TARGET`
+2. Start WireMock:
+   ```bash
+   WIREMOCK_PROXY_TARGET=<target> docker compose -f testing/wiremock/docker-compose.yaml up -d
+   ```
+   For stellar with default target, just: `docker compose -f testing/wiremock/docker-compose.yaml up -d`
+3. Wait for health: poll `curl -sf http://localhost:9090/__admin/health` (retry up to 10 times with 2s sleep)
+4. Verify scenarios loaded: `curl -s http://localhost:9090/__admin/scenarios`
+5. Remind the user:
+   - WireMock proxy is running on `http://localhost:9090` proxying to `<target>`
+   - To use it, update the relevant network config `rpc_urls` to `["http://localhost:9090"]`
+   - If the relayer runs in Docker, use `http://host.docker.internal:9090` instead
+   - Show only the **relevant** scenarios: generic + the chosen network's specific scenarios
+   - For Stellar: mention updating `config/networks/stellar.json` testnet entry
+   - For EVM: mention updating the relevant EVM network config
+
+### `teardown` or `stop`
+1. Run: `docker compose -f testing/wiremock/docker-compose.yaml down`
+2. Confirm stopped
+
+### `status`
+1. Check if WireMock is running: `docker compose -f testing/wiremock/docker-compose.yaml ps`
+2. If running, show scenario states: `curl -s http://localhost:9090/__admin/scenarios`
+3. Show recent requests: `curl -s 'http://localhost:9090/__admin/requests?limit=10'`
+4. Parse and display in a readable format
+
+### `check-metrics` or `metrics`
+1. Fetch relayer metrics: `curl -s http://localhost:8081/debug/metrics/scrape`
+2. Filter and display lines matching: `insufficient_fee|try_again_later|transactions_success|transactions_failed|stellar_submission_failures|stellar_try_again_later|rpc_call_latency|api_rpc_failures`
+3. Summarize what the metrics show
+
+### A scenario name or shorthand
+1. First ensure WireMock is running (check health endpoint, start if needed)
+2. Resolve the scenario name — the user may use shorthand:
+   - Generic: `timeout` → `rpc-timeout`, `500` → `rpc-500`, `reset` → `rpc-connection-reset`
+   - Stellar: `try-again-later` or `tal` → `stellar-try-again-later`, `insufficient-fee` or `fee` → `stellar-insufficient-fee`, `not-found` or `get-tx-not-found` → `stellar-get-transaction-not-found`
+3. Arm the scenario:
+   ```bash
+   curl -s -X PUT "http://localhost:9090/__admin/scenarios/{scenario-name}/state" \
+     -H 'Content-Type: application/json' -d '{"state": "armed"}'
+   ```
+4. Verify it's armed by checking the scenario state
+5. Tell the user what will happen on the next matching RPC call
+
+### `chaos` or `all-chaos`
+1. Ensure WireMock is running
+2. Arm ALL scenarios simultaneously
+3. Explain what will happen: scenarios are priority-based, so the most specific one fires first
+
+### `reset`
+1. Reset all scenarios: `curl -s -X POST http://localhost:9090/__admin/scenarios/reset`
+2. Clear request log: `curl -s -X DELETE http://localhost:9090/__admin/requests`
+3. Confirm all scenarios are back to initial (inactive) state
+
+## Important reminders
+
+Always remind the user about these when relevant:
+- The relayer's network `rpc_urls` must point to `http://localhost:9090` (or `http://host.docker.internal:9090` from Docker) for WireMock to intercept traffic
+- After testing, they should revert `rpc_urls` back to the real RPC endpoint
+- Scenarios are one-shot: they fire once and revert. Re-arm to test again.
+- The relayer's metrics endpoint is at `http://localhost:8081/debug/metrics/scrape` (configurable via METRICS_PORT)
+- Generic scenarios (timeout, 500, connection-reset) work with any network — no need to change them when switching between Stellar and EVM testing
+
+## Adding new network support
+
+When adding EVM or other network-specific scenarios:
+1. Create `testing/wiremock/mappings/<network>/` directory
+2. Add scenario JSON files following the existing pattern (use `"requiredScenarioState": "armed"`)
+3. Mount the directory in `docker-compose.yaml` (uncomment or add the volume line)
+4. Update this skill's scenario tables above
+5. Restart WireMock to load new mappings

src/domain/transaction/stellar/submit.rs

@@ -13,10 +13,10 @@ use super::{
 use crate::{
     constants::STELLAR_INSUFFICIENT_FEE_MAX_RETRIES,
     jobs::JobProducerTrait,
-    metrics::STELLAR_SUBMISSION_FAILURES,
+    metrics::{STELLAR_SUBMISSION_FAILURES, TRANSACTIONS_INSUFFICIENT_FEE},
     models::{
-        NetworkTransactionData, RelayerRepoModel, TransactionError, TransactionMetadata,
-        TransactionRepoModel, TransactionStatus, TransactionUpdateRequest,
+        NetworkTransactionData, RelayerRepoModel, TransactionError, TransactionRepoModel,
+        TransactionStatus, TransactionUpdateRequest,
     },
     repositories::{Repository, TransactionCounterTrait, TransactionRepository},
     services::{
@@ -166,17 +166,26 @@ where
                 // transaction alive. The status checker will handle retries:
                 // - Submitted txs: resubmitted with exponential backoff
                 // - Sent txs: re-enqueued via handle_sent_state
-                crate::metrics::STELLAR_TRY_AGAIN_LATER
-                    .with_label_values(&[&tx.relayer_id, &tx.status.to_string()])
-                    .inc();
+                let mut meta = tx.metadata.clone().unwrap_or_default();
+                meta.try_again_later_retries = meta.try_again_later_retries.saturating_add(1);
+
+                // Only push on first encounter (dedup: won't fire on retry 2, 3, etc.)
+                if meta.try_again_later_retries == 1 {
+                    crate::metrics::STELLAR_TRY_AGAIN_LATER
+                        .with_label_values(&[&tx.relayer_id, &tx.status.to_string()])
+                        .inc();
+                }
+
                 debug!(
                     tx_id = %tx.id,
                     relayer_id = %tx.relayer_id,
                     status = ?tx.status,
+                    try_again_later_retries = meta.try_again_later_retries,
                     "TRY_AGAIN_LATER — status checker will retry"
                 );
                 let update_req = TransactionUpdateRequest {
                     sent_at: Some(Utc::now().to_rfc3339()),
+                    metadata: Some(meta),
                     ..Default::default()
                 };
                 let updated_tx = self
@@ -199,13 +208,17 @@ where
                     .as_deref()
                     .is_some_and(is_insufficient_fee_error)
                 {
-                    let insufficient_fee_retries = tx
-                        .metadata
-                        .as_ref()
-                        .map_or(0, |metadata| metadata.insufficient_fee_retries)
-                        .saturating_add(1);
+                    let mut meta = tx.metadata.clone().unwrap_or_default();
+                    meta.insufficient_fee_retries = meta.insufficient_fee_retries.saturating_add(1);
+
+                    // Only push on first encounter (dedup: won't fire on retry 2, 3, etc.)
+                    if meta.insufficient_fee_retries == 1 {
+                        TRANSACTIONS_INSUFFICIENT_FEE
+                            .with_label_values(&[tx.relayer_id.as_str(), "stellar"])
+                            .inc();
+                    }
 
-                    if insufficient_fee_retries > STELLAR_INSUFFICIENT_FEE_MAX_RETRIES {
+                    if meta.insufficient_fee_retries > STELLAR_INSUFFICIENT_FEE_MAX_RETRIES {
                         STELLAR_SUBMISSION_FAILURES
                             .with_label_values(&["error", "tx_insufficient_fee"])
                             .inc();
@@ -218,23 +231,13 @@ where
                         tx_id = %tx.id,
                         relayer_id = %tx.relayer_id,
                         status = ?tx.status,
-                        insufficient_fee_retries,
+                        insufficient_fee_retries = meta.insufficient_fee_retries,
                         result_code = decoded_result_code.as_deref().unwrap_or("Unknown"),
                         "ERROR with insufficient fee — status checker will retry"
                     );
                     let update_req = TransactionUpdateRequest {
                         sent_at: Some(Utc::now().to_rfc3339()),
-                        metadata: Some(TransactionMetadata {
-                            consecutive_failures: tx
-                                .metadata
-                                .as_ref()
-                                .map_or(0, |metadata| metadata.consecutive_failures),
-                            total_failures: tx
-                                .metadata
-                                .as_ref()
-                                .map_or(0, |metadata| metadata.total_failures),
-                            insufficient_fee_retries,
-                        }),
+                        metadata: Some(meta),
                         ..Default::default()
                     };
                     let updated_tx = self
@@ -414,6 +417,7 @@ mod tests {
     use soroban_rs::xdr::WriteXdr;
 
     use crate::domain::transaction::stellar::test_helpers::*;
+    use crate::models::TransactionMetadata;
 
     /// Helper to create a SendTransactionResponse with given status
     fn create_send_tx_response(status: &str, hash: &str) -> SendTransactionResponse {
@@ -970,11 +974,18 @@ mod tests {
                     Box::pin(async move { Ok(r) })
                 });
 
-            // partial_update is called to refresh sent_at — status should NOT change
+            // partial_update is called to refresh sent_at and persist metadata — status should NOT change
             mocks
                 .tx_repo
                 .expect_partial_update()
-                .withf(|_, upd| upd.sent_at.is_some() && upd.status.is_none())
+                .withf(|_, upd| {
+                    upd.sent_at.is_some()
+                        && upd.status.is_none()
+                        && upd
+                            .metadata
+                            .as_ref()
+                            .is_some_and(|m| m.try_again_later_retries == 1)
+                })
                 .returning(|id, _upd| {
                     let mut tx = create_test_transaction("relayer-1");
                     tx.id = id;