Skip to content

Log NYM mixFetch request lifecycle#734

Open
j0ntz wants to merge 2 commits into
masterfrom
jon/nym-android-fee-hang
Open

Log NYM mixFetch request lifecycle#734
j0ntz wants to merge 2 commits into
masterfrom
jon/nym-android-fee-hang

Conversation

@j0ntz

@j0ntz j0ntz commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

CHANGELOG

Does this branch warrant an entry to the CHANGELOG?

  • Yes
  • No

Dependencies

none

Description

Asana: https://app.asana.com/1/9976422036640/project/1215088146871429/task/1216729024836433

QA reported on 2026-07-20 (staging 26071707, Android) that with NYM privacy on,
Coreum and Avalanche sends sit on "Calculating Fee" indefinitely. That report is
after the v1 revert landed, so the revert did not resolve it.

The exported logs attached to the QA task show the failure but not its cause:

avalanche-me: syncNetwork failed with unknown error: Error panic:todo: extract error message
coreum-U/:    queryTransactions error: Error: panic:todo: extract error message
<every EVM chain>: Error fetching fees from Info Server

panic:todo: extract error message is a placeholder emitted by mix-fetch v1's Go
layer, which discards the real reason. Our own logging covered only mixFetch
setup, so there was no way to tie a wedged request to a host or endpoint.

This adds per-request instrumentation. Each mixnet request writes one line on the
way out and exactly one when it settles:

mixFetch #1411 POST api.avax.network/ext/bc/C/rpc start (20 in flight)
mixFetch #1411 POST api.avax.network/ext/bc/C/rpc -> 200 in 2442ms (15 in flight)
mixFetch #1407 POST lb.drpc.org/ogrpc -> 403 in 3829ms (14 in flight)

A start line with no matching terminal line is now the signature of a request
that never returned, and it names the host.

These lines end up in user-uploaded support logs, so the URI is reduced first:
the query string is dropped entirely (RPC endpoints carry API keys there), and
any path segment that looks like an identifier is masked. Route paths survive
(api.avax.network/ext/bc/C/rpc, monerolws1.edge.app/get_address_info) while
addresses and txids do not (blockbook.example.com/api/v2/address/<redacted>).

Two details worth calling out:

  • These log at warn, not info. The app configures the core with
    defaultLogLevel: 'warn', and filterLogs drops info at that level, so an
    info line would be missing from a QA log export unless the tester had first
    enabled Verbose Logging. warn is what makes this reach the export by default.
    NYM is opt-in and low volume, so the extra lines only appear for the users whose
    reports we are diagnosing.
  • Both io backends now share one wrapper, so the browser A/B harness and the app
    produce the same log shape.

No mixnet behavior changes: no timeout, pinning, or transport changes. This is
instrumentation for the next QA retest.

Verified

Driven on the iOS sim with NYM enabled for Avalanche and Base, against the live
mixnet, with this core updot-linked into the app. Captured from the running
app's console pipeline:

mixFetch #1397 POST lb.drpc.org/ogrpc -> 403 in 2977ms (18 in flight)
mixFetch #1410 POST base-rpc.publicnode.com start (19 in flight)
mixFetch #1387 POST base.api.pocket.network -> 200 in 4510ms (19 in flight)
mixFetch #1393 POST api.avax.network/ext/bc/C/rpc -> 200 in 4130ms (17 in flight)
mixFetch #1403 GET  api.etherscan.io/v2/api -> 200 in 3174ms (14 in flight)

verify-repo.sh passed (prepare, eslint, jest).

Already surfacing something

The in-flight counter reached 20 concurrent mixnet requests, including four
simultaneous requests to the same host (base-rpc.publicnode.com). mix-fetch v1
carried a one-request-per-host limitation that this repo used to work around with
a per-host queue, removed in 5916d9d on the grounds that 1.4.2 fixed it. That
saturation is worth a look as a candidate mechanism for the Android wedge, on a
slower device with more wallets enabled. Not addressed here.

Avalanche completed normally over NYM on iOS throughout (2-4s per request), which
matches QA's own iOS result and is why the Android-only report needs logs from an
Android build to go further.


Update: Android emulator findings, and a fix

Ran this on a deliberately throttled Android emulator (2 cores, 2 GB, -netdelay umts,
API 36) with a fresh account and one Avalanche wallet, against the live mixnet. The logging
above localized the failure on its first run.

The panic is host-specific, not random

3 of 3 requests to api.avascan.info panicked in ~1s. Every other host settled normally:

mixFetch #10 GET api.avascan.info/v2/network/mainnet/evm/43114/etherscan/api failed after 938ms: panic:todo: extract error message
mixFetch #12 GET api.avascan.info/... failed after 1102ms: panic:todo: extract error message
mixFetch #18 GET api.avascan.info/... failed after 1011ms: panic:todo: extract error message

Across 32 requests: 29 settled OK (p50 2666ms), 3 failed, 0 never settled. api.avascan.info
is the etherscan-compatible transaction-history endpoint for Avalanche, which matches QA's
avalanche-me: syncNetwork failed ... panic:todo exactly. The panic originates inside NYM's Go
layer and is not something this wrapper can fix; it is now at least attributable to a host.

"Error fetching fees from Info Server" is a red herring

mixFetch #4 GET info1.edge.app/v1/networkFees/avalanche -> 404 in 2868ms
mixFetch #1 GET info2.edge.app/v1/networkFees/avalanche -> 404 in 3120ms

Those are plain 404s from the info server, not a NYM failure. That log line appears on every EVM
chain in QA's report and is unrelated to the mixnet.

What this commit changes

A single Avalanche wallet peaked at 12 concurrent mixnet requests. A real user's wallet list
multiplies that, and the app already warns that enabling NYM on multiple assets slows sends. So:

  • cap the mixnet at 6 concurrent overall, 2 per host, queueing the rest
  • drop the per-request timeout from 300s to 60s, since a 5-minute ceiling is what turns an
    unanswered request into an apparently permanent "Calculating Fee" spinner

Measured before/after on the same emulator, live mixnet:

before after
peak in-flight 12 6
p50 latency 2666ms 2669ms
never-settled 0 0

Queue waits up to 2.1s are now logged (queued 2106ms) so throttling is distinguishable from
network time. verify-repo.sh passes (prepare, eslint, jest 162 passing).

Honest scope

The QA wedge itself did not reproduce here: with one wallet, every request settled and
in-flight always drained to zero. These two changes are targeted mitigation for the mechanisms
that can produce the reported symptom, not a confirmed fix for it. The logging is retained
precisely so the next QA build says which host wedges if it happens again.

Review hardening

Four rounds of reviewer-bot findings were taken (one rejected, see below). Beyond the two
changes above, this PR now also:

  • bounds the queue itself at 10s (MAX_QUEUE_WAIT_MS), past which a request proceeds anyway.
    Without it, a starved queue would have added its own multi-minute delay on top of the request
    timeout and recreated the very hang this targets
  • removes each waiter's resolver after its race settles, so a stretch where nothing settles
    cannot grow the wait list without bound
  • routes error text through the same URI reducer as the request label, so a URL quoted inside an
    error (Post "https://host/tx/<txid>?apikey=...") cannot reintroduce an identifier or key that
    was masked a few characters earlier on the same line
  • broadens identifier masking to cover separator-bearing forms (UUIDs, cashaddr)

Final state on device, same emulator, running exactly this commit:

mixFetch #10 GET api.avascan.info/v2/network/mainnet/evm/43114/etherscan/api failed after 905ms (5 in flight): panic:todo: extract error message

peak in-flight 6, p50 2718ms, 0 never-settled, 162 unit tests passing.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@j0ntz

j0ntz commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

📸 Test evidence

agent proof 1216729024836433 01 nym base wallet

agent proof 1216729024836433 01 nym base wallet

Captured by the agent's in-app test run (build-and-test).

Comment thread src/util/nym.ts Outdated
The 2026-07-20 QA report on staging 26071707 shows NYM requests failing with
'panic:todo: extract error message', a placeholder the mix-fetch v1 Go layer
emits in place of the real reason. Our own logging recorded only mixFetch
setup, so a wedged request could not be attributed to a host or endpoint.

Log each mixnet request on the way out and once when it settles, with the
method, host, path, elapsed time, and either the HTTP status or the raw error
text. A start line with no matching terminal line is now the signature of a
request that never returned. Query strings are omitted because RPC endpoints
carry API keys.

Both io backends share one wrapper so the browser A/B harness and the app
produce the same log shape.
@j0ntz
j0ntz force-pushed the jon/nym-android-fee-hang branch from 3b7063a to 4a12e40 Compare July 20, 2026 18:11
@j0ntz

j0ntz commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

📸 Test evidence

agent proof 1216729024836433 02 android nym avax

agent proof 1216729024836433 02 android nym avax

Captured by the agent's in-app test run (build-and-test).

Comment thread src/util/nym.ts
Comment thread src/util/nym.ts
@j0ntz
j0ntz force-pushed the jon/nym-android-fee-hang branch from e5de482 to 2dd3d62 Compare July 20, 2026 23:33
Comment thread src/util/nym.ts
Comment thread CHANGELOG.md Outdated
@j0ntz
j0ntz force-pushed the jon/nym-android-fee-hang branch from 2dd3d62 to 0f654ce Compare July 20, 2026 23:45
Comment thread src/util/nym.ts
@j0ntz

j0ntz commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

📸 Test evidence

agent proof 1216729024836433 03 android fixed build

agent proof 1216729024836433 03 android fixed build

Captured by the agent's in-app test run (build-and-test).

Comment thread CHANGELOG.md Outdated
@j0ntz
j0ntz force-pushed the jon/nym-android-fee-hang branch from 0f654ce to e70ca55 Compare July 20, 2026 23:51

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e70ca55. Configure here.

Comment thread src/util/nym.ts
Comment thread CHANGELOG.md
Measured on a throttled Android emulator with the request logging from the
previous commit: a single Avalanche wallet opens 12 concurrent mixnet requests
during one sync. mix-fetch v1 has a history of serving one request per host at
a time, which this module used to work around with a per-host queue removed in
5916d9d on the understanding that 1.4.2 had fixed it.

Cap the mixnet at 6 concurrent requests overall and 2 per host, queueing the
rest. Measured before and after on the same emulator against the live mixnet:
peak in-flight drops from 12 to 6, per-request latency is unchanged (p50 2666ms
vs 2669ms), and every request still settles.

Also drop the per-request timeout from 300s to 60s. Healthy requests measure
2-4 seconds, so five minutes only ever meant that a request the mixnet never
answered pinned its caller for five minutes. On the send screen that is the
'Calculating Fee' spinner that QA reports as an infinite hang.

Neither change touches the request logging, which is still needed to attribute
the remaining NYM-internal failure.
@j0ntz
j0ntz force-pushed the jon/nym-android-fee-hang branch from e70ca55 to c023d0a Compare July 20, 2026 23:56
@j0ntz

j0ntz commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

📸 Test evidence

agent proof 1216729024836433 04 android final verified

agent proof 1216729024836433 04 android final verified

Captured by the agent's in-app test run (build-and-test).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant