Implement E2E tests for registrar pallet operations (#572)

* setup paras registrar test

* complete reserve extrinsic test

* add snapshot files

* rename

* regenerate snapshot files

* add ParaInfo assertion

* add register test

* generate snap file for kusama

* update registrar test

* add known good block numbers

* add function comments

* add deregister logic

* add force_register test

* add root deregister test

* add test comments for paras Root register

* add logs

* some progress

* whew, finally

* add polkadot registrar snapshots

* add deregister test

* reorder deregister logic

* complete root register deregister flow

* add lock unlock logic

* complete swap tests

* remove console logs and commenting out

* update known good

* update kusama snap file

* update kusama snap file

* fix lint

* add tests for code upgrade and current head set

* complete registrar test

* update known good

* upda polkadot registrar snap

* refactor

* remove duplicate comment

* update known good

* update test to assert AlreadyRegistered error

* Enable `Asset Hub authorizes Polkadot upgrade via Collectives` scenario (#577)

* Bring back assetHubPolkadot.polkadot.collectivesPolkadot scenario

* Update chopsticks to 1.3.0

* Negative/failure governance e2e tests (#306)

* Begin refactor of governance e2e tests

* Test low approval/support ref failures

* Update PAH/KAH governance snaps

* Create track capacity overflow test

Also, refactor tests to referenda negative execution into own tree.

* Reduce number of tracks being tested, upd. snaps

* Test track overflow mechanism

* Update AH governance snapshots

* Test track overflow handling in referenda submission

* Update governance snapshots

* Bump biome

* Fix lint in governance E2E tests

* Bump block numbers

* Remove unstable voting snapshots

* completely cover reserve function test

* add extra assertions for register

* add deregister error assertion

* cover CannotDeregister error case for deregister

* assert NotReserved error

* remove comment out

* update dot snap file

* update kusama snap file

* update known good

* update polkadot snap file

* update dev accounts

* update account address in snap file

* update polkadto snap file

* add error state check for swap operation

* add parachain parathread test

* add paras origin call

* update known good

* add CodeUpgrade assertion

* fix root schedule code upgrade

* add code hash and upgrade restriction update

* add event assertion

* add extra snapshots

* add extra snapshots

* update check events to use a consistent pattern

* update polkadot snap file

* update known good

* split register function into three distinct functions

* start fixing register logic

* update snap file

* fix deregister test - add missing register call

* remove single split test

* add comments for constants

* update

* replace extrinsic failed helper

* add helper function comments

* update

* update known good

* remove submitAndAdvanceBlock helper

* resolve conflicts

* resolve conflicts

* Bump @swc/core from 1.15.30 to 1.15.32 (#603)

Bumps [@swc/core](https://github.com/swc-project/swc/tree/HEAD/packages/core) from 1.15.30 to 1.15.32.
- [Release notes](https://github.com/swc-project/swc/releases)
- [Changelog](https://github.com/swc-project/swc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/swc-project/swc/commits/v1.15.32/packages/core)

---
updated-dependencies:
- dependency-name: "@swc/core"
  dependency-version: 1.15.32
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump uuid in the npm_and_yarn group across 1 directory (#604)

Bumps the npm_and_yarn group with 1 update in the / directory: [uuid](https://github.com/uuidjs/uuid).


Updates `uuid` from 11.1.0 to 11.1.1
- [Release notes](https://github.com/uuidjs/uuid/releases)
- [Changelog](https://github.com/uuidjs/uuid/blob/v11.1.1/CHANGELOG.md)
- [Commits](uuidjs/uuid@v11.1.0...v11.1.1)

---
updated-dependencies:
- dependency-name: uuid
  dependency-version: 11.1.1
  dependency-type: indirect
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Skip `coretimeKusama -> assetHubKusama` XCM test (#605)

* Skip coretimeKusama -> assetHubKusama XCM test

Fails due to missing ConcatenatedOpaqueVersionedXcm variant in
@polkadot/types XcmpMessageFormat enum. Fix pending upstream in
polkadot-js/api.

* Skip coretimePolkadot -> assetHubPolkadot XCM test

Same root cause as coretimeKusama -> assetHubKusama: missing
ConcatenatedOpaqueVersionedXcm in @polkadot/types. polkadot-js/api#6255.

* Update block numbers

* Apply linter

* resolve conflicts

* resolve conflicts

* Bump ip-address in the npm_and_yarn group across 1 directory (#608)

Bumps the npm_and_yarn group with 1 update in the / directory: [ip-address](https://github.com/beaugunderson/ip-address).


Updates `ip-address` from 10.1.0 to 10.2.0
- [Commits](https://github.com/beaugunderson/ip-address/commits)

---
updated-dependencies:
- dependency-name: ip-address
  dependency-version: 10.2.0
  dependency-type: indirect
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Improve performance by sharing client object within test suites (#609)

* Add beforeEach to DescribeNode

* Snapshot restore truncates storage layers

* Fix network teardown

* Share network across accounts suite

* Share network across scheduler suite

* Share network across bounties suite

* Share network across childBounties suite

* Share network across vesting suite

* Share network across governance suite

* Share network across multisig suite

* Share network across multisig.proxy suite

* Share network across nomination-pools suite

* Share network across staking suite

* Share network across proxy suite

* Share network across remoteProxy suite

* Share network across bounties suite

* Share network across childBounties suite

* Share network across governance suite

* Share network across multisig suite

* Share network across multisig.proxy suite

* Share network across preimage suite

* Share network across proxy suite

* Share network across remoteProxy suite

* Share network across staking suite

* Patch chopsticks-core to expose resetStorageLayers on Block

* Update yarn.lock for chopsticks-core patch

* Switch CI to forks pool with 5 workers

The shared-client refactor is incompatible with vitest's threads pool:
multiple workers contend for the same chopsticks WS server, causing
RPC timeouts and retry loops that see leaked state from earlier
attempts. Forks pool gives each worker its own process and avoids
the contention entirely.

* Fix accounts suite teardown and await dev.setHead

Two issues flagged by automated review on PR #609:

1. The accounts suite's beforeEach was missing an await on
   baseClient.dev.setHead(blockNumber). The setHead RPC is async
   and the unawaited promise leaves a pending request that can
   reject after the test completes.

2. The relay-fallback paths in the force_transfer/force_unreserve/
   force_set_balance/force_adjust_total_issuance tests called
   setupNetworks from inside the test body. setupNetworks registers
   beforeEach and afterAll hooks at the describe level, conflicting
   with the suite-level lifecycle and producing resource leaks. The
   chains that take this path (e.g. bridgeHubPolkadot, which lacks
   the scheduler pallet) hit assertion failures and timeouts.

   Replaced with createNetworks plus a per-function teardownExtras
   closure that disconnects and tears down both the relay and
   re-created base clients at the end of the test.

* Await setupBalances calls in preimage tests

Nine call sites kicked off the storage seed without awaiting it. The
underlying setStorage RPC can reject asynchronously on chains with
strict transaction-pool validation (basilisk, karura), and the
unawaited rejection escaped as an unhandled rejection after the
test had moved on.

* Bump block numbers

* Fix lint errors after shared-client refactor

- postAhmFiltering: PostAhmTest.testFn was typed (chain: Chain<>) but
  the leaf functions it points to all take Client<>; updated the type.
- preimage, proxy, scheduler: drop unused setupNetworks imports left
  over from the refactor.
- governance: drop the dead 'chain' parameter from injectDecisionPeriodEnd
  (unused since the function was refactored to take a Client).

* Share network across people suite

* Share networks across treasury suite

* Share networks across collectives suite

* Share network across configuration suite

* Use createNetworks with explicit teardown in upgrade suite

The conditional 1- to 3-chain network creation in upgrade tests cannot
be lifted to a suite-level beforeAll because the same-vs-distinct chain
decision is per-test. Replaced setupNetworks (which registers
describe-level beforeEach/afterAll hooks at runtime, leaking state
across tests) with createNetworks plus a try/finally that disconnects
every distinct client at the end of each test.

* Share networks across system suite

* Share network across recovery suite

* Patch chopsticks-utils.sendTransaction to unsubscribe and swallow late rejections

After a tx hits isInBlock/isFinalized, sendTransaction's deferred is
resolved but the polkadot.js subscription stays active and keeps
firing. A subsequent isError callback (e.g. a tx-pool rejection that
arrives after the block was already produced) calls deferred.reject
on the already-settled promise — a no-op for that promise, but the
underlying signed.send subscription's own error path can still produce
unhandled rejections.

Two changes: explicitly unsubscribe after either terminal status, and
attach a no-op catch to the deferred promise so any late settle is
silenced rather than escaping as an unhandled rejection.

* Update Bifrost Kusama <-> KAH XCM snap

* Skip failing CoretimeK <-> KAH XCM test

* Bump block numbers

* Skip failing KAH <-> PeopleKusama XCM tests

* Regenerate chopsticks patches from pet-perf-stack branch tip

Replaces the hand-crafted patches with full rebuilds from AcalaNetwork/chopsticks#pet-perf-stack (commit e267237). Adds chopsticks-db patch covering the new PagedKeys and RpcCall entities (§1 of #606). Updates chopsticks-core and chopsticks-utils patches to include all remaining commits: getKeysPaged guard refinement, getKeysPaged disk cache, cache-hit extension fix, in-flight dedup, setHead subscriber fix, and dryRunExtrinsicsAmortized. Drop when chopsticks#1028 is published.

* Use dryRunExtrinsicsAmortized for proxy call-filtering loop

Replaces the sendTransaction + newBlock per-proxy-action loop with a single dryRunExtrinsicsAmortized batch. Pre-signs every action with the proxy account's current nonce (storage layer pops between extrinsics revert nonce, so sequential nonces would be rejected). Adds eventsFromAmortizedDryRunResult helper that decodes the system events Vec from a per-extrinsic storageDiff and returns a { events: Promise<Codec[]> } shape compatible with checkEvents. Verified locally on Polkadot proxy filtering tests (16 of 16 pass).

* Hoist accounts relay client to suite scope

Each XCM-based test in the accounts suite was spinning up its own ephemeral relay+parachain pair via createNetworks inside the test body, then tearing them down with a per-test teardownExtras closure. Move that to the suite-level beforeAll: accountsE2ETests now takes an optional relayChain parameter, creates the connected pair once in beforeAll, restores both via captureSnapshot in beforeEach, and tears both down in afterAll. The 13 XCM test functions now take an already-connected relayClient instead of a relayChain plus the boilerplate to bootstrap one. Drops the relayChain field from AccountsTestConfig (it is no longer test-local data) and the TInitStoragesRelay generic from the test fns. Verified locally: peoplePolkadot.accounts (55/55 pass, uses relayClient path) and assetHubPolkadot.accounts (46/46 pass, no-relay path).

* Hoist people relay client to suite scope

addRegistrarViaRelayAsRoot was creating its own ephemeral relay+people pair via createNetworks on every invocation, ignoring the peopleClient already provisioned by basePeopleChainE2ETests's beforeAll. Move relay creation to suite-level beforeAll, restore both via captureSnapshot(peopleClient, relayClient) in beforeEach, and tear both down in afterAll. The function now takes the suite-shared (relayClient, peopleClient) directly. Verified: peoplePolkadot.e2e (5/5 pass).

* Use more vitest forks in CI workflow

* Re-enable XCM tests broken by `XcmpMessageFormat::ConcatenatedOpaqueVersionedXcm` (#613)

* Update dependencies

* Patch `@polkadot/types` to add `XcmpMessageFormat::ConcatenatedOpaqueVersionedXcm`

* Re-enable `assetHubKusama` <-> `coretimeKusama` XCM tests

* Re-enable `assetHubPolkadot` <-> `coretimePolkadot` XCM tests

* Re-enable `assetHubPolkadot` <-> `bridgeHubPolkadot` XCM tests

* Re-enable `assetHubKusama` <-> `bridgeHubKusama` XCM tests

* Update block numbers

* Bump Chopsticks to 1.4.0 to downstream performance improvements (#614)

* Bump dependencies (Chopsticks -> 1.4.0)

This bump brings in the performance improvements from
chopsticks#1027/#1028, thus removing the need for the existing patch.

* Drop `@acala-network/chopsticks` patches now covered by 1.4.0

* Bump `@acala-network/chopsticks` to `^1.4.0`

* Regen lockfile to drop obsolote ref to tar@npm:^6.1.11

* Introduce Polkadot Bulletin chain definition and accounts E2E test (#615)

* Rank RPC endpoints by health and use the healthiest one (#616)

* Add endpoint-health probe script

scripts/probe-endpoints.mjs measures every RPC endpoint in
`packages/networks/src/pet-chain-endpoints.json` and emits a per-endpoint
health report sorted by score. Each endpoint is probed over a single
persistent WebSocket (matching production traffic shape used by Subway
and Chopsticks/polkadot.js) with SAMPLES pings paced across WINDOW_MS,
mixing `chain_getHeader`, `chain_getBlockHash`, and `state_getStorage`
at the pinned block from KNOWN_GOOD_BLOCK_NUMBERS_*.env. The score
combines failure rate, p95 ping latency, and an archive-capable flag.

Chains are probed serially by default to avoid host-IO contention
inflating latency numbers; CHAIN_CONCURRENCY=N can override for local
fast iterations. All other knobs (SAMPLES, WINDOW_MS, INFLIGHT,
CONNECT_TIMEOUT_MS, RPC_TIMEOUT_MS, CHAINS, KEEP_PROBE_LOG) have safe
defaults documented in the script header.

* Add script to apply probe ranking to `pet-chain-endpoints.json`

scripts/apply-probe-ranking.mjs reads `endpoint-probe.json` and rewrites
`packages/networks/src/pet-chain-endpoints.json` with each chain's endpoint
array sorted by ascending health score from the probe.

Policy is sort, never filter: a poorly-scoring endpoint is moved to the
end of the list rather than removed. This preserves the failover ladder
that both Subway (via `shuffle_endpoints: false` + round-robin rotation)
and Chopsticks (via polkadot.js `WsProvider`'s `#endpointIndex + 1` on
reconnect) rely on to escape from a runtime stall.

Bails with non-zero exit if the probe and the endpoints JSON disagree
on chain set (added/removed chains require a deliberate human decision,
not a silent overwrite). Idempotent: exit 0 with no rewrite if the file
already matches the probe order.

* Add daily endpoint-probe workflow

Runs scripts/probe-endpoints.mjs over every chain in
`pet-chain-endpoints.json`, then runs scripts/apply-probe-ranking.mjs
to rewrite the JSON sorted by score. If the rewrite changed the file,
opens a PR against master.

Cadence is daily (04:17 UTC); endpoint health drifts on the order of
weeks but daily caps the lag between an endpoint going bad and the
JSON reflecting it. `workflow_dispatch` accepts overrides for the
probe knobs (samples, window_ms, inflight, chains) so a human can
re-run with tighter or looser parameters as needed.

The PR is opened by the github-actions app (same token pattern as
`update-known-good.yml`) and gated by human review.

* Sort `pet-chain-endpoints.json` by initial probe ranking

Bootstrap result from running scripts/apply-probe-ranking.mjs against
the first probe of all 23 chain pools. 14 chains had a different order;
9 were already sorted by score. Notable demotions to the bottom of
their respective lists (non-archive or sustained loss under repeated
`state_getStorage` at the pinned block):
- polkadot: `polkadot.public.curie.radiumblock.co/ws`, `polkadot-rpc.publicnode.com`
- assetHubPolkadot: `statemint.public.curie.radiumblock.co/ws`
- assetHubKusama: `statemine.public.curie.radiumblock.co/ws`
- bridgeHubKusama: `bridgehub-kusama.public.curie.radiumblock.co/ws`

The daily probe-endpoints workflow keeps this ordering current.

* Disable Subway endpoint shuffling

`shuffle_endpoints: true` randomly permuted the endpoint list at Subway
startup, which made the ranking maintained in `pet-chain-endpoints.json`
meaningless: Subway would pick whatever endpoint landed at index 0 of
the shuffle, regardless of health.

With shuffling off, Subway uses array[0] as its initial endpoint and
treats array[1..N-1] as the round-robin failover sequence on stall.
This matches the behavior of Chopsticks/polkadot.js `WsProvider` for
chains that don't go through Subway, so both paths now respect the
health-sorted order set by the daily probe workflow.

* Add local-CI runner script

scripts/run-ci-locally.sh reproduces the CI Subway setup on the local
machine: it boots one Subway per chain in the chosen network using the
same chain/port mapping that `.github/workflows/ci.yml` uses, waits for
each `/health` endpoint, sources the matching KNOWN_GOOD_BLOCK_NUMBERS
env file, exports the per-chain `<CHAIN>_ENDPOINT=ws://localhost:<port>`
vars, and runs `yarn test:<network>` inside a backgrounded tmux session
with two panes (Subway logs above, test output below).

Cleanup on script exit (or Ctrl-C) kills the Subway PIDs but leaves
`/tmp/subway-*.log` in place for after-the-fact inspection. Useful for
sanity-checking endpoint-ranking changes or reproducing CI-only flakes
without waiting for a GitHub Actions run.

* Document scoring, pacing, and tmux/env idioms in endpoint scripts

The bare scripts compiled fine but the rationale behind the non-obvious
decisions wasn't preserved anywhere:

- scripts/probe-endpoints.mjs: scoring weights (50 per loss-percent,
  5000 archive penalty, p95 over median), the three-way method mix and
  why each is included, the pacing-and-inflight model, why pinned-hash
  failures fall back to header-only sampling, the universal
  System.Number key.
- scripts/apply-probe-ranking.mjs: the chain-set invariant and why the
  per-chain disagreement branches return without mutating order.
- scripts/run-ci-locally.sh: the $$-suffixed session name for
  parallel-run safety, the `set -a`/`set +a` env-sourcing idiom, the
  `${arr[*]@q}` re-quoting, the two-pane detached-tmux layout.

Matches the existing inline-comment style in scripts/start-subway.sh.

* Fix and polish `run-ci-locally.sh`

- Replace `(( elapsed++ ))` and `(( elapsed < 60 ))` with `[[ ]]` / arith
  substitution. The post-increment form returns exit status 1 when the
  pre-increment value is 0, which trips `set -e` and kills the script
  immediately after Subways finish booting (before any 'ready' line
  prints).
- Split the pane vertically (`split-window -h`) so Subway logs and
  test output sit side by side instead of stacked.
- Auto-attach to the new session after a 3-second settle, using
  `switch-client` if already inside tmux and `attach-session` otherwise.
- Pass `--pool=forks --maxWorkers=8` to the test command to match the
  CI invocation in `.github/workflows/ci.yml`.

* Add `both` mode to `run-ci-locally.sh`

Adds a third network argument that boots all 17 Polkadot+Kusama Subways
in one tmux session and runs `yarn test` (every package). Westend is
not booted because CI does not run a Westend job; Westend tests fall
back to their public endpoints from `pet-chain-endpoints.json`.

The two-array model also keeps the polkadot and kusama single-network
modes intact: `both` is the concatenation of `POLKADOT_CHAINS` and
`KUSAMA_CHAINS`, and the KNOWN_GOOD env files for the chosen networks
are concatenated into the per-run env file.

* Commit re-ranked endpoints directly to master

Drop the `peter-evans/create-pull-request` step in favour of a
`git config` / `git add` / `git commit -m '[ci skip] ...' ` / `git push`
pattern matching `update-known-good.yml`. The probe is bot-driven,
deterministic, and analogous to the known-good block bump (also
committed directly to master), so the PR layer added no review value.

* Lint fixes and Westend exclusion in `both` mode

- scripts/probe-endpoints.mjs, scripts/apply-probe-ranking.mjs: Biome
  formatting (tabs) and template-literal lint fixes.
- scripts/run-ci-locally.sh: `both` mode now invokes
  `vitest packages/polkadot packages/kusama` rather than `yarn test`,
  so Westend tests do not run (their Subway pool is not booted).
- biome.json: bump schema version from 2.4.12 to 2.4.15 to match the
  installed CLI.

* Bump block numbers

* Refactor known good block number update workflow (#601)

* Refactor known good block number update workflow

* Add CI matrix generator script

Reads chain definitions from source configs (packages/networks/src/chains/*.ts)
and endpoint lists from pet-chain-endpoints.json to generate the GitHub Actions
matrix JSON. Replaces hardcoded chain/port/endpoint mappings in workflows.

* Replace hardcoded matrix with generator script in known-good workflow

Removes the inline JSON blob and uses scripts/generate-ci-matrix.mjs.
Also removes the workflow-level retry mechanism (--retry=3 in Vitest is
sufficient) and adds per-chain failure notification via Vitest JSON output.

* Use matrix generator script in ci.yml and update-snapshot.yml

Replaces the hardcoded chain/port/endpoint matrix in both workflows with
a define-matrix job that runs scripts/generate-ci-matrix.mjs.

* [ci skip] Update known good block numbers

* [ci skip] Re-rank RPC endpoints by health

* Write documentation for recovery E2E tests (#617)

* Add comments to recovery E2E tests

* Fix tests/comments based on further review

* refactor registrar test to use new client pattern

* update known good

* update

* exclude para from snapshot

* redact data

* update snapshots

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Alexandre R. Baldé <alexandre.balde@parity.io>
Co-authored-by: Karol Kokoszka <karol@parity.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Author: 5 people