Refactor known good numbers update workflow

[rockbmb]

Original Scope: Split CI pipeline into XCM/E2E workflows with decoupled block updates

New Scope: refactor block update workflow to use Subway to increase stability and reliability

Current situation

update-known-good.yml is a single workflow that:

• Runs all tests (1000+ XCM + E2E) every 6 hours (cron schedule, line 15)
• Gates block number updates on all tests passing (save job, line 83: if: success())
• Uses a per-file matrix (lines 62-65), unlike ci.yml which moved to two monolithic jobs with Subway caching
• That per-file matrix is prone to RPC throttling

The save job downloads the KNOWN_GOOD_BLOCK_NUMBERS_*.env artifacts and commits them to master. Because it runs if: success(), a single persistent failure (e.g. Bifrost FlexibleFee bug, bifrost-io/bifrost#2083) blocks block number updates for every other chain indefinitely.

The schedule-retry job re-dispatches the workflow with only the failed tests after a 5-minute sleep. If the retry also fails, notify posts a comment to the relevant chain's notification issue (mapped via notifications.json).

Additionally, update-snapshot.yml opens PRs per-network, but if any test in that network fails, the entire network's snapshot update is dropped silently.

Proposed changes

1. Split into two workflows by test type

xcm.yml 0 */6 * * * (4x/day)

• Runs *.xcm.test.ts files only
• XCM tests are integration-level (did the bridge between A and B break?), high-frequency makes sense

e2e.yml 0 6 * * * (once daily)

• Runs *.e2e.test.ts files only
• E2E tests are behavioral (does this pallet still work?), runtime upgrades don't happen 4x/day

2. Decouple block number updates from test results

The save job should commit block numbers based on whether define-matrix (the fetch step) succeeded, not whether all downstream tests pass. Block numbers are valid regardless of test outcomes.

3. Commit block numbers regardless of test outcomes, per test-file granularity

Block numbers are fetched at the start of the run by define-matrix. They are valid as soon as that step completes. The save job should commit them unconditionally (provided define-matrix itself succeeded), regardless of which test files pass or fail downstream.

For test results: each test file (e.g. polkadot/src/polkadot.staking.e2e.test.ts) is atomic. If it passes, its snapshots are valid and can be committed. If it fails, its snapshots are not committed. The save job should collect results per test file and commit only those that passed, rather than requiring all-or-nothing. The schedule-retry and notify jobs continue to handle the failures as before.

4. Adopt Subway caching + monolithic jobs (like ci.yml)

The current per-file matrix hits production endpoints directly and is prone to RPC throttling. ci.yml already solved this with two monolithic jobs (Polkadot/Kusama) + Subway caching. The new workflows should follow that pattern.

Out of scope

• update-snapshot.yml stays as-is (manual trigger, opens PRs per network)
• The actual test failures (e.g. Bifrost) are tracked separately
• Per-file test result reporting: tracked as a follow-up issue

[rockbmb]

@xlc input appreciated, particularly on points 2 and 3.

[xlc]

If we are unconditionally commit block numbers, then we might as well not having them. Just always fetch the latest one before test start and print them (for reproduce purposes)

[rockbmb]

@xlc you're right that autocommitting block numbers -> just drop them entirely; the reason why I proposed this is because the alternative (i.e. the system at present) is that a single RPC timeout (they happen often) can block all updates, and then block numbers go stale until manually updated.

Another option is to update block numbers only for chains whose tests all pass, but this will be a relatively involved task for low value.

If we always use the latest block numbers, the Vitest test reports will become the main source of information about the state of the tests; it then becomes possible for the reports to differentiate between success | failure | timeout | etc. If block numbers are no longer in the git tree, then report persistence will also need to be addressed.

For now, I would reduce this issue to using Subway caching in the known good block number update workflow (keeping the requirement of all tests passing).

I can think more deeply about the other points.

[rockbmb]

This issue's original scope is deferred to future work; the new scope is considered fixed by #601 .

I will return to this after further redesigns to PET's CI infrastructure.