Skip to content

chore(ci,tooling): speed up json-loader job#3010

Merged
danceratopz merged 6 commits into
ethereum:forks/amsterdamfrom
danceratopz:speedup-json-loader
Jun 19, 2026
Merged

chore(ci,tooling): speed up json-loader job#3010
danceratopz merged 6 commits into
ethereum:forks/amsterdamfrom
danceratopz:speedup-json-loader

Conversation

@danceratopz

@danceratopz danceratopz commented Jun 18, 2026

Copy link
Copy Markdown
Member

🗒️ Description

The json-loader job takes ~23 min. This PR applies two small quick wins to reduce this:

🔗 Related Issues or PRs

✅ Checklist

  • All: Ran fast static checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    just static
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).

Cute Animal Picture

🐱

Coverage was enabled for the `json-loader` job in ethereum#2975, but it still
runs on the default Python 3.13, so `coverage` uses the slower C tracer.
Pin the job to 3.14 like `fill`, where coverage.py auto-selects
`sys.monitoring`.
The fixture-consuming `pytest` phase was pinned to `-n auto
--maxprocesses 6`, capping it at 6 workers, while the fill phase uses
`-n {{ xdist_workers }}` (`auto` in CI) and scales with the runner. The
cap carried over from the `tox` config in ethereum#2555. Use the same worker
setting so phase 2 is not throttled below the available cores.
@danceratopz danceratopz added C-chore Category: chore A-tooling Area: Improvements or changes to auxiliary tooling such as uv, ruff, mypy, ... A-ci Area: Continuous Integration labels Jun 18, 2026
@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.82%. Comparing base (5f8c109) to head (854a6e9).
⚠️ Report is 1 commits behind head on forks/amsterdam.

Additional details and impacted files
@@                 Coverage Diff                 @@
##           forks/amsterdam    #3010      +/-   ##
===================================================
- Coverage            93.20%   92.82%   -0.39%     
===================================================
  Files                  620      620              
  Lines                38777    36714    -2063     
  Branches              3342     3341       -1     
===================================================
- Hits                 36144    34081    -2063     
  Misses                1773     1773              
  Partials               860      860              
Flag Coverage Δ
unittests 92.82% <ø> (-0.39%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Add `--durations=50` to both phases of the `json-loader` recipe (the
`fill` step and the `pytest` step) so CI surfaces the slowest fills and
fixture executions. The standalone `fill` recipe already reports these;
this brings `json-loader` to parity and gives data to guide further
trimming of the `eels_base_coverage` set.
`--dist=loadfile` pinned every test in a file to one worker, so the two
`test_tools_new_fork.py::test_end_to_end` cases (~256s and ~203s) ran
back-to-back on a single worker while the rest sat idle.

Switch phase 2 to `--dist=loadgroup` and tag each fixtures file's cases
with an `xdist_group` keyed by the file. Fixture cases keep file-locality
(the `data` JSON cache parses once per file), while the ungrouped tooling
tests distribute individually onto separate workers.
The previous commit added the `xdist_group` in a default-ordered
`pytest_collection_modifyitems`, but xdist reads each item's group before
that hook runs, so the marker was ignored and phase 2 still ran as plain
`load` (fixtures scattered, re-parsing their JSON per case).

Mark the hook `tryfirst=True` so the group is assigned first: fixture
cases regroup by file (parsed once) and the slow tooling tests still
spread across workers.
Reverts the `loadgroup` rebalance and its `tryfirst` follow-up. In CI the
regrouping did not reliably split the two slow `new_fork` CLI tests:
phase 2 stayed ~420-470s either way (monsters serialized under one
layout, fixtures imbalanced by file-grouping under the other), within
runner variance. Phase 2 returns to `--dist=loadfile`.

The slow tooling tests are better moved out of json-loader entirely,
tracked as a follow-up.
@danceratopz

Copy link
Copy Markdown
Member Author

Hey @SamWilsn, thanks for the review here. I reverted the attempts to speed-up the json-loader job by splitting up the test_tools_new_fork.py::test_end_to_end test across different workers. This didn't seem to be working as intended. I think we can leave this as for a follow-up and get the other quick wins in for speed-up.

The new_fork tests take ~225s each and it looks like they get placed on one xdist worker, leading to poorly-balanced workers/starvation. As these tests are unit tests for the new fork tool and have nothing todo with JSON, I'd consider moving these out of the json-loader tests; there might be other candidates, will have to check. This would make it easier to split them and is cleaner.

@danceratopz danceratopz marked this pull request as ready for review June 19, 2026 06:47
@danceratopz danceratopz merged commit babe45c into ethereum:forks/amsterdam Jun 19, 2026
18 checks passed
@danceratopz

Copy link
Copy Markdown
Member Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ci Area: Continuous Integration A-tooling Area: Improvements or changes to auxiliary tooling such as uv, ruff, mypy, ... C-chore Category: chore

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants