feat(load): Add metrics + gun reliability#758
Open
rodrigombsoares wants to merge 5 commits into
Open
Conversation
|
👋 rodrigombsoares, thanks for creating this pull request! To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team. Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks! |
7851fc3 to
6e06d2a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First stacked PR for devenv load-test observability and gun reliability. Extracted from the
e2e-load-prod-enablementmonolith; still scoped to local devenv only.Running existing load tests now prints per-phase timing breakdowns, failure counts, and a message ID summary at the end of each run.
CCIPLoadGunusessync.Condsingle-flight concurrency so overlapping WASP calls wait instead of failing.This PR also moves token minting on e2e tests to top level. To avoid holdings conflicts between tests leading to flakyness. Holding picking function is still sub-optimal, we need to harden that in follow up PRs.
Features
LoadMetricsCollector(metrics.go) — records Send, Confirm Send, Confirm Exec, and Total durations per successful message; tracks phase failure counts (send / confirm send / confirm exec)PrintPhaseMetricsSummarylogs Min/P90/P95/P99 per phase; bridges successful records to CCVMessageMetricssummaryCCIPLoadGunreliability —sync.Cond-based single-flight (peak concurrency asserted ≤ 1); collectsmessageIDs after confirm-send (canton needs to run sequentially due to CIDs changing on every call)LoadGunOptionswith injectableConfirmSendFuncandConfirmExecTimeout(prod-testnet workaround wired in a later PR); devenv tests usecantonSourceConfirmSend/evmSourceConfirmSendrunWASPenhancements — configurableCallTimeoutviaCANTON_LOAD_CALL_TIMEOUT; schedule logging; deferred message summary; metrics printed at end of runimpl.gologsmessageID+seqNoafter CCIP send parseTesting
Requires a running local devenv and
ccip/devenv/env-canton-evm-out.toml.make start-devenvmake run-canton2evm-loadmake run-evm2canton-loadmake run-canton2evm-token-loadmake run-evm2canton-token-load