Skip to content

try to fix deadlock/timeout in logpoller tests#22337

Merged
Tofel merged 1 commit intodevelopfrom
fix/logpoller-close-replay-deadlock
May 8, 2026
Merged

try to fix deadlock/timeout in logpoller tests#22337
Tofel merged 1 commit intodevelopfrom
fix/logpoller-close-replay-deadlock

Conversation

@Tofel
Copy link
Copy Markdown
Contributor

@Tofel Tofel commented May 7, 2026

15 successful runs: https://github.com/smartcontractkit/chainlink/actions/runs/25488891367
Requires: smartcontractkit/chainlink-evm#463 (merged ✅)

Made with AI and explained as:

  That said, the fix is mechanically sufficient: recvReplayComplete now also selects on lp.stopCh, so when Close() runs close(lp.stopCh) and then wg.Wait(), every recvReplayComplete goroutine — whether spawned before or after close(stopCh) — observes the closed channel and exits. The previous "send must arrive before
  goroutine starts receiving" timing dependence is gone.

Hoping to solve deadlocks like this one, which are endemic in chainlink repo:

    FAIL    github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/ocr2keeper/evmregistry/v21/logprovider    900.816s

    === Failed
    === FAIL: core/services/llo/telem TestSample (0.00s)
        sampling_test.go:140:
                Error Trace:    /home/runner/_work/chainlink/chainlink/core/services/llo/telem/sampling_test.go:140
                Error:          Should be false
                Test:           TestSample

    === FAIL: core/services/ocr2/plugins/ocr2keeper/evmregistry/v21/logprovider  (0.00s)
    panic: test timed out after 15m0s
        running tests:
            TestIntegration_LogEventProvider_UpdateConfig (14m53s)

or

     === FAIL: core/services/ocr2/plugins/ocr2keeper/evmregistry/v21/logprovider
     (0.00s)
     panic: test timed out after 15m0s
        running tests:
                TestIntegration_LogEventProvider_Backfill (14m52s)

     goroutine 93719 [running]:
     testing.(*M).startAlarm.func1()
        /opt/hostedtoolcache/go/1.26.2/x64/src/testing/testing.go:2802 +0x34b
     created by time.goFunc
        /opt/hostedtoolcache/go/1.26.2/x64/src/time/sleep.go:215 +0x2d

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

✅ No conflicts with other open PRs targeting develop

@trunk-io
Copy link
Copy Markdown

trunk-io Bot commented May 7, 2026

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@Tofel Tofel force-pushed the fix/logpoller-close-replay-deadlock branch from 272723f to 95e0ddb Compare May 8, 2026 06:35
@Tofel Tofel marked this pull request as ready for review May 8, 2026 06:35
@Tofel Tofel requested a review from a team as a code owner May 8, 2026 06:35
Copilot AI review requested due to automatic review settings May 8, 2026 06:35
@Tofel Tofel requested review from a team as code owners May 8, 2026 06:35
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Risk Rating: MEDIUM

This PR updates the repository’s github.com/smartcontractkit/chainlink-evm dependency to pick up an upstream fix intended to reduce logpoller-related test deadlocks/timeouts, and regenerates generated artifacts impacted by that bump (notably config docs).

Changes:

  • Bump github.com/smartcontractkit/chainlink-evm to v0.3.4-0.20260507171202-46e6a397da2d across root and submodule go.mod files (and corresponding go.sum updates).
  • Regenerate docs/CONFIG.md, which (notably) changes Mantle chain default GasEstimator.LimitDefault/LimitMax values.

Scrupulous human review needed:

  • Verify the chainlink-evm bump actually contains the intended goroutine shutdown / deadlock fix (since the core logic change is upstream and not visible in this diff).
  • Confirm the Mantle Mainnet/Sepolia default gas estimator limit changes in docs/CONFIG.md are intentional and acceptable, and update the PR description accordingly (see comments).

Reviewed changes

Copilot reviewed 8 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
system-tests/tests/go.mod Updates chainlink-evm version for system test module.
system-tests/tests/go.sum Updates checksums for new chainlink-evm version.
system-tests/lib/go.mod Updates chainlink-evm version for system test library module.
system-tests/lib/go.sum Updates checksums for new chainlink-evm version.
integration-tests/load/go.mod Updates chainlink-evm version for load test module.
integration-tests/load/go.sum Updates checksums for new chainlink-evm version.
integration-tests/go.mod Updates chainlink-evm version for integration test module.
integration-tests/go.sum Updates checksums for new chainlink-evm version.
deployment/go.mod Updates chainlink-evm version for deployment module.
deployment/go.sum Updates checksums for new chainlink-evm version.
core/scripts/go.mod Updates chainlink-evm version for scripts module.
core/scripts/go.sum Updates checksums for new chainlink-evm version.
go.mod Updates root module chainlink-evm requirement.
go.sum Updates root module checksums for new chainlink-evm version.
docs/CONFIG.md Regenerated config docs; changes Mantle chain gas estimator limit defaults.

Comment thread docs/CONFIG.md
Comment thread docs/CONFIG.md
@cl-sonarqube-production
Copy link
Copy Markdown

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@Tofel Tofel enabled auto-merge May 8, 2026 09:33
@Tofel Tofel added this pull request to the merge queue May 8, 2026
Merged via the queue into develop with commit 70dbd23 May 8, 2026
222 of 224 checks passed
@Tofel Tofel deleted the fix/logpoller-close-replay-deadlock branch May 8, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants