Skip to content

fix(ccip/integration-tests): mine each tx in commitSqNrs to avoid nonce race#22017

Closed
Fletch153 wants to merge 1 commit intodevelopfrom
fix-ccip-reader-nonce-race
Closed

fix(ccip/integration-tests): mine each tx in commitSqNrs to avoid nonce race#22017
Fletch153 wants to merge 1 commit intodevelopfrom
fix-ccip-reader-nonce-race

Conversation

@Fletch153
Copy link
Copy Markdown
Collaborator

@Fletch153 Fletch153 commented Apr 14, 2026

Smoking gun

ccip_reader_test.go:515:
    Error Trace: .../integration-tests/smoke/ccip/ccip_reader_test.go:515
    Error:       Received unexpected error:
                 replacement transaction underpriced
    Test:        TestCCIPReader_ExecutedMessages_MultiChainDisjoint

CI: #22016 run 24424913522 job 71356857697. Reproduces on base branch (not introduced by #22016).

Root cause

commitSqNrs (ccip_reader_test.go:1204) loops EmitExecutionStateChanged with no Commit() between iterations:

for _, sqnr := range seqNums {
    _, err := s.contract.EmitExecutionStateChanged(s.auth, ...)
    ...
}

Generated binding in chainlink-ccip/.../ccip_reader_tester/ccip_reader_tester.go:357 is a straight-through contract.Transact(opts, "emitExecutionStateChanged", ...) — it sends a tx to the simulated mempool and does not mine. Only s.sb.Commit() mines.

s.auth is constructed by setupSimulatedBackendAndAuth with Nonce==nil, so BoundContract.transact calls PendingNonceAt(opts.From) for each tx. On ethclient/simulated the pending-pool nonce update is asynchronous with SendTransaction returning, so two back-to-back sends can read the same pending nonce. The second tx then lands at an already-occupied mempool nonce with the same suggested gas price → go-ethereum returns ErrReplaceUnderpriced ("replacement transaction underpriced").

Why only _MultiChainDisjoint

Only that test passes multi-element seqNums:

Caller (line) seqNums Loops?
429, 433, 437, 441, 471, 475 1 element no
514 [15, 17, 70] yes
518 [15, 16] yes

Single-element callers never hit the race.

Fix

Mirror emitCommitReports (line 1635), which already commits inside its loop. Add s.sb.Commit() at the end of each iteration of commitSqNrs so the next iteration's PendingNonceAt sees the bumped state nonce. Single-element callers are unaffected — the extra commit is an empty block after the existing outer Commit(); lp.Replay(ctx, 1) reads from block 1 regardless.

Verification

go test -count=10 -run TestCCIPReader_ExecutedMessages_MultiChainDisjoint ./integration-tests/smoke/ccip/ — 10/10 green.
Full TestCCIPReader_ExecutedMessages* — green.

Scope

Single file, +1 line. No production code touched.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 14, 2026

✅ No conflicts with other open PRs targeting develop

@trunk-io
Copy link
Copy Markdown

trunk-io bot commented Apr 14, 2026

Static BadgeStatic BadgeStatic BadgeStatic Badge

Failed Test Failure Summary Logs
Test_CCIPProgrammableTokenTransfer_EVM2Sui_BurnMintTokenPool The test failed due to an unspecified error during execution, with no clear indication of the specific issue from the provided logs. Logs ↗︎

View Full Report ↗︎Docs

@Fletch153 Fletch153 force-pushed the fix-ccip-reader-nonce-race branch from 18a8767 to 67eb50c Compare April 15, 2026 08:30
…ce race

TestCCIPReader_ExecutedMessages_MultiChainDisjoint flakes with
"replacement transaction underpriced" because commitSqNrs loops
EmitExecutionStateChanged without committing. Back-to-back sends race
the simulated txpool's nonce accounting so bind.TransactOpts.Nonce==nil
resolves to the same value twice, triggering the replacement-price
rule.

Mirror emitCommitReports (line 1635): Commit() after each tx so the
next iteration's PendingNonceAt sees the bumped nonce. Single-element
callers are unaffected (extra empty commit is harmless).
@cl-sonarqube-production
Copy link
Copy Markdown

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@Fletch153
Copy link
Copy Markdown
Collaborator Author

Superseded by #22031 (fresh branch off latest develop, identical diff).

@Fletch153 Fletch153 closed this Apr 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant