Skip to content

fix(stdlib): prevent block hash deletion for blocks still referenced by tips#22450

Closed
AztecBot wants to merge 2 commits into
merge-train/barretenbergfrom
claudebox/fix-bb-merge-train-ci
Closed

fix(stdlib): prevent block hash deletion for blocks still referenced by tips#22450
AztecBot wants to merge 2 commits into
merge-train/barretenbergfrom
claudebox/fix-bb-merge-train-ci

Conversation

@AztecBot
Copy link
Copy Markdown
Collaborator

@AztecBot AztecBot commented Apr 9, 2026

Summary

Fixes a crash loop in the p2p block stream caused by handleChainFinalized in L2TipsStoreBase deleting block hashes that are still referenced by other tips.

Root Cause

handleChainFinalized called deleteBlockHashesBefore(event.block.number) which could delete hashes for blocks still referenced by other tips (e.g. proposedCheckpoint, proven). When getL2Tips() subsequently tried to look up the hash for those tips via getBlockId(), it threw Block hash not found for block number N, putting the p2p block stream into an unrecoverable error loop that prevented block sync and caused the e2e_bot test to timeout.

Fix

Compute a safe deletion bound as the minimum of all active tip block numbers before deleting. This ensures no block hash is deleted while it's still referenced by any tip.

Test plan

  • All 45 l2_block_stream tests pass
  • All 207 archiver KV store tests pass
  • Full yarn-project build succeeds

ClaudeBox log: https://claudebox.work/s/4aa87310c1ecd68c?run=1

…by tips

handleChainFinalized was calling deleteBlockHashesBefore(event.block.number) which
could delete hashes still referenced by other tips (proposedCheckpoint, proven, etc).
This caused getBlockId to throw 'Block hash not found', putting the p2p block stream
into an unrecoverable error loop.

Now computes a safe deletion bound as the minimum of all active tip block numbers,
ensuring no referenced block hash is deleted.
@AztecBot AztecBot added ci-draft Run CI on draft PRs. claudebox Owned by claudebox. it can push to this PR. labels Apr 9, 2026
@AztecBot
Copy link
Copy Markdown
Collaborator Author

Automatically closing this stale claudebox draft PR (no updates for 5+ days). Re-open if still needed.

@AztecBot AztecBot closed this Apr 16, 2026
@guglez
Copy link
Copy Markdown

guglez commented May 22, 2026

We hit this exact bug on our mainnet validator running v4.2.1. The node stopped attesting silently - the container kept running, archiver and HTTP API were healthy, but the sequencer entered an internal error loop at ~2 errors/second (Block hash not found for block number 46069). No crash, no obvious signal. Recovery: stop container, delete the p2p LMDB stores (p2p, p2p-archive, p2p-peers, p2p-attestation), restart. Node came back immediately. Any ETA on a release with this fix included?

@aminsammara
Copy link
Copy Markdown
Contributor

We hit this exact bug on our mainnet validator running v4.2.1. The node stopped attesting silently - the container kept running, archiver and HTTP API were healthy, but the sequencer entered an internal error loop at ~2 errors/second (Block hash not found for block number 46069). No crash, no obvious signal. Recovery: stop container, delete the p2p LMDB stores (p2p, p2p-archive, p2p-peers, p2p-attestation), restart. Node came back immediately. Any ETA on a release with this fix included?

Hi @guglez looking into this

@spypsy spypsy reopened this May 22, 2026
@spypsy spypsy marked this pull request as ready for review May 22, 2026 11:02
@spalladino
Copy link
Copy Markdown
Contributor

Hey @guglez, we fixed an instance of this issue on the upcoming V5 release in #23295. Can you share some logs, to confirm your error is indeed the same and would be fixed by that?

@spypsy spypsy closed this May 22, 2026
@spypsy
Copy link
Copy Markdown
Member

spypsy commented May 22, 2026

relevant fix now in #23505

@guglez
Copy link
Copy Markdown

guglez commented May 22, 2026

Hey @spalladino

Is this log line enough?

  ERROR: kv-store:lmdb-v2 Failed to commit transaction: Error: Block hash not found for block number 46069
      at L2TipsKVStore.getBlockId (file:///usr/src/yarn-project/stdlib/dest/block/l2_block_stream/l2_tips_store_base.js:76:19)
      at async Promise.all (index 2)
      at async file:///usr/src/yarn-project/stdlib/dest/block/l2_block_stream/l2_tips_store_base.js:14:93

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants