Merged
Conversation
- Rename config field for clarity (better describes what it does) - Change default from 1 to 16 segments - Rationale: Bridge already limits concurrent Erigon requests via buffering - Streaming architecture means data flows through without accumulation - 16 concurrent segments provides good parallelism while preventing the 48-segment explosion that caused memory/stream issues The semaphore limits how many segments can process logs concurrently, but the bridge's max_concurrent_executions prevents too much in-flight work at the Erigon level. This allows better throughput without risking resource exhaustion.
- Use renamed config field when creating logs semaphore - Update comment to reflect "limiting" rather than "sequencing" - Log message shows semaphore capacity for observability
Semaphore changes: - Format semaphore acquire chain for readability - Permit is held for entire log sync operation per segment Clippy/dead_code fixes: - Rename bytes_written to _bytes_written (unused but kept for future use) - Add #[allow(dead_code)] to ProgressUpdate struct (legacy tracking code) - Add #[allow(dead_code)] to update_stage and update_progress methods (may be used in future for enhanced progress tracking)
Apply rustfmt formatting to multi-line field call.
Collaborator
Author
|
Ethereum mainnet data sync (from customized erigon) Before changeset: 
After changeset: 
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Start syncing logs at the beginning, don't wait. Full sync takes ~8 hours on my hardware. Tested with a cold erigon.