fix: chunk store_load over RPC to avoid msgpack BufferFull (#8440)#9540
Draft
hiijoshi wants to merge 1 commit intoborgbackup:masterfrom
Draft
fix: chunk store_load over RPC to avoid msgpack BufferFull (#8440)#9540hiijoshi wants to merge 1 commit intoborgbackup:masterfrom
hiijoshi wants to merge 1 commit intoborgbackup:masterfrom
Conversation
Large repositories can have a chunks index exceeding the msgpack unpacker buffer size, causing BufferFull errors over SSH/RPC. Instead of loading the entire value in one RPC call, add two new server-side methods (store_get_size, store_load_chunk) and override store_load in RemoteRepository to fetch data in MAX_DATA_SIZE pieces, reassembling on the client side. Fixes borgbackup#8440
hiijoshi
force-pushed
the
fix/store-load-chunked-rpc
branch
from
2a9334e to
688afbf
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9540 +/- ##
=======================================
Coverage 83.39% 83.40%
=======================================
Files 87 87
Lines 15469 15484 +15
Branches 2321 2322 +1
=======================================
+ Hits 12901 12915 +14
- Misses 1818 1819 +1
Partials 750 750 ☔ View full report in Codecov by Sentry. |
ThomasWaldmann
requested changes
Mar 24, 2026
Contributor
Author
|
I apologize, Thomas. I was eager to help with #8440, but I realize I should have prioritized the previous PRs as you instructed. I will set this aside and go back to other PR right now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Large repositories can have a chunks index exceeding the
msgpackunpacker buffer size, causingBufferFullerrors over SSH/RPC connections. This occurs because the currentstore_loadimplementation attempts to fetch the entire value in a single RPC call.Changes
To resolve this, I have:
store_get_sizeandstore_load_chunk.store_loadinRemoteRepositoryto fetch data inMAX_DATA_SIZEpieces.This ensures that no single RPC response exceeds the buffer limits while maintaining compatibility.
Fixes
Fixes #8440