perf: parallel range-GET S3 downloads for large objects#224
perf: parallel range-GET S3 downloads for large objects#224joshfriend wants to merge 1 commit intomainfrom
Conversation
Replace the single GetObject stream in S3.Open with parallel range-GET requests for objects larger than 32 MiB. 8 workers download chunks concurrently and reassemble them in order via io.Pipe, multiplying S3 throughput for cold snapshot downloads (observed 32 MB/s single-stream vs expected ~250+ MB/s with parallel connections).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: afbf08ad55
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| results := make([]chan chunkResult, numChunks) | ||
| for i := range results { | ||
| results[i] = make(chan chunkResult, 1) |
There was a problem hiding this comment.
This allocates a dedicated buffered channel for every chunk, so workers can complete far ahead of the writer and park each 32 MiB []byte in memory; with slow downstream reads (or one early chunk lagging), buffering grows toward the full object size instead of numWorkers × chunkSize, which can OOM on multi-GB objects. The current structure needs bounded in-flight chunk buffering tied to worker count rather than chunk count.
Useful? React with 👍 / 👎.
| if writeErr != nil { | ||
| continue // drain remaining channels so goroutines can exit | ||
| } |
There was a problem hiding this comment.
After writeErr is set, the loop intentionally keeps draining all chunk channels, and because the work queue is prefilled and never canceled, workers continue fetching the rest of the object even when the pipe/write path has already failed (for example, client disconnects). That turns a single failure into full-object S3 traffic and delayed cleanup; the code should cancel outstanding work as soon as a terminal write/read error occurs.
Useful? React with 👍 / 👎.
1 participant
Summary
GetObjectstream inS3.Openwith parallel range-GET requests for objects larger than 32 MiBio.Pipe, multiplying S3 throughput for cold snapshot downloadsSplit out from #218 (now closed) — the parallel tar extraction and in-process zstd changes were dropped after discovering they cause a 30x regression on git mirror snapshots with multi-GB packfiles (see #217).