perf: parallel range-GET S3 downloads for large objects#225
perf: parallel range-GET S3 downloads for large objects#225joshfriend wants to merge 1 commit intomainfrom
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: afbf08ad55
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
joshfriend
force-pushed
the
jfriend/parallel-s3-range-get
branch
2 times, most recently
from
caf48da to
7ba1a28
Compare
Replace the single GetObject stream in S3.Open with parallel range-GET requests for objects larger than 32 MiB. 8 workers download chunks concurrently and reassemble them in order via io.Pipe, multiplying S3 throughput for cold snapshot downloads (observed ~100 MB/s single-stream vs 400+ MB/s with parallel connections on staging hardware).
joshfriend
force-pushed
the
jfriend/parallel-s3-range-get
branch
from
7ba1a28 to
3294383
Compare
| var writeErr error | ||
| for _, ch := range results { | ||
| r := <-ch | ||
| if writeErr != nil { |
There was a problem hiding this comment.
If you get an error this code will still wait for all the S3 Gets to run, I think you need a new context that you can cancel on error to cancel existing gets.
| var wg sync.WaitGroup | ||
| for range numWorkers { |
There was a problem hiding this comment.
Use errgroup.WithCancel() here instead of a sync.WaitGroup. That will cancel the entire set of workers on any error or parent context cancellation.
|
|
||
| // Write chunks in order. Each receive blocks until that chunk's worker | ||
| // finishes, while other workers continue downloading concurrently. | ||
| var writeErr error |
There was a problem hiding this comment.
When you switch to errgroup, run this in a separate goroutine and do a return eg.Wait() at the end.
4 participants
Replace the single GetObject stream in S3.Open with parallel range-GET requests for objects larger than 32 MiB. 8 workers download chunks concurrently and reassemble them in order via io.Pipe, multiplying S3 throughput for cold snapshot downloads (observed ~100 MB/s single-stream → 400+ MB/s with parallel connections on staging hardware).