ref(snapshots): Use async I/O for snapshot image processing#3187
ref(snapshots): Use async I/O for snapshot image processing#3187
Conversation
Use `put_file` instead of `put` when uploading images to objectstore, streaming file contents from disk rather than loading entire files into memory. Hash computation also now uses buffered reads. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enter the tokio runtime early with a single `block_on` call instead of calling it multiple times in the upload loop. Convert `compute_sha256_hash` to async using `tokio::fs::File` and `AsyncReadExt`, and use async file opens for uploads. Wrap `manifest_entries` in `Arc<tokio::sync::Mutex>` for safe mutation inside the async block. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
|
|
||
| Ok(Arc::try_unwrap(manifest_entries) | ||
| .expect("all references should be dropped after runtime completes") | ||
| .into_inner()) |
There was a problem hiding this comment.
Unnecessary Arc/Mutex when HashMap could be returned directly
Medium Severity
manifest_entries is wrapped in Arc<Mutex<>> and later extracted via Arc::try_unwrap(...).expect(...), but there's no concurrent access — everything runs sequentially in a single async block on a single-threaded runtime. The HashMap could simply be created as a local mut variable inside the async block, mutated directly (no .lock().await needed), and returned as part of the Ok(manifest_entries) result. This would eliminate the Arc, Mutex, try_unwrap, into_inner, and the .expect() panic site entirely.
There was a problem hiding this comment.
@lcian Can you please check whether this comment is valid before I do a full review 🙏
805e229 to
effb34b
Compare


Enters the tokio runtime early with a single
block_oncall wrapping allasync work, instead of calling
block_onmultiple times in the upload loop.compute_sha256_hashis now async, usingtokio::fs::FileandAsyncReadExtfor chunked reading instead of sync
std::fs::File/std::io::Readtokio::fs::File::openinside the async blockmanifest_entrieswrapped inArc<tokio::sync::Mutex>for mutation insidethe async block, extracted via
Arc::try_unwrapafter the runtime completesfsandio-utiltokio features inCargo.tomlStacked on #3186.
We're still computing the hash and enqueuing the files sequentially. Eventually we can make this concurrent or even parallel in case we want to use a multi-threaded runtime.