Skip to content

ref(snapshots): Use async I/O for snapshot image processing#3187

Draft
lcian wants to merge 5 commits intomasterfrom
lcian/ref/snapshots-async-io
Draft

ref(snapshots): Use async I/O for snapshot image processing#3187
lcian wants to merge 5 commits intomasterfrom
lcian/ref/snapshots-async-io

Conversation

@lcian
Copy link
Member

@lcian lcian commented Mar 4, 2026

Enters the tokio runtime early with a single block_on call wrapping all
async work, instead of calling block_on multiple times in the upload loop.

  • compute_sha256_hash is now async, using tokio::fs::File and AsyncReadExt
    for chunked reading instead of sync std::fs::File / std::io::Read
  • File opens for uploads also use tokio::fs::File::open inside the async block
  • manifest_entries wrapped in Arc<tokio::sync::Mutex> for mutation inside
    the async block, extracted via Arc::try_unwrap after the runtime completes
  • Added fs and io-util tokio features in Cargo.toml

Stacked on #3186.

We're still computing the hash and enqueuing the files sequentially. Eventually we can make this concurrent or even parallel in case we want to use a multi-threaded runtime.

lcian and others added 5 commits March 4, 2026 15:04
Use `put_file` instead of `put` when uploading images to objectstore,
streaming file contents from disk rather than loading entire files into
memory. Hash computation also now uses buffered reads.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enter the tokio runtime early with a single `block_on` call instead of
calling it multiple times in the upload loop. Convert `compute_sha256_hash`
to async using `tokio::fs::File` and `AsyncReadExt`, and use async file
opens for uploads. Wrap `manifest_entries` in `Arc<tokio::sync::Mutex>`
for safe mutation inside the async block.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lcian lcian marked this pull request as ready for review March 4, 2026 18:13
@lcian lcian requested review from a team and szokeasaurusrex as code owners March 4, 2026 18:13
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.


Ok(Arc::try_unwrap(manifest_entries)
.expect("all references should be dropped after runtime completes")
.into_inner())
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unnecessary Arc/Mutex when HashMap could be returned directly

Medium Severity

manifest_entries is wrapped in Arc<Mutex<>> and later extracted via Arc::try_unwrap(...).expect(...), but there's no concurrent access — everything runs sequentially in a single async block on a single-threaded runtime. The HashMap could simply be created as a local mut variable inside the async block, mutated directly (no .lock().await needed), and returned as part of the Ok(manifest_entries) result. This would eliminate the Arc, Mutex, try_unwrap, into_inner, and the .expect() panic site entirely.

Fix in Cursor Fix in Web

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lcian Can you please check whether this comment is valid before I do a full review 🙏

@lcian lcian force-pushed the lcian/fix/snapshots-object-key branch from 805e229 to effb34b Compare March 4, 2026 18:45
Base automatically changed from lcian/fix/snapshots-object-key to master March 4, 2026 18:58
@lcian lcian marked this pull request as draft March 6, 2026 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants