S5

S5 is a modular, high-performance content-addressed storage (CAS) network built on top of Iroh.

It gives you:

• A small, well-factored Rust API for content addressing, blob transport, and mutable registries.
• FS5, an encrypted, content-addressed filesystem with snapshot semantics.
• A node + vup CLI that wires everything together with pluggable S5 store backends (local FS, S3, Sia, in-memory).

If you are familiar with IPFS or Iroh: think of S5 as a focused, Rust-native toolkit for building distributed storage and sync flows, with strong separation between wire-stable protocol types and high-level ergonomics.

Status: Version 1.0.0-beta.1. Wire-level protocol types (Hash, BlobId, BlobLocation, registry/stream messages) are treated as stable for the 1.0 series; library APIs may still evolve between betas.

Quick Install

cargo install --git https://github.com/s5-dev/s5-rs vup_cli

Minimal workflow (sigil grammar — +name is a vault reference; the daemon auto-starts on first invocation):

vup new +music                # create + onboard a vault, prints recovery key
vup +music add ~/Music        # configure a source path
vup +music snap               # snapshot + publish to configured stores
vup +music mount /mnt/music   # FUSE-mount the vault read-only

Architecture

The S5 stack is composed of layered crates, moving from low-level primitives to high-level applications:

graph TD
    Vup[vup_cli] --> Node[s5_node]
    LegacyCLI[s5_cli] --> Node
    Node --> Fuse[s5_fuse]
    Node --> FS[s5_fs_v2]
    Node --> Net[s5_blobs / s5_registry]
    FS --> Core[s5_core]
    Net --> Core
    Node --> Stores[blob_stores/*]
    Stores --> Core

Loading

1. Core (s5_core): Defines the wire-stable protocol types (Hash, BlobId, BlobLocation) and abstract traits (Store, RegistryApi).
2. Storage (blob_stores/*): Concrete backends for storing immutable blobs (Local disk, S3, Sia, Memory).
3. Network (s5_blobs, s5_registry): Iroh-based transport protocols for exchanging blobs and registry updates.
4. Filesystem (s5_fs_v2): A content-addressed filesystem abstraction with Node-based snapshots and prolly-tree merges.
5. Orchestration (s5_node): Wires together storage, networking, filesystem logic, and FUSE mounts into a runnable daemon.
6. Interface (vup_cli, s5_cli): vup_cli is the primary user-facing tool, a thin RPC frontend over the daemon; s5_cli is a low-level operational CLI (GC, store verify, HTTP import, FS5-on-disk snapshot ops) kept until vup_cli reaches parity. s5_fuse lives inside the daemon; mounts are driven via the MountVault RPC.

Typical Data Flow

1. Ingest: an importer (local filesystem or HTTP) writes file contents into a BlobStore and stores FileRefs in an FS5 directory snapshot.
2. Addressing: the directory snapshot and blobs are addressed by BLAKE3 hashes (Hash / BlobId).
3. Publication: a registry stream entry points to the latest snapshot hash.
4. Networking: other nodes fetch blobs via s5_blobs and the registry state via s5_registry over Iroh.

Workspace Crates

Crate	Description
s5_core	Protocol Primitives. Hash, BlobId, Store trait, RegistryApi trait. The foundation of the stack.
s5_fs	Filesystem Logic. Implements DirV1 snapshots, directory actors, and the high-level FS5 API.
s5_node	Server & Orchestration. Configures and runs the S5 node, managing stores, networking, mounts, and sync.
vup_cli	Primary Command Line Interface. Vup Vault — vup new +<vault>, vup +<vault> snap, vup +<vault> mount, etc. Thin RPC frontend over the daemon.
s5_cli	Low-level operational CLI. GC, verify-local, HTTP import, FS5-on-disk snapshot ops. Will fold into vup_cli once it reaches parity.
s5_blobs	Blob Transport. Iroh-based protocol for serving and fetching blobs over the network.
s5_registry	Registry Transport. Iroh-based protocol for syncing mutable registry entries.
s5_fuse	FUSE Mount. Mounts an S5 filesystem locally using FUSE.
blob_stores	Storage Backends. Implementations of the Store trait: local, s3, sia, memory.
importers	Data Ingestion. Tools to import data into S5 from external sources (local fs, http).

Key Concepts

• Blob: Immutable bytes, identified by a BLAKE3 content hash. Everything else in S5 is ultimately built on blobs.
• BlobId: Canonical identifier combining the content Hash with the content size (u64) for efficient allocation and verification.
• Snapshot (DirV1): A content-addressed directory state at a specific point in time. FS5 mutability comes from appending new snapshots; each snapshot is a single CBOR structure.
• Registry: A mutable, append-only log of signed messages mapping a public key (StreamKey) to payloads (e.g., a root snapshot hash). This is how S5 handles mutability on top of immutable blobs.
• Storage (Store / BlobStore): A trait for pluggable backends (local disk, S3, Sia renterd, memory, or your own), with a BlobStore façade on top.
• Location (BlobLocation): Decouples what data is (hash) from where it lives (peer, bucket, Sia host, etc.).
• Networking (Iroh): Two small protocols, s5_blobs and s5_registry, run on top of Iroh for blob and registry exchange.

Comparison

vs. IPFS

• Library-First: S5 is designed as a modular Rust toolkit, not a monolithic daemon. You can use just the s5_fs crate without the networking, or just s5_blobs without the filesystem.
• Transport: S5 uses Iroh (QUIC + BLAKE3) for peer-to-peer connections, avoiding the complexity and overhead of the full libp2p stack and global DHT.
• Storage: S5 treats external storage (S3, Sia) as first-class backends via the Store trait, rather than requiring "pinning services" or sidecar processes.

vs. Iroh

• Layering: S5 is built on top of Iroh. Iroh provides the "pipes" (connections, magichash, hole punching); S5 provides the "OS" (hierarchical filesystem, mutable registry, multi-backend storage abstraction).
• Protocol: S5 defines specific wire formats (DirV1, StreamMessage) that are distinct from Iroh's internal protocols.

vs. Object Storage (S3)

• Addressing: S3 is location-addressed (bucket/key). S5 is content-addressed (Hash). This enables automatic deduplication and verification.
• Encryption: S5 supports client-side encryption (XChaCha20-Poly1305) at the filesystem layer (s5_fs), meaning the storage backend (S3, Sia) never sees plaintext or directory structures.

Innovative Workflows

Encrypted Sync via Untrusted Peers

S5 enables end-to-end encrypted file synchronization between two nodes using a third, untrusted node as a relay.

1. Node A writes files to an encrypted FS5 snapshot.
2. Node A pushes the encrypted blobs and registry update to Node B (untrusted relay).
3. Node C pulls the registry update from Node B, verifies the signature, and fetches the encrypted blobs.
4. Node C decrypts the snapshot locally using the shared secret. Node B sees only opaque blobs and signed messages.

Decentralized CDN (Sia + S5)

By implementing the Store trait for Sia (s5_store_sia), an S5 node can act as a gateway.

• Upload: Users upload files to the S5 node, which streams them directly to the Sia network.
• Access: The S5 node provides BlobLocation::SiaFile to clients, allowing them to fetch data directly from Sia hosts (if supported) or proxy through the node, all addressed by a stable S5 BlobId.

Documentation

• CLI Reference: Comprehensive guide to the s5 command.
• Configuration: Node configuration schema.
• S5 Workflows: Detailed examples of sync and import flows.
• Sia Web CDN: Guide for using S5 with Sia.

Contributing

Contributions are welcome! Please feel free to open an issue or submit a pull request.

• GitHub Issues: Bug reports, feature requests, and questions.
• AGENTS.md: Guidelines for working with the codebase (code style, design principles, crate organization).

Development

Prerequisites

• Rust (stable)

Build & Test

# Build all crates
cargo build --workspace

# Run all tests
cargo test --workspace

# Run lints
cargo clippy --workspace --all-targets -- -D warnings

Running the CLI

# Run the CLI directly
cargo run -p vup_cli -- --help

# Initialise a vault and snapshot some data (daemon auto-starts as needed)
cargo run -p vup_cli -- new +photos
cargo run -p vup_cli -- +photos add ~/Pictures
cargo run -p vup_cli -- +photos snap

License

This project is dual-licensed under the MIT and Apache 2.0 licenses.

This project is supported by a Sia Foundation grant.