SQD Data Processing Crates

This repo contains Rust crates for blockchain data processing, indexing, and querying. They are used in various SQD components.

Crate Overview

Core Primitives & Types

Crate	Description
sqd-primitives	Basic types, range types, and SID (sequence ID) primitives used throughout the system
sqd-dataset	Dataset descriptions, table schemas, and schema options for defining data structures
sqd-array	Arrow array operations including building, slicing, sorting, I/O, and chunking utilities
sqd-bloom-filter	Bloom filter implementation for arrow buffers

Data Models & Processing

Crate	Description
sqd-data	Data models for various blockchain protocols. Used in the query engine and parquet writers
sqd-data-core	Core data processing utilities: chunk building, table processing, serialization, and sorting algorithms
sqd-data-client	HTTP client for fetching data from SQD data sources with streaming support
sqd-data-source	Abstractions for standard and mapped data sources

Query & Storage

Crate	Description
sqd-query	Query engine used by the Worker
sqd-query-example	Example demonstrating how to use the query engine with Parquet files
sqd-storage	RocksDB-based storage layer with key-value store and table management
sqd-polars	Polars DataFrame integration for high-performance data analysis

Services & Ingestion

Crate	Description
sqd-archive	Archive service for ingesting and storing data to S3 with layout management, progress tracking, and Prometheus metrics
sqd-bds	Big Data Service (WIP) - Cassandra-based data storage
sqd-hotblocks	Hotblocks database with portal-like API

Building Docker image

The "docker" workflow should be triggered manually with the following inputs:

• target — either hotblocks or sqd-archive.
• tag — a 8-byte hash of the commit. This will be the published docker tag.
• platforms — platforms to build for. Using only linux/amd64 instead of default values can save a lot of building time.