[DISCUSSION] `Runtime` support enhancement

[CTTY]

Iceberg-rust provides a centralized Runtime used to spawn parallelizable work (e.g., table scans, loading delete files). Today our runtime crate exposes a JoinHandle that is essentially just a thin wrapper around Tokio’s handle and offers little configurability:

pub struct JoinHandle<T>(task::JoinHandle<T>);

Tokio has a rich set of runtime knobs (thread counts, blocking pool size, thread naming, etc.), but our current abstraction doesn’t give users a good way to tune these settings.

There are two broad directions:

Option A: Make Runtime pluggable

Expose a Runtime trait so users can provide their own runtime implementation.
One possible shape:

pub trait Runtime: Send + Sync + 'static {
    fn spawn<T>(
        &self,
        fut: std::pin::Pin<Box<dyn Future<Output = T> + Send + 'static>>,
    ) -> std::pin::Pin<Box<dyn Future<Output = T> + Send + 'static>>
    where
        T: Send + 'static;

    fn spawn_blocking<T>(
        &self,
        f: Box<dyn FnOnce() -> T + Send + 'static>,
    ) -> std::pin::Pin<Box<dyn Future<Output = T> + Send + 'static>>
    where
        T: Send + 'static;
}

Pros:

• Maximum flexibility: users can integrate with whatever executor/runtime they already use.

Cons:

• More complexity for users (they may need to build/manage a runtime).

Option B: Expose Tokio runtime configuration

Keep Tokio as the default runtime, but add a RuntimeConfig that maps to Tokio’s builder options.

Pros:

• Straightforward for most users

Cons:

• Less flexible: users are effectively opting into Tokio (though in practice many already are).

[liurenjie1024]

More complexity for users (they may need to build/manage a runtime).

We could eliminate this complexity by providing a concrete runtime in this repo, backed by tokio.

I would +1 for option a.