feat: pluggable index cache via CacheBackend trait

[wjones127]

Previously the Session's index cache was hardcoded to Moka. This adds a
CacheBackend trait so users can provide their own cache implementation,
with MokaCacheBackend as the default.

Serialization and zero-IO reconstruction are in the follow-up PR #6223.

[github-actions]

PR Review: feat: pluggable index cache via CacheBackend trait

P0: TOCTOU race in get_or_insert — dedup is broken

The check-then-register pattern in get_or_insert has a race between the two separately-locked critical sections:

// Block 1: check (drops lock at end of block)
{
    let map = self.in_flight.lock().await;
    if let Some(rx) = map.get(&cache_key) { ... }
}
// <-- gap: another task can also see an empty map here

// Block 2: register as leader (re-acquires lock)
let (tx, rx) = tokio::sync::watch::channel(None);
{
    let mut map = self.in_flight.lock().await;
    map.insert(cache_key.clone(), rx);
}

Between dropping the lock in block 1 and re-acquiring it in block 2, N tasks can all see an empty in-flight map and all decide they are the "leader." The last one to insert overwrites the earlier receivers, so:

1. The loader runs N times (dedup completely defeated).
2. Tasks that cloned an earlier receiver get RecvError when that sender drops without the value propagating through the map.

Fix: Merge the check and register into a single critical section:

let mut map = self.in_flight.lock().await;
if let Some(rx) = map.get(&cache_key) {
    let mut rx = rx.clone();
    drop(map);
    // ... wait for leader ...
} else {
    let (tx, rx) = tokio::sync::watch::channel(None);
    map.insert(cache_key.clone(), rx);
    drop(map);
    // ... run loader ...
}

The test_get_or_insert_dedup test passes by luck because tokio::task::yield_now() with a broadcast barrier doesn't reliably interleave into this gap. A test with a longer sleep or explicit synchronization at the gap point would expose this.

P1: type_tag relies on unspecified pointer identity

fn type_tag<T: 'static>() -> [u8; 8] {
    (std::any::type_name::<T>().as_ptr() as u64).to_le_bytes()
}

The Rust spec does not guarantee that type_name::<T>().as_ptr() returns the same address across calls, nor that distinct types produce distinct addresses (string deduplication/interning is compiler-implementation-defined). The old code used TypeId as a key component, which has correct identity semantics.

If two different types happen to collide (same pointer or same string literal address), the cache will silently return a wrong type and the downcast::<T>().unwrap() will panic at runtime.

Consider using TypeId bytes instead. TypeId implements Hash and Eq, so you could hash it to get a stable discriminator, or use transmute to extract its bytes (it's currently 128 bits).

P1: invalidate_prefix is fire-and-forget

pub fn invalidate_prefix(&self, prefix: &str) {
    let prefix_bytes = self.make_prefix(prefix);
    let cache = self.cache.clone();
    tokio::spawn(async move {
        cache.invalidate_prefix(&prefix_bytes).await;
    });
}

This means invalidation is not guaranteed to complete before subsequent cache reads. A caller that invalidates then immediately reads could get stale data. The old code was synchronous. If this must remain sync (non-async signature), at minimum document this caveat prominently, or consider changing the signature to async fn.

Minor notes

• DeepSizeOf for LanceCache now returns 0, which is a silent behavioral regression for any code that relies on deep_size_of() for memory accounting. Consider at least delegating to approx_size_bytes().
• WeakLanceCache::get_or_insert lost all dedup behavior — concurrent loads for the same key will all run the loader independently. The old code used moka's optionally_get_with which handled this.

[codecov]

Codecov Report

❌ Patch coverage is 82.33487% with 115 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-core/src/cache/mod.rs	87.58%	53 Missing and 5 partials ⚠️
rust/lance-core/src/cache/backend.rs	28.00%	18 Missing ⚠️
rust/lance/src/session.rs	30.43%	16 Missing ⚠️
rust/lance-core/src/cache/moka.rs	76.59%	11 Missing ⚠️
rust/lance/src/dataset/builder.rs	46.66%	8 Missing ⚠️
rust/lance/src/session/index_caches.rs	66.66%	3 Missing ⚠️
rust/lance-index/src/scalar/inverted/index.rs	80.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[westonpace]

Just some nit picking

[westonpace]

All this description feels kind of redundant / obvious. Like, if im a cache and i keep the loader around im gonna start leaking memory or file handles or something.

My first instinct is that its clutter but maybe in this day of agents it is better safe than sorry?

[westonpace]

We should assume here that no two entries share buffers? It's probably the best we can do.

[wjones127]

Yes, we do. There are ways we can detect this in the future. You may be interested in this PR: #6229

[westonpace]

Would it be better for these to return Option with default of None?

[westonpace]

Why is there both InternalCacheKey and CacheKey? Should this trait be documented?

[westonpace]

Document this? Why is it needed? (Yes, I know this is just copied from existing but still)

[westonpace]

"Unique per value type" is strange wording. I think you're trying to say that two type names are equal only if their value types are equal?

Or, to put another way, if two value types are not equal they must not have equal type names?

[wjones127]

Yes. This is meant to be used as a key in a registry of types we'll use later for deserialization of cache entries.

[westonpace]

Didn't you say not to use this?

[wjones127]

Added a comment. We can use this internally, but shouldn't call it directly from other crates.

[westonpace]

What does consistent across crate boundaries mean?