feat: introduce CacheCodec for serializing index cache entries

[wjones127]

Introduce a codec: CacheCodec parameter to CacheBackend methods, providing the ability to serialize and deserialize entries, if desired. This struct is just a pair of function pointers: serialize and deserialize. So it's very cheap if not used.

To provide this for a type, implement the CacheKey::codec() method to return a CacheCodec. By default, this trait method returns None. We only implement serialization for some types right now, but the eventual goal is to implement it for all types. To keep this PR focused, I only implement this for vector indexes, but the same approach could be used for FTS and other indexes.

To make it easy to implement, we provide a CacheCodecImpl trait:

pub trait CacheCodecImpl: Send + Sync {
    fn serialize(&self, writer: &mut dyn std::io::Write) -> Result<()>;

    fn deserialize(reader: &mut dyn std::io::Read) -> Result<Self>
    where
        Self: Sized;
}

The implementors of CacheKey can then write:

    fn codec() -> Option<CacheCodec> {
        Some(CacheCodec::from_impl::<Self::ValueType>())
    }

The implementation of serialization / deserialization is designed to have very low overhead. We use serde-derived strings for small metadata, and use Arrow IPC for Arrow data. When deserializing, we use zero-copy deserialization to re-use the buffers read in.

One important change is we no longer cache Arc<dyn VectorIndex> for V2+ vector indices. This is because IVFIndex (which is the V3 implementation of VectorIndex) contains non-serializable data structures like FileReader. Instead we cache IvfIndexState, which is a serializable representation of the vector index. Importantly, this does not require any IO to reconstruct, which is validated by the pre-existing test test_prewarm_ivf_pq.

[github-actions]

PR Review: feat: add serialize/deserialize for IVF PQ partition cache entries

Clean implementation overall — the zero-copy IPC approach is well-suited for cache serde. A few issues to flag:

P1: Integer overflow in deserialization offset arithmetic

In deserialize(), the section boundary calculations use unchecked addition:

let sub_index_end = sub_index_start + header.sub_index_len as usize;
let codebook_end = codebook_start + header.codebook_len as usize;
let storage_end = storage_start + header.storage_len as usize;

If the header contains corrupted or adversarial values (e.g., lengths close to u64::MAX), these additions can wrap around, causing storage_end to be a small value that passes the data.len() < storage_end check. The subsequent Buffer::slice_with_length calls would then read incorrect regions or panic.

Use checked_add and return an error on overflow, same as the defensive pattern already used for trailer_start / footer_start in read_ipc_zero_copy.

P1: Missing Hamming in distance type roundtrip test

test_roundtrip_preserves_distance_type covers L2, Cosine, and Dot but omits Hamming, which is handled in the distance_type_to_u8/u8_to_distance_type mapping. Either add it to the test or document why it's excluded (e.g., PQ storage doesn't support Hamming).

Minor

• The module is pub mod partition_serde — if this is only intended for internal caching, consider pub(crate) to avoid leaking it as public API.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 80.81049% with 322 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/vector/ivf/v2.rs	77.52%	92 Missing and 26 partials ⚠️
rust/lance-arrow/src/ipc.rs	61.41%	72 Missing and 21 partials ⚠️
rust/lance/src/index/vector/ivf/partition_serde.rs	86.64%	50 Missing and 42 partials ⚠️
rust/lance-index/src/vector/pq/storage.rs	37.50%	3 Missing and 2 partials ⚠️
rust/lance-core/src/cache/codec.rs	89.18%	3 Missing and 1 partial ⚠️
rust/lance-index/src/vector/quantizer.rs	0.00%	2 Missing and 1 partial ⚠️
rust/lance-table/src/format/index.rs	96.20%	0 Missing and 3 partials ⚠️
rust/lance-core/src/cache/mod.rs	90.90%	0 Missing and 2 partials ⚠️
rust/lance/src/index/vector/ivf.rs	87.50%	0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[wjones127]

My big worry with this is that the DeepSizeOf will no longer be accurate after a roundtrip.

[wjones127]

This concern will be addressed by #6229. As long as we aren't sharing buffers across cache entries, we should be fine.

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[westonpace]

I think this PR might be in need of a rebase or something?

[wjones127]

@westonpace Okay, I've finally finished rebasing and refactoring. I think it's good to review again.

[westonpace]

Some questions but overall I think this is some great work. I guess I didn't appreciate how much complexity there would be in serializing all the different kinds of cache keys.

[westonpace]

Suggested change

	/// Backend authors receive these ready-made from [`LanceCache`](super::LanceCache)
	/// CacheBackend impls receive these ready-made from [`LanceCache`](super::LanceCache)

[westonpace]

Huh...why not Box<dyn CacheCodecImpl>? Is it because you're expecting to need to violate the orphan rule?

[wjones127]

Box<dyn CacheCodecImpl> only works for serialize(). deserialize() returns Result<Self>, so it can't be called from &dyn CacheCodecImpl. This is why there is a where Self: Sized bound on the trait method.

In CacheCodec::deserialize, it returns Result<ArcAny>, which works without having to know the concrete type.

I went through a lot of iterations on these traits and structs. This is one place Rust gets really complicated 🫣

[westonpace]

What happens in a disk cache if this returns None? Do these caches fail? Can we document that fact?

[wjones127]

Up to the cache backend. If it's a disk-only cache, then it would not cache. If it's a memory and disk cache, then it can store in memory but when evicted it won't go to disk.

[westonpace]

Why is this in lance-core?

[westonpace]

I guess there is no need to worry about stability here? Do we plan for disk caches to be completely lost on upgrades? Or are we actually thinking we might want to keep these stable between versions?

[wjones127]

For now, I'm marking them as unstable and experimental. I think we can see about making them stable in the future. I'd probably put some magic bytes and version in the header when we do.

Do we plan for disk caches to be completely lost on upgrades?

Given we have a pre-warming mechanism, I feel okay with letting disk caches be ephemeral to start.

[westonpace]

Would it make more sense to have these structs in their respective modules? E.g. rq, pq, etc.?

[westonpace]

This is nice

[westonpace]

Should this comment be on SerializingBackend?