Skip to content

fix(python): preserve index details in python index metadata#6279

Merged
wjones127 merged 6 commits intomainfrom
xuanwo/segment-unified-pr1
Apr 1, 2026
Merged

fix(python): preserve index details in python index metadata#6279
wjones127 merged 6 commits intomainfrom
xuanwo/segment-unified-pr1

Conversation

@Xuanwo
Copy link
Copy Markdown
Collaborator

@Xuanwo Xuanwo commented Mar 24, 2026

This PR fixes Python index metadata round-tripping so Index.index_details is preserved when metadata crosses the Python/Rust boundary.

Without this, segment metadata returned from create_index_uncommitted(...) can lose index_details when passed back through Python, which breaks merge_existing_index_segments(...) and makes committed index metadata less faithful.

@github-actions
Copy link
Copy Markdown
Contributor

PR Review

Clean refactoring that eliminates the uses_segment_commit_path branch and routes all index types through the same execute_uncommitted → segment builder → commit pipeline. The identity plan shortcut for single-segment inputs is a good way to generalize without adding overhead for non-vector indices.

One thing to consider

is_vector_segment relies on type URL suffix matching (create.rs new helper):

fn is_vector_segment(segment: &IndexMetadata) -> bool {
    segment.index_details.as_ref()
        .is_some_and(|details| details.type_url.ends_with("VectorIndexDetails"))
}

This works today but is fragile — if the protobuf type URL format changes or a new vector index type is added with a different message name, this will silently misclassify segments and hit the multi-input rejection error. Consider matching on an enum/variant from IndexType or the protobuf full name constant instead, if one is available. If not, a comment noting the coupling would help future readers.

Looks good otherwise

  • Identity plan logic in plan() and build() is consistent and correct.
  • target_segment_bytes == 0 validation is a nice addition.
  • Test coverage for scalar identity build, multi-input scalar rejection, and scalar commit metadata is thorough.
  • Terminology updates (delta → segment) are consistent throughout.

🤖 Generated with Claude Code

@Xuanwo Xuanwo force-pushed the xuanwo/segment-unified-pr1 branch from 9153063 to 27c06ff Compare March 24, 2026 17:04
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 24, 2026

Codecov Report

❌ Patch coverage is 91.19497% with 14 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance/src/index/create.rs 91.19% 11 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

Xuanwo added 2 commits March 26, 2026 03:20
…d-pr1

# Conflicts:
#	rust/lance-index/src/traits.rs
#	rust/lance-index/src/types.rs
@Xuanwo Xuanwo changed the title refactor: unify segment build flow for index creation fix: preserve index details in python index metadata Apr 1, 2026
@github-actions github-actions bot added the bug Something isn't working label Apr 1, 2026
@wjones127 wjones127 merged commit b08528e into main Apr 1, 2026
16 checks passed
@wjones127 wjones127 deleted the xuanwo/segment-unified-pr1 branch April 1, 2026 15:56
@wjones127 wjones127 changed the title fix: preserve index details in python index metadata fix(python): preserve index details in python index metadata Apr 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants