sql: add well-known protobuf types for CSR schema compilation #34635
+1,042
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When creating a Protobuf source using CONFLUENT SCHEMA REGISTRY CONNECTION, Materialize previously failed to compile schemas that import well-known types like google/protobuf/timestamp.proto. This happened because the compile_proto function only added files fetched from the schema registry to the VirtualSourceTree, but well-known types are typically not registered in the schema registry (they are implicitly available to protoc). This commit adds a new protobuf module that embeds the standard Google protobuf well-known types: - any.proto - api.proto - duration.proto - empty.proto - field_mask.proto - source_context.proto - struct.proto - timestamp.proto - type.proto - wrappers.proto These types are now added to the VirtualSourceTree before compiling schemas fetched from the schema registry, allowing schemas that import well-known types to compile successfully. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add a test that verifies Materialize can compile Protobuf schemas that import well-known types (google/protobuf/timestamp.proto, google/protobuf/duration.proto) even when those types are NOT registered in the Confluent Schema Registry. This tests the fix in commit ca7584c where we embed well-known types in the VirtualSourceTree before compiling CSR schemas. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Reorder mod declarations alphabetically in pure.rs - Split long method chain in protobuf.rs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Wrap URL in angle brackets to satisfy rustdoc::bare-urls lint. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
5 tasks
Member
Author
|
Claude did not think to fmt and doc check. I will PR a docs change that it claims will help with that. |
Member
|
Thanks for this! I think this is directionally right but I propose just fixing at the source, we already control our protobuf wrapper, easy enough to fix at the source: MaterializeInc/rust-protobuf-native#29 |
Member
Author
|
Closing in favor of MaterializeInc/rust-protobuf-native#29. Discussion here: https://materializeinc.slack.com/archives/C08A7H11KP0/p1767643709343349?thread_ts=1767197380.132829&cid=C08A7H11KP0
|
Member
Author
|
Actually, we will salvage the extra test from here once MaterializeInc/rust-protobuf-native#29 is merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context for my making this PR is here. (And this replaces #34630.)
When creating a Protobuf source using CONFLUENT SCHEMA REGISTRY
CONNECTION, Materialize fails to compile schemas that import
well-known types like google/protobuf/timestamp.proto. This happens
because the compile_proto function only adds files fetched from the
schema registry to the VirtualSourceTree, but well-known types are
typically not registered in the schema registry (they are implicitly
available to protoc).
This change adds a new protobuf module that embeds the standard
Google protobuf well-known types:
any.proto
api.proto
duration.proto
empty.proto
field_mask.proto
source_context.proto
struct.proto
timestamp.proto
type.proto
wrappers.proto
These types are now added to the VirtualSourceTree before compiling
schemas fetched from the schema registry, allowing schemas that import
well-known types to compile successfully.
Motivation
This PR fixes a recognized bug.
https://github.com/MaterializeInc/database-issues/issues/10003
Checklist
This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (https://github.com/MaterializeInc/cloud/pull/5021).
If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.