enhancement(clickhouse sink): add support for Arrow complex types #24409
+3,635
−1,855
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for complex types (arrays, maps and tuples) to Clickhouse. Support for the corresponding Arrow types (lists, maps, structs) has been added to facilitate this.
Refactoring of the existing code was required due to the complex, recursive nature of these changes -- specifically edge cases like nested maps not being supported by Arrow's
make_builder().Vector configuration
How did you test this PR?
Unit tests, ran locally against several tables with complex column types
Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
ArrowStreamformat #24373, Add ArrowStream format to Clickhouse sink #24074Notes
@vectordotdev/vectorto reach out to us regarding this PR.pre-pushhook, please see this template.make fmtmake check-clippy(if there are failures it's possible some of them can be fixed withmake clippy-fix)make testgit merge origin masterandgit push.Cargo.lock), pleaserun
make build-licensesto regenerate the license inventory and commit the changes (if any). More details here.