Skip to content

Conversation

@benjamin-awd
Copy link
Contributor

Summary

This PR adds support for complex types (arrays, maps and tuples) to Clickhouse. Support for the corresponding Arrow types (lists, maps, structs) has been added to facilitate this.

Refactoring of the existing code was required due to the complex, recursive nature of these changes -- specifically edge cases like nested maps not being supported by Arrow's make_builder().

Vector configuration

sinks:
  clickhouse:
    type: clickhouse
    inputs: [...]
    endpoint: ...
    database: ...
    table: ...
    format: arrow_stream
    batch_encoding:
      codec: arrow_stream
      allow_nullable_fields: true

How did you test this PR?

Unit tests, ran locally against several tables with complex column types

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • make fmt
      • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
      • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

@benjamin-awd benjamin-awd requested review from a team as code owners December 24, 2025 09:01
@github-actions github-actions bot added domain: sinks Anything related to the Vector's sinks domain: external docs Anything related to Vector's external, public documentation labels Dec 24, 2025
Copy link
Contributor

@evazorro evazorro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from a Docs perspective! (We review .cue files.) Thanks for contributing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants