Skip to content

compute/correction_v2: make chunk capacity configurable#36046

Merged
teskje merged 3 commits intoMaterializeInc:mainfrom
teskje:chunk-size-dyncfg
Apr 15, 2026
Merged

compute/correction_v2: make chunk capacity configurable#36046
teskje merged 3 commits intoMaterializeInc:mainfrom
teskje:chunk-size-dyncfg

Conversation

@teskje
Copy link
Copy Markdown
Contributor

@teskje teskje commented Apr 13, 2026

Previously, CorrectionV2 used TimelyStack's default capacity (8KiB) for the Chunk capacity. We suspect that this size is smaller than ideal and would like a way to experiment with different values in production.

This commit makes the chunk capacity configurable through a dyncfg. Like the chain proportionality, changes to this configuration only take effect when a CorrectionV2 instance is created.


First commit is pure code movement, to simplify the review of the logic changes in the second commit.

@github-actions
Copy link
Copy Markdown
Contributor

Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone.

PR title guidelines

  • Use imperative mood: "Fix X" not "Fixed X" or "Fixes X"
  • Be specific: "Fix panic in catalog sync when controller restarts" not "Fix bug" or "Update catalog code"
  • Prefix with area if helpful: compute: , storage: , adapter: , sql:

Pre-merge checklist

  • The PR title is descriptive and will make sense in the git log.
  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

@teskje teskje marked this pull request as ready for review April 13, 2026 16:12
@teskje teskje requested review from a team as code owners April 13, 2026 16:12
@teskje teskje force-pushed the chunk-size-dyncfg branch from eda7427 to a066b91 Compare April 14, 2026 09:45
teskje added 2 commits April 15, 2026 10:21
This moves the `merge_*` functions into the `CorrectionV2` impl block,
in preparation of making them methods. This is pure code movement, to
simplify code review. The code is left in a broken state and is fixed in
the next commit.
Previously, `CorrectionV2` used `TimelyStack`'s default capacity (8KiB)
for the `Chunk` capacity. We suspect that this size is smaller than
ideal and would like a way to experiment with different values in
production.

This commit makes the chunk capacity configurable through a dyncfg. Like
the chain proportionality, changes to this configuration only take
effect when a `CorrectionV2` instance is created.
@teskje teskje force-pushed the chunk-size-dyncfg branch from a066b91 to c31814d Compare April 15, 2026 08:21
@teskje teskje requested a review from antiguru April 15, 2026 08:23
Copy link
Copy Markdown
Member

@antiguru antiguru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but I'd like to see the use of SizableContainer to go away.

Comment thread src/compute/src/sink/correction_v2.rs Outdated
@teskje teskje requested a review from antiguru April 15, 2026 11:15
Copy link
Copy Markdown
Member

@antiguru antiguru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@teskje
Copy link
Copy Markdown
Contributor Author

teskje commented Apr 15, 2026

TFTR!

@teskje teskje merged commit 2491930 into MaterializeInc:main Apr 15, 2026
123 checks passed
@teskje teskje deleted the chunk-size-dyncfg branch April 15, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants