#18939 added a new template that can be used with Compaction supervisors. It is experimental in nature, and will be experimental in Druid 37, but I'd like to scope out work needed to get it as close to production ready as possible for Druid 37 so that we can pull the experimental tag in Druid 38
Todo
Drop Native Compaction Support.
This was recently completed in #19078. Rationale here is that MSQ engine is the desired future of compaction supervisors. Supporting two engines is extra overhead, can lead to confusion, and slows development of more powerful features.
Refactor Rules Concept
This warrants a sub issue perhaps. But with #19061 adding CLUSTERED BY support on expressions, it has exposed some gaps in the initial rule structure that is coupled closely with compaction state. The original proposal for #18939 was even more coupled than the version that got merged. In the merged version we have ReindexingDataSchemaRule which is more general and swallows up compaction state concepts for dims, metrics, query gran, etc.
We are thinking we need to go further with rule refactoring. One thought is to create a rule that encompasses partitioning ideas like segment gran, partitions spec, etc. The tuning config rule stuff that is pretty static in nearly all cases would be moved outside of the rules and into the top level config for the template. This would be shared by all generated tasks.
Documentation
As the design has been in flux, we have not committed to any druid docs yet. Once the above two sections are finalized a new set of docs about this template + integration with the existing automatic compaction docs is needed. This will be an experimental feature in d37. The docs should make it easy to see the value of this template + guide an operator to onboard to it and evaluate it
UI in the router
#19051 is open for adding visualization of the timeline of search intervals and compaction configs generated by a ruleset for an existing supervisor
I also think it would be cool to add a supervisor builder UI where you can visualize the timeline as you create/edit a supervisor spec instead of only when it already exists.
More robust rule provider
Right now the only rule provider is inline which works and is nice for many smaller use cases, but it becomes hard to reason about quickly. A rule provider in core that sources rules from something like druid-catalog feels like a more production ready long term approach
#18939 added a new template that can be used with Compaction supervisors. It is experimental in nature, and will be experimental in Druid 37, but I'd like to scope out work needed to get it as close to production ready as possible for Druid 37 so that we can pull the experimental tag in Druid 38
Todo
Drop Native Compaction Support.
This was recently completed in #19078. Rationale here is that MSQ engine is the desired future of compaction supervisors. Supporting two engines is extra overhead, can lead to confusion, and slows development of more powerful features.
Refactor Rules Concept
This warrants a sub issue perhaps. But with #19061 adding CLUSTERED BY support on expressions, it has exposed some gaps in the initial rule structure that is coupled closely with compaction state. The original proposal for #18939 was even more coupled than the version that got merged. In the merged version we have
ReindexingDataSchemaRulewhich is more general and swallows up compaction state concepts for dims, metrics, query gran, etc.We are thinking we need to go further with rule refactoring. One thought is to create a rule that encompasses partitioning ideas like segment gran, partitions spec, etc. The tuning config rule stuff that is pretty static in nearly all cases would be moved outside of the rules and into the top level config for the template. This would be shared by all generated tasks.
Documentation
As the design has been in flux, we have not committed to any druid docs yet. Once the above two sections are finalized a new set of docs about this template + integration with the existing automatic compaction docs is needed. This will be an experimental feature in d37. The docs should make it easy to see the value of this template + guide an operator to onboard to it and evaluate it
UI in the router
#19051 is open for adding visualization of the timeline of search intervals and compaction configs generated by a ruleset for an existing supervisor
I also think it would be cool to add a supervisor builder UI where you can visualize the timeline as you create/edit a supervisor spec instead of only when it already exists.
More robust rule provider
Right now the only rule provider is
inlinewhich works and is nice for many smaller use cases, but it becomes hard to reason about quickly. A rule provider in core that sources rules from something like druid-catalog feels like a more production ready long term approach