feat: Implement common subexpression elimination optimizer rule#3444

Closed

andygrove wants to merge 22 commits intoapache:mainfrom

andygrove:native-physical-optimizer

Commits on Feb 8, 2026

Add opt-in DataFusion physical optimizer for native execution

andygrove
and
claude
committed
Pass spark.comet.datafusion.* configs through to DataFusion session

andygrove
and
claude
committed
improve config docs
andygrove
committed
optimize
andygrove
committed
Log physical plan before and after DataFusion optimization

andygrove
and
claude
committed
Run make format

andygrove
and
claude
committed
mark feature as experimental
andygrove
committed
Skip EnsureCooperative optimizer to fix infinite busy loop

andygrove
and
claude
committed
Skip DataFusion optimizer rules incompatible with Comet execution model

andygrove
and
claude
committed
Re-enable metrics when native physical optimizer is active

andygrove
and
claude
committed
Fix clippy clone_on_ref_ptr lint

andygrove
and
claude
committed
Skip FilterPushdown, CoalesceBatches, and ProjectionPushdown optimizer rules

andygrove
and
claude
committed
Configure physical optimizer rules at session creation and add logging

andygrove
and
claude
committed
Add physical common subexpression elimination optimizer rule

andygrove
and
claude
committed
Log before/after plans on optimizer change and isolate CSE rule for testing

andygrove
and
claude
committed
Fix false positive plan-change detection in physical CSE optimizer

andygrove
and
claude
committed

Commits on Feb 10, 2026

Merge remote-tracking branch 'apache/main' into native-physical-optimizer
andygrove
committed