Consolidate filter classification into physical planner #20092
+782
−920
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR moves ALL filter expressions to `TableScan.filters` during logical optimization, deferring classification (Exact/Inexact/Unsupported) to the physical planner. ## Motivation Currently, the optimizer calls `supports_filters_pushdown()` to classify filters during logical optimization, resulting in: - Exact/Inexact filters going to `TableScan.filters` - Unsupported/Inexact/Volatile filters staying as `Filter` nodes above the scan This creates a split representation where filters can exist in two places, leading to complexity in DML operations and potential for filter duplication. ## Changes **Target behavior:** - Optimizer moves ALL filters (except scalar subqueries) to `TableScan.filters` - Physical planner calls `supports_filters_pushdown()` and creates `FilterExec` for post-scan filters (Unsupported/Inexact/Volatile) **Key implementation details:** - Simplified `push_down_filter.rs` TableScan case to push all filters - Enhanced `physical_planner.rs` to classify filters and create FilterExec - Added projection expansion for filter columns not in user's projection - Updated `extract_dml_filters` to also extract from `TableScan.filters` Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related to:
TableScan.filters#19894 - UnifiedTableScan.filtersrepresentationUPDATE ...FROMbug #19950 -UPDATE ...FROMbug (filter extraction improvements)Rationale
Currently, the optimizer calls
supports_filters_pushdown()to classify filters during logical optimization. This results in a split representation where:TableScan.filtersFilternodes above the scanThis creates several problems (as described in #19894):
What changes are included in this PR?
This PR moves ALL filter expressions to
TableScan.filtersduring logical optimization, deferring classification (Exact/Inexact/Unsupported) to the physical planner.Changes to
push_down_filter.rs:TableScan.filtersChanges to
physical_planner.rs:supports_filters_pushdown()FilterExecfor Unsupported/Inexact/Volatile filterscompute_scan_projection_with_filters()helpercreate_filter_exec()helper with async UDF supportextract_dml_filters()to also extract fromTableScan.filtersBehavior Changes:
TableScan.filtersinstead of as separateFilternodesFilterExecnodes for Unsupported/Inexact/Volatile filtersProjectionExecto trim extra columnsAre these changes tested?
Yes - updated existing tests to match new plan representations:
Are there any user-facing changes?
Plan output changes: Users will see filters in
TableScanwithpartial_filters=orunsupported_filters=annotations in logical plans, rather than separateFilter:nodes. Physical plans remain functionally equivalent withFilterExecnodes where needed.🤖 Generated with Claude Code