Antalya 26.1 Backport of #95476, #98360, #100361 - enable prewhere for iceberg (and fixes)

[mkmkme]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Enable PREWHERE optimization for Iceberg tables. (ClickHouse#95476 by @scanhex12)
Fix LOGICAL_ERROR exception in the Parquet V3 reader when PREWHERE references a column not present in the Parquet file (ClickHouse#98360 by @alexey-milovidov)
Fix exception "Logical error: prewhere_info" when querying URL/S3 Parquet tables with a row-level security policy and no PREWHERE clause (ClickHouse#100361 by @alexey-milovidov)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [9fbbcaf]

[mkmkme]

AI audit note: This review comment was generated by AI (claude-4.6-sonnet-medium-thinking).

Audit update for PR #1581 (Antalya 26.1 backport of ClickHouse#95476, ClickHouse#98360 – enable PREWHERE for Iceberg):

Confirmed defects:

Medium: PREWHERE crashes on Iceberg tables that have undergone schema evolution (column renames)

• Impact: Any SELECT with a filterable WHERE condition (auto-promoted to PREWHERE by the planner) on an Iceberg table whose Parquet files predate a column rename will throw a runtime exception at read time, making those files unreadable once prewhere is active.
• Anchor: src/Storages/ObjectStorage/StorageObjectStorageSource.cpp / createReader (schema_changed + mapper path, line ~856); src/Storages/ObjectStorage/StorageObjectStorage.cpp TODO block lines ~222-241
• Trigger: Iceberg table has at least one column rename in its schema history; pre-rename Parquet files are still present; any query with a filterable WHERE/PREWHERE on the renamed column is executed against a table-engine (not table function).
• Why defect: When getInitialSchemaByPath() returns a non-null schema (underlying_format_read_schema_id != current schema_id), initial_header is built with OLD ClickHouse column names (e.g. old_b). The prewhere_info passed alongside it was constructed against the CURRENT ClickHouse schema (b). In Reader::preparePrewhere(), extended_sample_block.getPositionByName("b") is called on a block that only contains old_b, throwing "Not found column".

Code evidence — StorageObjectStorage.cpp, the TODO predating this PR:

/// TODO: Known problems with datalake prewhere:
///  * If the iceberg table went through schema evolution, columns read from file
///    may need to be renamed or typecast before applying prewhere. [...]
///    We should probably change this and delete that additional code?

Code evidence — StorageObjectStorageSource.cpp, the schema-changed path:

return std::make_shared<FormatFilterInfo>(
    format_filter_info->filter_actions_dag,
    format_filter_info->context.lock(),
    mapper,
    format_filter_info->row_level_filter,
    format_filter_info->prewhere_info  // ← references CURRENT column names,
);                                     //   but initial_header has OLD names

• Upstream status: This defect is NOT introduced by this Altinity PR. It originates from upstream PR enable prewhere for iceberg ClickHouse/ClickHouse#95476 (scanhex12/enable_prewhere_for_iceberg, merged Jan 2026) and is present verbatim in the current upstream master. History:
  • Upstream commit bba27ea6bfa explicitly disabled datalake prewhere and authored the TODO block documenting the schema-evolution risk.
  • Upstream PR enable prewhere for iceberg ClickHouse/ClickHouse#95476 re-enabled prewhere for Iceberg via supportsPrewhere() without resolving the TODOs. This Altinity PR is a faithful backport of that change.
  • Recommended action: open a follow-up issue against upstream enable prewhere for iceberg ClickHouse/ClickHouse#95476 referencing the unresolved schema-evolution TODO.
• Fix direction (short): When schema_changed == true and a mapper is present, either (a) strip prewhere_info from the new FormatFilterInfo (safe fallback to WHERE for evolved files), or (b) remap the prewhere DAG column names through the mapper before handing them to the Parquet reader.
• Regression test direction (short): Iceberg table, write Parquet files, rename a column, write more files, SELECT with WHERE on the renamed column — assert no exception and correct results from both old and new files.

---

Coverage summary:

• Scope reviewed: 5 changed source files + 2 new test files; Parquet reader INPUT-node fix, Iceberg supportsPrewhere() dispatch chain (DataLakeConfiguration template + StorageIcebergConfiguration delegation), FormatFilterInfo schema-change path in createReader, StorageObjectStorage construction-time guard, integration and stateless tests.
• Categories failed: schema-evolution × PREWHERE interaction (confirmed, upstream origin)
• Categories passed: INPUT-node external_columns fix correctness; compile-time USE_AVRO flag semantics; base-class supportsPrewhere() default regression check; FormatFilterInfo shared-ptr lifetime and thread safety; error-message improvement in preparePrewhere; stateless test 04001 logic; StorageIcebergConfiguration::getImpl() init ordering; Paimon/DeltaLake/Hudi remain correctly excluded from prewhere.
• Assumptions/limits: Schema-evolution path analysis is static; no Iceberg tables with column-rename history were runtime-tested; upstream tracking issue does not yet exist.

[ianton-ru]

LGTM

[CarlosFelipeOR]

QA Verification: Issue found

The CI run for this PR shows significant test failures caused by enabling PREWHERE for Iceberg tables:

• 82 regression test failures across row_policy, prewhere clause, and iceberg settings tests — row policies are silently ignored on Iceberg tables, returning all rows instead of only the filtered ones (security concern: row-level access control bypassed)
• 122 integration test failures across 3 test suites (test_read_constant_columns_optimization, test_partition_pruning_with_subquery_set, test_writes_statistics_by_minmax_pruning) — query results differ when Iceberg optimizations are toggled on vs off

All failures are deterministic and reproduce across all architectures and job configurations (amd_asan, amd_binary, amd_tsan, arm_binary).

The same test (combination #30) was run against upstream v26.3.2.3-lts (which contains the original upstream PRs 95476 and 98360) and passed, indicating the issue is specific to the interaction with the antalya-26.1 codebase.

Details: #1595

Adding verified-with-issue label.