Antalya 26.3: Fix empty partition_key and sorting_key in system.table…

[il9ue]

Closes #1235. Supersedes #1819 — reopened from a branch in the Altinity/ClickHouse repo so CI exposes direct .deb package URLs for clickhouse-regression (fork PRs only produce GitHub Actions artifact zips).

Summary

SELECT partition_key, sorting_key FROM system.tables returned empty strings for Iceberg tables that had no data snapshot. Reliably observable via the Glue catalog (its metadata_location more frequently points at a snapshot-free metadata file), but also reproduced for any empty Iceberg table regardless of catalog (REST, Glue, or direct IcebergS3).

Root cause

IcebergMetadata::partitionKey() and IcebergMetadata::sortingKey() (#959, refined in #1026, ported to 25.8 in #1095) gated their work on the existence of a data snapshot:

auto [actual_data_snapshot, actual_table_state_snapshot] = getRelevantState(context);
if (!actual_data_snapshot)
    return std::nullopt;

This is semantically wrong. Partition spec and sort order are table-level properties recorded at the top level of the Iceberg metadata file (default-spec-id, default-sort-order-id, partition-specs, sort-orders) and exist independently of whether any data snapshot has been written. getState() populates actual_table_state_snapshot (schema_id, metadata_file_path, metadata_version) regardless of snapshot existence; only snapshot_id is std::nullopt, and that field is never read by getPartitionKey() / getSortingKey(). The gate was dead-gating valid data; the fix removes it.

Change list

• src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp — removed the if (!actual_data_snapshot) early return in partitionKey() and sortingKey().
• src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp — getSortingKeyDescriptionFromMetadata() now guards on has(sort-orders) / has(default-sort-order-id). Pre-existing null-deref, previously unreachable behind the snapshot gate; after removing the gate, empty Iceberg V1 tables without sort-orders (optional in V1, required from V2) would hit it. Mirrors the shape in getSortingKeyDisplayStringFromMetadata.

No header changes. No StorageSystemTables.cpp changes — the #1210 (Glue segfault) null/exception guards remain untouched.

Behavior preservation

• REST path output for non-empty tables is byte-identical, including the Follow-up for #959: proper spaces, remove nulls order info #1026 formatting (proper spaces, no NULLS FIRST/NULLS LAST).
• Tables with no partitioning still return empty string (getPartitionKeyStringFromMetadata already guards on missing partition-specs).
• Non-Iceberg data lakes (Hudi, Paimon, Delta Lake) are untouched.
• The Segfault when getting info from Glue catalog #1210 defensive guards in StorageSystemTables.cpp remain in place.

Out of scope

Glue's metadata_location pointer can lag schema-evolution events, which could surface a stale spec. Orthogonal to the snapshot gate; not addressed here.

Test plan

New regression test reproduces the root cause without a catalog mock: creates an Iceberg table with a non-trivial partition spec and sort order, asserts system.tables.partition_key / sorting_key are non-empty before any insert. Existing test_system_tables_partition_sorting_keys continues to pass with byte-identical output.

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixed system.tables.partition_key and system.tables.sorting_key returning empty strings for Iceberg tables that have no data snapshot, including all empty tables and (more frequently) tables accessed via the Glue catalog. Also added a defensive guard against Iceberg V1 metadata files missing sort-orders.

Documentation entry for user-facing changes

Not required — bug fix to existing system.tables columns; no new user-facing surface.

[github-actions]

Workflow [PR], commit [3844ea1]

[il9ue]

Status update on the Iceberg (2) regression failures (Release + Aarch64):

These are the expected behavior change from this fix, not a real regression. The failing assertion lives in clickhouse-regression (iceberg/tests/iceberg_table_engine/system_tables_partition_sorting_keys.py, the Antalya >=26.1 branch), where it asserts partition_key == "" / sorting_key == "" for an empty Iceberg table — i.e. the #1235 bug this PR fixes. With the snapshot gate removed, system.tables now correctly returns partition_key = name / sorting_key = name asc, so the old assertion fails by design.

I've flagged this to QA to confirm ownership of the regression-side update. Will link the test PR here once it's open. (cc. @alsugiliazova )

The other failures (clickhouse_keeper_failover, tiered_storage_local/minio, s3_gcs_2, disk_level_encryption) are unrelated infra/flake — each is "module errored" with zero failed scenarios, none touch the Iceberg path, and keeper_failover passes on Release.

[Selfeer]

AI audit note: This review comment was generated by AI (gpt-5.3-codex).

Audit update for PR #1874:

Confirmed defects:

No confirmed defects in reviewed scope.

Coverage summary:

• Scope reviewed: PR diff in src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp and src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp; audited system.tables entrypoints to partitionKey/sortingKey, metadata parsing/formatting paths, and exception/fallback behavior.
• Categories failed: None in reviewed scope.
• Categories passed: Call-graph and transition coverage for changed branches; fail-open/fail-closed checks for missing snapshot and missing sort-order metadata; exception propagation and fallback behavior in StorageSystemTables; C++ lifetime/iterator/race/deadlock/overflow/UB classes (no new unsafe mutation or shared-state pattern introduced by this diff).
• Assumptions/limits: Static audit only (no runtime execution, no CI artifact replay); conclusions are limited to PR-introduced behavior and directly reachable adjacent paths.

[Selfeer]

PR #1874 CI Triage

PR: Antalya 26.3: Fix empty partition_key and sorting_key in system.tables for Iceberg tables without data snapshots
Branch: fix/antalya/1235-glue-partition-sorting-keys → antalya-26.3
SHA: 3844ea1b990779f47ee034723c7de6d98aac14ec
Run: 27000320264

Changed files:

• src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp — 4 deletions (removes if (!actual_data_snapshot) guard)
• src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp — 5 additions (adds defensive has() guard for Iceberg V1 metadata)

---

Summary

Category	Count	Jobs
PR-caused regression	1	iceberg_2
Infrastructure issue (runner hang / teardown timeout)	4	tiered_storage_local, tiered_storage_minio, s3_gcs_2, disk_level_encryption
Infrastructure issue (runner setup timeout)	1	clickhouse_keeper_failover (aarch64)
Pre-existing XFail only	1	clickhouse_keeper_failover (x86_64) — passed overall

Verdict: 1 failure is caused by this PR. 5 are infrastructure issues unrelated to the code change.

---

Failure Detail

1. Regression release iceberg_2 — PR-caused regression ✗

Report: https://altinity-build-artifacts.s3.amazonaws.com/REFs/1874/merge/3844ea1b990779f47ee034723c7de6d98aac14ec/regression/x86_64/with_analyzer/zookeeper/without_thread_fuzzer/iceberg2/report.html

Failing test:

/iceberg/iceberg table engine/system tables partition sorting keys

File: iceberg/tests/iceberg_table_engine/system_tables_partition_sorting_keys.py, line 56

Root cause: The regression test system_tables_partition_sorting_keys was written to document the old buggy behavior where partition_key returned "" for an empty Iceberg table (no data snapshot). The test contains a conditional guard:

if check_if_antalya_build(self) and check_clickhouse_version(">=26.1")(self):
    assert partition_key == "", error()   # line 56 — FAILS
    assert sorting_key == "", error()

After the PR fix, partition_key now correctly returns 'name' even before any data is inserted (since partition spec is a table-level property, not snapshot-level). The test assertion expects "" but receives 'name'.

Assertion failure:

assert partition_key == "", error()
       ^ is 'name'

The same stale assertion also appears at line 74:

assert partition_key == "", error()  # after "read from empty table" step

Action required: The test needs updating at two points:

Line	Current assertion	Correct assertion after fix
56	assert partition_key == ""	assert partition_key == "name"
57	assert sorting_key == ""	assert sorting_key == "name asc"
74	assert partition_key == ""	assert partition_key == "name"

The step description at line 47 ("check partition_key and sorting_key are not set in system.tables before first insert") should also be updated to reflect the correct new expectation.

Note: The other failures in iceberg_2 (predicate push-down S3 request counts, partition pruning counts) are all classified as Known Fails (XFail) in the report and are pre-existing issues unrelated to this PR.

---

2. Regression aarch64 tiered_storage_minio — Infrastructure issue (runner teardown hang)

Report: https://altinity-build-artifacts.s3.amazonaws.com/REFs/1874/merge/3844ea1b990779f47ee034723c7de6d98aac14ec/regression/aarch64/with_analyzer/zookeeper/without_thread_fuzzer/tiered_storage/minio/report.html

Status summary: Module errored, but the inner feature tiered storage/with minio completed OK (1h 33m, 95.6% pass).

Error:

ExpectTimeoutError: Timeout 1000.000s for '[#\$] '
  at: bash(f"cd {self.docker_compose_project_dir}", timeout=1000)

The bash shell became completely unresponsive during cluster teardown — a Docker/resource hang on the aarch64 runner. No test assertions were affected; all 90 scenarios passed, and the 4 XFail scenarios (attach or replace partition different policies, double move while select, manual move with downtime) are pre-existing known failures with tracked Jira issues.

Caused by PR: No

---

3. Regression release tiered_storage_local — Infrastructure issue (runner teardown hang)

Report: https://altinity-build-artifacts.s3.amazonaws.com/REFs/1874/merge/3844ea1b990779f47ee034723c7de6d98aac14ec/regression/x86_64/with_analyzer/zookeeper/without_thread_fuzzer/tiered_storage/local/report.html

Status summary: Module errored, but the inner feature tiered storage/normal completed OK (1h 37m, 96% pass).

Error: Same ExpectTimeoutError: Timeout 1000.000s during bash teardown as above, on the x86_64 runner this time.

All 90 scenarios passed; the 4 XFail scenarios are the same pre-existing known failures as in the minio variant.

Caused by PR: No

---

4. Regression aarch64 s3_gcs_2 — Infrastructure issue (runner teardown hang)

Report: https://altinity-build-artifacts.s3.amazonaws.com/REFs/1874/merge/3844ea1b990779f47ee034723c7de6d98aac14ec/regression/aarch64/with_analyzer/zookeeper/without_thread_fuzzer/s32/gcs/report.html

Error: ExpectTimeoutError: Timeout 1000.000s for '[#\$] ' during Cluster.__exit__ teardown — same pattern as tiered_storage failures above. The bash shell hung on the aarch64 runner.

All 39 OK scenarios passed. The 31 XFail scenarios are pre-existing known failures. No actual test assertions failed.

Caused by PR: No

---

5. Regression aarch64 clickhouse_keeper_failover — Infrastructure issue (runner setup timeout)

Report: https://altinity-build-artifacts.s3.amazonaws.com/REFs/1874/merge/3844ea1b990779f47ee034723c7de6d98aac14ec/regression/aarch64/with_analyzer/zookeeper/without_thread_fuzzer/clickhouse_keeper_failover/report.html

Error: ExpectTimeoutError: Timeout 10.000s while copying the binary file at cluster setup (bash(f"cp {self.binary_path} {new_path}")). The aarch64 runner was so resource-starved that even a cp command timed out.

The one test scenario (dynamic failover) that was attempted is marked XFail (known issue). The x86_64 version of the same job ran successfully (all 3 scenarios XFail as expected, module OK).

Caused by PR: No

---

6. RegressionTestsRelease / Common (disk_level_encryption) — Infrastructure issue (likely)

Job: https://github.com/Altinity/ClickHouse/actions/runs/27000320264/job/79688545408
Duration: 30m 4s, failed

No S3 report was uploaded for this job (404 on expected report path; no disk_level_encryption/ directory in the artifact listing). This indicates the test cluster setup failed before any tests ran, or results were never uploaded. The job completed at 2026-06-05T08:37:42Z.

Disk-level encryption tests are completely orthogonal to the PR's Iceberg changes (IcebergMetadata.cpp / Utils.cpp). There is no plausible code path connecting them.

Caused by PR: No

---

Infrastructure Pattern

The aarch64 runner (and to a lesser extent the x86_64 runner) exhibited widespread Docker hang symptoms during this CI run — 3 of the 4 aarch64 jobs errored with Timeout 1000.000s for '[#\$] ' in bash teardown, and one errored with a 10s cp timeout at setup. This is consistent with runner resource exhaustion (OOM, I/O saturation, or Docker daemon instability) on that particular run date (Jun 05, 2026), not with any code introduced by this PR.

---

Recommendations

1. Fix the regression test in iceberg/tests/iceberg_table_engine/system_tables_partition_sorting_keys.py:

   • Lines 55–57: change assertions from == "" to the new expected values ("name" / "name asc") and update the step description.
   • Line 72–74: change assert partition_key == "" to assert partition_key == "name".
   • The test was documenting the old bug; it should now verify the corrected behavior.

2. Re-run infrastructure failures — tiered_storage_minio, tiered_storage_local, s3_gcs_2, clickhouse_keeper_failover, disk_level_encryption will all pass on a healthy runner.

3. No code changes needed to the production fix itself — only the test expectations need updating.

[DimensionWieldr]

Updated the regression test iceberg/tests/iceberg_table_engine/system_tables_partition_sorting_keys.py to reflect this PR's bugfixes.

Currently rerunning failed jobs...