-
Notifications
You must be signed in to change notification settings - Fork 17
Description
Summary
During export partition, when a task is cancelled (e.g. via SYSTEM STOP MOVES), the cancellation handler performs a best-effort zk->tryRemove to release the part lock but does not check the result or handle failure. If ZooKeeper tryRemove fails transiently (network blip, version mismatch race), the lock remains in ZooKeeper and subsequent scheduler cycles skip the part as "locked", leaving the export stuck until ZooKeeper session expiry or manual intervention.
Severity
Medium
Affected code
src/Storages/MergeTree/ExportPartitionTaskScheduler.cpp — ExportPartitionTaskScheduler::handlePartExportFailure
if (exception->code() == ErrorCodes::QUERY_WAS_CANCELLED)
{
zk->tryRemove(export_path / "locks" / part_name, locked_by_stat.version);
LOG_INFO(storage.log, "ExportPartition scheduler task: Part {} export was cancelled, skipping error handling", part_name);
return;
}Affected subsystem
Replicated MergeTree export-partition scheduler state in ZooKeeper (exports/<key>/locks/<part>). Impacts recovery/progress of pending partition exports on the affected replica.
Steps to reproduce
- Start
EXPORT PARTITIONon a replicated MergeTree table - Issue
SYSTEM STOP MOVESto trigger cancellation - Inject a transient ZooKeeper failure (or version mismatch race) during the
tryRemovecall forexports/<key>/locks/<part> - Issue
SYSTEM START MOVES - Observe: the part remains skipped by the scheduler due to the stale lock; export does not resume for that part
Expected behavior
The lock should be reliably released on cancellation. If tryRemove fails, the failure should be detected and remediated (retry, backoff, or fallback cleanup) so that the part becomes schedulable again after SYSTEM START MOVES.
Actual behavior
tryRemove result is ignored. If the remove fails, the stale lock persists in ZooKeeper and blocks rescheduling of the affected part until ZooKeeper session expiry or manual cleanup.
References
- PR: Export Partition - release the part lock when the query is cancelled #1593
- Audit comment: Export Partition - release the part lock when the query is cancelled #1593 (comment)
Origin
Identified via static audit of PR #1593 ("Export Partition - release the part lock when the query is cancelled"). The defect is in the new cancellation branch introduced by the PR.