Skip to content

Remove continual tasks (CT) feature#35967

Open
antiguru wants to merge 1 commit intomainfrom
claude/remove-continual-tasks-akICD
Open

Remove continual tasks (CT) feature#35967
antiguru wants to merge 1 commit intomainfrom
claude/remove-continual-tasks-akICD

Conversation

@antiguru
Copy link
Copy Markdown
Member

@antiguru antiguru commented Apr 11, 2026

Summary

  • Remove the entire continual tasks (CT) feature from the codebase
  • Add catalog migration v81→v82 that deletes any existing CT items and comments
  • Preserve audit log ObjectType::ContinualTask for backward compatibility with persisted entries

Details

Catalog migration (v81→v82):

  • Iterates over all persisted items and deletes any whose create_sql starts with CREATE CONTINUAL TASK
  • Deletes any comments on continual task objects
  • Builtin CT removal is handled automatically by the builtin item migration (they're no longer in BUILTINS_STATIC)

What was removed (~4,600 lines deleted across 99 files):

  • SQL parser: CREATE/DROP/SHOW/ALTER/COMMENT ON CONTINUAL TASK syntax, ContinualTaskStmt, CreateContinualTaskSugar (TRANSFORM/RETAIN)
  • SQL planning: CreateContinualTaskPlan, feature flags (enable_continual_task_create/transform/retain), max_continual_tasks session var, CatalogItemType::ContinualTask, ObjectType::ContinualTask, CommentObjectId::ContinualTask
  • Adapter: sequence_create_continual_task, CT bootstrap/dataflow logic, CT quota checks, ExecuteResponse::CreatedContinualTask
  • Compute: ContinualTaskCtx, ContinualTaskConnection, CT sink rendering, render/continual_task.rs
  • Catalog builtins: BuiltinContinualTask, 3 system CTs (mz_cluster_replica_metrics_history_ct, mz_cluster_replica_status_history_ct, mz_wallclock_lag_history_ct), mz_continual_tasks table/view/indexes
  • BuiltinsConfig emptied (was only used for include_continual_tasks); BUILTINS::iter() no longer takes a config parameter
  • All CT-specific test files (9 SLT files, 1 testdrive file, 1 parser test file)

What was preserved:

  • mz_audit_log::ObjectType::ContinualTask — audit log entries are immutable
  • audit_log_event_v1::ObjectType::ContinualTask in catalog protos — same reason
  • Historical catalog proto snapshots (v74–v81) — frozen for migration support

https://claude.ai/code/session_01Rx3uSReMsvp3zDTxnXuNUd

@github-actions
Copy link
Copy Markdown
Contributor

Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone.

PR title guidelines

  • Use imperative mood: "Fix X" not "Fixed X" or "Fixes X"
  • Be specific: "Fix panic in catalog sync when controller restarts" not "Fix bug" or "Update catalog code"
  • Prefix with area if helpful: compute: , storage: , adapter: , sql:

Pre-merge checklist

  • The PR title is descriptive and will make sense in the git log.
  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

@antiguru antiguru force-pushed the claude/remove-continual-tasks-akICD branch from e9990a5 to 9187f22 Compare April 15, 2026 14:50
@antiguru antiguru marked this pull request as ready for review April 15, 2026 15:00
@antiguru antiguru requested review from a team as code owners April 15, 2026 15:00
@antiguru antiguru requested review from SangJunBak, aljoscha and teskje and removed request for a team and aljoscha April 15, 2026 15:00
@def-
Copy link
Copy Markdown
Contributor

def- commented Apr 15, 2026

@def-
Copy link
Copy Markdown
Contributor

def- commented Apr 15, 2026

I'd like to see a platform check that creates a CT if the version is old enough to allow it and then checks that we handle the migration properly, could be done with the existing tests in misc/python/materialize/checks/all_checks/continual_task.py

@antiguru antiguru force-pushed the claude/remove-continual-tasks-akICD branch from 4a2664a to b36e831 Compare April 15, 2026 16:22
Copy link
Copy Markdown
Contributor

@teskje teskje left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got halfway through, but need to get lunch now!

Comment thread misc/wasm/Cargo.lock Outdated
Comment thread src/adapter/src/catalog/open.rs Outdated
Comment thread src/catalog/src/durable/upgrade/v81_to_v82.rs Outdated
Comment thread src/catalog/src/builtin.rs Outdated
Comment thread src/compute-client/src/controller.rs Outdated
@antiguru antiguru force-pushed the claude/remove-continual-tasks-akICD branch 2 times, most recently from 3a75b2c to 8e95b61 Compare April 16, 2026 07:46
@antiguru antiguru requested a review from a team as a code owner April 16, 2026 07:46
Remove the continual tasks feature from Materialize. CTs were an
experimental feature that is no longer needed.

Changes:
* Remove CT SQL syntax from parser
* Remove CT catalog items, builtins, and system tables
* Remove CT rendering in compute
* Add catalog migration v81→v82 that panics if CTs are found
* Remove BuiltinsConfig (empty after CT removal)
* Add platform check for CT migration verification
* Clean up related comments and dead code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@antiguru antiguru force-pushed the claude/remove-continual-tasks-akICD branch from 8e95b61 to e80bd1d Compare April 16, 2026 07:48
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems unintentional!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reverts parts of #36032. Worth double-checking if other merge skew made it in as well.

Copy link
Copy Markdown
Contributor

@teskje teskje left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, except for the merge skew!

Comment thread src/persist/src/azure.rs
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge merge skew

{
self.parse_create_materialized_view()
.map_parser_err(StatementKind::CreateMaterializedView)
} else if self.peek_keywords(&[CONTINUAL, TASK]) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anything we can delete from keywords.txt now?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge skew

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge skew

Comment thread LICENSE
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge skew

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants