adapter: replacement materialized views #34234

teskje · 2025-11-20T17:09:07Z

This PR implements an MVP enabling replacement of materialized views. It follows the same ideas as #34032 but models replacement MVs as special MVs instead of introducing a separate catalog item type. This matches the product requirements we arrived at in this Slack thread.

The PR adds the following syntax:

-- To replace a materialized view, the user creates a replacement:
CREATE MATERIALIZED VIEW mv_replacement REPLACING mv AS SELECT ...;
-- The user then observes hydration progress.
-- At some point, they decide it wasn't a good change after all:
DROP MATERIALIZED VIEW mv_replacement;
-- Or, they figure it's a good change and apply it:
ALTER MATERIALIZED VIEW mv APPLY REPLACEMENT mv_replacement;

Modelling replacement MVs as materialized views means that the amount of code changes required is much less than in the alternative approach that introduces a new item type. It also means that diagnostic queries that work for MVs also immediately work for replacements, including SHOW CREATE, EXPLAIN, and EXPLAIN ANALYZE.

There are a few special cases of replacement MVs that this PR doesn't address yet:

It shouldn't be possible to select from replacements.
It shouldn't be possible to create objects that depend on replacements.
It shouldn't be possible to have more than one replacement for the same MV.
Comments on replacements should overwrite comments on the target MV when the replacement is applied.

All of these are left as follow ups.

Motivation

This PR adds a known-desirable feature.

Part of https://github.com/MaterializeInc/database-issues/issues/9903

Tips for reviewer

Commits are meaningful and can be reviewed individually.

Checklist

This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

teskje · 2025-11-27T13:09:45Z

Note that currently it's possible to panic envd by creating an MV that depends on itself, like this:

CREATE TABLE t (a int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t;
CREATE MATERIALIZED VIEW rp REPLACING mv AS SELECT * FROM mv;
ALTER MATERIALIZED VIEW mv APPLY REPLACEMENT rp;

Though this doesn't actually do anything cool, just panics envd 😞 I'll fix this in a follow-up as well.

antiguru · 2025-12-02T12:42:10Z

src/adapter/src/coord/sequencer/inner.rs

+        // TODO(alter-mv): Wait until there is overlap between the old MV's write frontier and the
+        // new MV's as-of, to ensure no times are skipped.


Is this necessary? During planning we should ensure that we're using the existing MV as a dependency and hence should select a as-of that is valid wrt. the old MV. The new MV would eventually catch up to the old MV's write frontier, and then start writing. If the new MV is already past the old MV's write frontier, we should still write the uncompacted history between the two, potentially in a batch spanning many time stamps. That said, I might be missing something, so at least worth explaining more :)

Let's consider the most simple example of a MV that has only a single input, and we replace that with a MV that also only has a single input.

I1 --> MV I2 --> RPL

The interesting case is the one where the since of I2 is greater than the upper of MV. In timestamp selection, we cannot select an as-of that's less than the since of the input, so for RPL we'll most likely select I2's since as the as-of. But since that since is greater than MV's upper, if we'd cut over without waiting for MV's upper to catch up, we'd skip times in the output.

It doesn't really matter that RPL also depends on MV in this example. An additional dependency can only push the as-of further into the future, not pull it into the past. We can't select an as-of that's before I2's since, since then we wouldn't be able to read the input data anymore.

But yeah, seems like a good idea to put a comment somewhere!

src/adapter/src/coord/sequencer/inner.rs

aljoscha

I had some comments inline. I think the code is good but needs the follow-ups that you're already aware off, and maybe resolving some of the murkiness around dependencies, sinces, and how that system currently works.

aljoscha · 2025-12-03T10:16:38Z

src/adapter/src/coord/sequencer/inner/create_materialized_view.rs

+
+            // For now, we don't support schema evolution for materialized views.
+            if &target.desc.latest() != global_lir_plan.desc() {
+                return Err(AdapterError::Unstructured(anyhow!("incompatible schemas")));


Might eventually want a structured error for this one

I think I left that here because I wanted to look into whether this check should/can be moved to planning instead, as a follow-up.

aljoscha · 2025-12-03T10:21:20Z

src/adapter/src/coord/sequencer/inner/create_materialized_view.rs

            .and_then(|r| r.try_step_forward());
        let until = Antichain::from_iter(until_ts);

+        // If this is a replacement MV, ensure that `storage_as_of` > the `since` of the target


This shows that we're in somewhat weird territory with the primary/target business. The replacement doesn't really depend on the target, and so normally we wouldn't want these since invariants here. What have you thought about that?

Yeah, this is needed because in storage collections there is this soft assert. The comment above it explains that we need this for source exports depending on their remap shards. However, the soft assert also applies to storage collections depending on their primaries, where the check doesn't quite make sense.

I didn't want to muck with the deepest storage collections internals here, so I went with the easy workaround.

The replacement doesn't really depend on the target

I would say it does! At least at the SQL-level you can't drop the target without dropping the replacement, as you'd end up with a "dangling pointer". On the storage collection level you could drop them separately, but if you dropped the target GlobalId, there wouldn't be anything left to downgrade the shard's critical since, which seems bad.

aljoscha · 2025-12-03T10:22:44Z

src/sql/src/plan/statement/ddl.rs

        .map(|gid| scx.catalog.resolve_item_id(&gid))
        .collect();

+    if let Some(id) = replacement_target {


Will dropping an MV while there is a live replacement block unless you use CASCADE? And will CASCADE also drop the replacements?

Yep! See the tests in replacement-materialized-views.td.

src/adapter/src/coord/ddl.rs

aljoscha · 2025-12-03T10:51:32Z

Approved now, because we know that we will keep iterating on this one, and it unblocks some of the follow-up work, demos, the like!

Once materialized views can have multiple historical storage collections associated with them, we'll need to make sure to drop all of those when dropping a materialized view. There is always only a single compute collection to drop. This commit ensures we'll do the right thing and also simplifies the item dropping code a bit in the process: Instead of keeping track of indexes/MVs/CTs to drop, we instead only track compute and storage collections to drop. Doing so allows deleting a bunch of duplicate code.

This commit adds a feature flag to gate the creation and application of replacement materialized views. It's disabled by default and enabled in CI.

antiguru

LGTM. I left one comment around SQL syntax.

antiguru · 2025-12-04T08:57:45Z

src/sql-parser/src/ast/defs/statement.rs

        }

+        if let Some(target) = &self.replacing {
+            f.write_str(" REPLACING ");


I wonder about the SQL syntax, but don't have a strong opinion:

replacing might indicate something continuous, but we're only replacing once.

replaces also seems to be that it actively replaces.

replace would be less opinionated. Seems similar to SQL's CASCADE.
Just putting this out here, maybe a native speaker has some input!

No strong opinion from me either! I've put this down as one of the remaining things to figure out in https://github.com/MaterializeInc/database-issues/issues/9903

When a replacement MV is applied, the catalog is updated by dropping the replacement and updating the target MV with a new version. This transfers both the replacement's compute and storage collection to the target MV, so they should not be dropped. The implication application logic must thus be extended to account for that.

For compute collections, it is wrong to assume that all GlobalIds associated with a catalog entry (specifically an MV catalog entry) also map to live compute collections. Instead, only the "write GlobalId" has a live compute collection. Compute collections that were previously maintaining the catalog item have likely been dropped already.

teskje · 2025-12-04T13:28:24Z

TFTRs!

teskje force-pushed the replacement-mvs branch 8 times, most recently from 2bf4ec7 to fec6902 Compare November 25, 2025 11:37

teskje changed the title ~~[wip] replacement materialized views~~ adapter: replacement materialized views Nov 25, 2025

teskje marked this pull request as ready for review November 25, 2025 11:54

teskje requested review from a team as code owners November 25, 2025 11:54

teskje requested review from SangJunBak and aljoscha and removed request for SangJunBak November 25, 2025 11:54

teskje force-pushed the replacement-mvs branch from fec6902 to 2db420a Compare November 26, 2025 11:06

antiguru reviewed Dec 2, 2025

View reviewed changes

aljoscha approved these changes Dec 3, 2025

View reviewed changes

teskje added 4 commits December 3, 2025 12:42

test: add tests for replacement materialized views

6ca2948

sql-parser: support for replacement materialized views

70408a5

adapter,sql: plan and sequence replacement MV creation

417c7f7

teskje force-pushed the replacement-mvs branch from 2db420a to 937ba62 Compare December 3, 2025 11:59

teskje added 2 commits December 3, 2025 13:06

adapter,sql: plan and sequence replacement MV application

8b09f9b

sql: enable_replacement_materialized_views feature flag

a0bbb0b

This commit adds a feature flag to gate the creation and application of replacement materialized views. It's disabled by default and enabled in CI.

teskje force-pushed the replacement-mvs branch from 937ba62 to a0bbb0b Compare December 3, 2025 12:10

antiguru approved these changes Dec 4, 2025

View reviewed changes

antiguru mentioned this pull request Dec 4, 2025

Replace materialized views #34032

Closed

teskje added 2 commits December 4, 2025 13:53

teskje force-pushed the replacement-mvs branch from 9c732d0 to 4d63825 Compare December 4, 2025 12:53

teskje merged commit 3cd5592 into MaterializeInc:main Dec 4, 2025
131 checks passed

teskje deleted the replacement-mvs branch December 4, 2025 13:28

		// TODO(alter-mv): Wait until there is overlap between the old MV's write frontier and the
		// new MV's as-of, to ensure no times are skipped.

adapter: replacement materialized views #34234

adapter: replacement materialized views #34234

Uh oh!

Conversation

teskje commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Tips for reviewer

Checklist

Uh oh!

teskje commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teskje Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aljoscha left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aljoscha commented Dec 3, 2025

Uh oh!

antiguru left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teskje commented Dec 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

teskje commented Nov 20, 2025 •

edited

Loading

teskje commented Nov 27, 2025 •

edited

Loading

teskje Dec 2, 2025 •

edited

Loading