[design doc] EXPLAIN in Postgres syntax

[mgree]

An outline of how to implement the last stage of https://github.com/MaterializeInc/database-issues/issues/8889.

Rendered document.

Motivation

• This PR adds a known-desirable feature.

Tips for reviewer

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

[ggevay]

I wrote some comments.

It looks good; hopefully the Postgres syntax will be more familiar to users.

And I'm happy that we are basing it on LIR!

[ggevay]

Missing ]. Also, maybe you wanted to make this into a sentence?

[ggevay]

Does Postgres have Flat Map? If not, we could consider Table Function.

[mgree]

Not exactly:

michaelgreenberg=# EXPLAIN SELECT DISTINCT l_discount, generate_series(1, 5) from lineitem;
                               QUERY PLAN                               
------------------------------------------------------------------------
 HashAggregate  (cost=23.50..36.00 rows=1000 width=22)
   Group Key: l_discount, generate_series(1, 5)
   ->  ProjectSet  (cost=0.00..18.50 rows=1000 width=22)
         ->  Seq Scan on lineitem  (cost=0.00..12.00 rows=200 width=18)
(4 rows)

I'm fine with Table Function or ProjectSet.

[frankmcsherry]

I vote Table Function!

[ggevay]

Why not just Distinct?

[mgree]

Just following Postgres, which always calls reduces a GroupAggregate:

EXPLAIN SELECT DISTINCT l_discount FROM lineitem;
                            QUERY PLAN                            
------------------------------------------------------------------
 HashAggregate  (cost=12.50..14.50 rows=200 width=18)
   Group Key: l_discount
   ->  Seq Scan on lineitem  (cost=0.00..12.00 rows=200 width=18)
(3 rows)

[ggevay]

I think it's fine to show it as Distinct. I'd say this is a clear improvement over Postgres not showing it as Distinct. There is no need to follow Postgres just for familiarity here, because we don't really need to rely on familiarity, as it should be pretty clear for users what Distinct is.

[ggevay]

I'm wondering whether users need to know about whether a Union consolidates. Seems like a somewhat esoteric implementation-detail, so maybe not. Also, I can't really think of a situation where user would need to make some decision based on knowing whether a Union consolidates or not.

[mgree]

Happy to skip it! My understanding was that we sometimes cared about this in accounting for memory footprint, but maybe that's an us/VERBOSE TEXT thing?

[ggevay]

Yeah, it's more for us, enough in VERBOSE TEXT. (But even I haven't looked at this in a long time. Currently, the heuristic for whether a Union consolidates is very simple: if there is at least 1 negated input, then we consolidate. If we were to make the heuristic more complex then we'd need to more often look at it. There was an abandoned attempt on this here: #30360 )

[ggevay]

I think GetPlan's correspondence to reality is a bit more complicated. I don't have the time today to delve into this, but here are my raw notes on what each of the variants are, and some possible refactorings:

GetPlan is unncessarily hard to understand:

• GetPlan::PassArrangements:
  • no MFP
  • ! BUT: there might or might not be an input arrangement that we are passing (so it's not always an Index Scan)
• GetPlan::Arrangement:
  • there is an input arrangement
  • no output arrangement
  • there is an MFP
  • we might be seeking a key (but that code path is deprecated, so it occurs very rarely; still occurs in aggregates.slt and is_null_propagation.slt) (so it's usually not an Index Lookup)
• GetPlan::Collection:
  • no input arrangement (and no output arrangement)
  • currently, there is always an MFP (but this will change when we change PassArrangements)

possible refactoring (which would be useful even independently from EXPLAIN, because the above state of affairs is IMHO really counter-intuitive):

• GetPlan::PassArrangements -- add an if to only produce it if there is an arrangement
• split off a new lookup variant from GetPlan::Arrangement
• rename GetPlan variants
• Get::keys -- make it clear that these are output arrangements

[mgree]

Let's talk about this (maybe with @frankmcsherry?) and find the right way to factor this out so that things are actually clear.

[frankmcsherry]

I'm for sure available to discuss. I think we have a few moments of inherent complexity (from the pov of explain) that is more incidental from the pov of plans. Probably the right thing at the moment is to land a thing that speaks unambiguously about the important details. I'm all for figuring out the complexity here, boiling it away, but .. we'll have to iterate and perhaps the right thing at the moment is eat the fact that the truth is gross, for now.

[ggevay]

Yes, it could be off by default when we have good column names.

[ggevay]

I don't think that's viable, because there are a lot of minor things to take care of when implementing plan printing, and it would be hard to do all those in SQL.

[mgree]

Fair enough!

[ggevay]

We could leave off nulls_first/nulls_last if they are at their default setting. (Which I think is nulls_first for desc, and nulls_last for asc.

[ggevay]

I'm not sure what Postgres prints for LIMIT/OFFSET, but we might try to match that too if it makes sense.

[mgree]

They seem to show LIMIT but not OFFSET:

michaelgreenberg=# EXPLAIN SELECT DISTINCT l_discount FROM lineitem LIMIT 10 OFFSET 15;
                               QUERY PLAN                               
------------------------------------------------------------------------
 Limit  (cost=12.65..12.75 rows=10 width=18)
   ->  HashAggregate  (cost=12.50..14.50 rows=200 width=18)
         Group Key: l_discount
         ->  Seq Scan on lineitem  (cost=0.00..12.00 rows=200 width=18)
(4 rows)

[ggevay]

As mentioned elsewhere, we'll have a problem with self-joins: we'll be seeing conditions like x = x, where one x is from one join input, and the other x is from another join input. Even including the table name won't resolve it if it's a self-join. And we have a lot of self-joins due to outer join lowering / subquery lowering creating them.

A way out could be to write something like %1.x = %2.x, where %1, %2, ... are the join inputs.

How does Postgres resolve this btw.?

[mgree]

They use the relevant aliases:

michaelgreenberg=# explain select l1.l_orderkey, l2.l_orderkey from lineitem l1, lineitem l2 where l1.l_partkey = l2.l_partkey and l1.l_orderkey <> l2.l_orderkey and l1.l_shipdate < l2.l_shipdate;
                                      QUERY PLAN                                       
---------------------------------------------------------------------------------------
 Hash Join  (cost=14.50..35.00 rows=66 width=8)
   Hash Cond: (l1.l_partkey = l2.l_partkey)
   Join Filter: ((l1.l_orderkey <> l2.l_orderkey) AND (l1.l_shipdate < l2.l_shipdate))
   ->  Seq Scan on lineitem l1  (cost=0.00..12.00 rows=200 width=12)
   ->  Hash  (cost=12.00..12.00 rows=200 width=12)
         ->  Seq Scan on lineitem l2  (cost=0.00..12.00 rows=200 width=12)
(6 rows)

I don't think it's possible to have a self-join without such aliases, but I'm not 100% on that.

michaelgreenberg=# explain select * from lineitem join lineitem on (l_partkey);
ERROR:  table name "lineitem" specified more than once

[ggevay]

(Discussed in the Optimizer sync meeting: Our lowering for outer joins / subqueries also creates self-joins. The solution will probably be to make the lowering think up synthetic aliases.)

[mgree]

The changes in #31878 should make it so we have table aliases in joins:

materialize/src/sql/src/plan/query.rs

Lines 6625 to 6633 in 8e29cdc

	pub fn intern_scope_item(&mut self, item: &ScopeItem) -> Arc<str> {
	if let Some(table_name) = &item.table_name {
	// In order to avoid clutter, we're just going to use the table name (not database or schema)
	self.intern(format!("{}.{}", table_name.item, item.column_name))
	} else {
	self.intern(item.column_name.as_str())
	}
	}
	}

[mgree]

Okay, I tried to incorporate these changes---please take a look!

[ggevay]

Do the % numbers mean which join path uses the index? (If yes, then there seems to be a mixup: %0 and %2 user different indexes.)

[mgree]

It's meant to indicate the join path, yeah... I might have messed it up manually coming up with this!

[ggevay]

We might want to leave off the Consolidating case.

Edit: Oh, I see we discussed this already.

[ggevay]

Great, thanks!

[antiguru]

(I don't have a strong opinion as I lack some background in the area, so take my words with a grain of salt.)

I think this proposal is fine, as in it moves us from one potentially confusing representation to something that users might be more familiar with. I like the way it maps LIR nodes to more practical terms (what does Basic even mean?), it should make it easier to follow for users who aren't familiar with the internal workings of Materialize.

The flipside is that I'm not sure by how much we'll improve user experience, and for how long the proposal will make sense in the future. While LIR is closest to what we're executing, it doesn't need to stay like this forever, or we might change LIR to something that is more amenable to dataflow rendering, i.e., different MirScalarExpr evaluation. Basing our explains off something that is even more tuned for a computer to be interpreted than a human might have the effect that explaining will need to recover information that is otherwise not available.

The other part I'm missing is whether a tree-based representation is the best we can do. We've had different approaches in the past, including the current tree-based one, and they all had trade-offs. A stack-machine-like syntax is much closer to what we're rendering (we're not rendering trees), but confusing for people who assume the database follows some volcano-style execution pattern. The tree-based variant is in some way a mis-representation of what's happening, but caters to people who've used explain in the past.

What I mean to say is that this proposal seems like a step in one direction, and fine in the small, but I'm missing the bigger picture -- what is the best way to represent what essentially boils down to a program?

[mgree]

Thank you for the review!

I'm going to merge this---not as a commitment to do exactly what I describe here, but to record that this is the best revision of EXPLAIN we have in mind just now. I'd be very happy to have this proposal superseded by something better before I have time to implement it.

Other databases use treelike or tabular formats. The only exception I can think of is Soufflé, which compiles Datalog programs to the RAM (relational abstract machine, IIRC). In principle you could use the RAM to debug your programs, though it doesn't seem like Soufflé makes that easy. Even so, RAM programs are tree structured (but some of the nodes are loops!).

Our EXPLAIN layouts are already more linear than most databases, because of our heavy use of Let. We could emphasize that "flatness" more in the output, but I'm not sure how to emphasize the reuse we get---that our programs are DAGs, not trees (nevermind that WMR means they're just DGs, hold the A). Some of the graphical renderings we've done in the past are quite nice for that... maybe we should direct more effort to graphical explanation than textual explanation?

[ggevay]

how long the proposal will make sense in the future. While LIR is closest to what we're executing, it doesn't need to stay like this forever, or we might change LIR to something that is more amenable to dataflow rendering

What I feel is the essence of this design doc is that we want to do the following two things:

1. Base the default EXPLAIN on some low-level IR, so that it's fairly faithful to what actually happens during execution. (This is in contrast to the current default EXPLAIN, which is based on MIR, so e.g., it's not clear what arrangements will actually get created, and it lies about MFPs being on top of joins when in fact they are pushed into joins, etc.)
2. At the same time, make a serious effort to present the large amount of information that is in the low-level IR in a concise, readable form. (This is in contrast to the current LIR EXPLAIN, which appears as a giant pile of unreadable minutiae to users.)

I'd say that even if in the future we make some big changes to LIR, we'll still want to keep doing the above 1. and 2.

The other part I'm missing is whether a tree-based representation is the best we can do.

maybe we should direct more effort to graphical explanation than textual explanation?

:10000: I think that's the long-term solution to the "how to lay it out on the screen?" question. Non -tree DAGs due to Lets? No problem, we can just draw those non-tree edges. Cycles due to WMR? No problem we can just draw those edges too!

Btw., the other long-term thing that would hugely help EXPLAIN's readability is to do away with the minimalism of MIR: introduce explicit representations at the MIR/LIR levels for outer joins, some of the subquery machinery (shared with outer joins), window functions, and possibly even global aggregates. The minimalism might have made sense at an early-stage startup, but it seems it requires too much cleverness from the optimizer to robustly deal with the convoluted plans that we currently get from outer joins, subqueries, and window functions, plus it makes EXPLAIN hard to read.

[ggevay]

Random thought about MFP fusion. Maybe we could print fused MFP as a separate operator, similarly to our current MIR EXPLAIN, but add a note that it's fused. For example, in the following example it's critically important that the MFP is fused into the FlatMap, because otherwise data would be copied for every attendee:

explain
SELECT
   (e.data -> 'fullDocument' -> '_id' ->> '$oid') AS id,
   jsonb_array_elements(e.data -> 'fullDocument' -> 'attendees') AS attendee
FROM materialize.prod_sources.src_mongodb_events e;


Explained Query:
  Project (#2, #1) // { arity: 2 }
    Map ((((#0{data} -> "fullDocument") -> "_id") ->> "$oid")) // { arity: 3 }
      FlatMap jsonb_array_elements(((#0{data} -> "fullDocument") -> "attendees")) // { arity: 2 }
        Project (#1{data}) // { arity: 1 }
          ReadStorage materialize.prod_sources.src_mongodb_events // { arity: 2 }

explain physical plan for
SELECT
   (e.data -> 'fullDocument' -> '_id' ->> '$oid') AS id,
   jsonb_array_elements(e.data -> 'fullDocument' -> 'attendees') AS attendee
FROM materialize.prod_sources.src_mongodb_events e;


Explained Query:
  FlatMap jsonb_array_elements(((#0 -> "fullDocument") -> "attendees"))
    mfp_after
      project=(#2, #1)
      map=((((#0 -> "fullDocument") -> "_id") ->> "$oid"))
    Get::Collection materialize.prod_sources.src_mongodb_events
      raw=true

Source materialize.prod_sources.src_mongodb_events
  project=(#1)

And it could be something like:

  Project (#2, #1) // (fused down into FlatMap)
    Map ((((#0{data} -> "fullDocument") -> "_id") ->> "$oid")) // (fused down into FlatMap)
      FlatMap jsonb_array_elements(((#0{data} -> "fullDocument") -> "attendees"))
        Project (#1{data}) // { arity: 1 }
          ReadStorage materialize.prod_sources.src_mongodb_events // { arity: 2 }

And maybe we could do the same also for MFPs that are pushed into joins: put the MFP on top of the join, but add a note that it's fused/pushed into the join.

[mgree]

If the new explain (after #31878 lands) will be using LIR, I think we'll be presenting things closer to this way anyway! When I have a prototype for that new EXPLAIN, let's revisit this and make sure we're happy with what we're producing.