Skip to content

Conversation

@ggevay
Copy link
Contributor

@ggevay ggevay commented Jun 4, 2025

This PR implements the first part of #32504, namely the moving of began_at (and the corresponding execution-began event in the lifecycle table) up to the Adapter frontend (protocol.rs) from the Coordinator. For this, we pass down the timestamp from the Adapter frontend to the Coordinator through the Portal. Specifically, we take the timestamp after the self.conn.recv() returns, and pass it down for Simple Queries as well as for the FrontendMessage::Execute of the Extended Query flow.

The LifecycleTimestamps struct contains only one field for now, but I plan to add more things to it later.

Testing is a bit hard for this, but I did the following:

  • A minor thing is that I've added a test in statement-logging.td to test that mz_statement_execution_history.began_at is always exactly the same as mz_statement_lifecycle_history.occurred_at on the execution-began event. On main, these were close, but not always exactly the same, as the two timestamps were taken in separate calls to self.now(). (I've verified that the test indeed doesn't always pass on main.)
  • Verifying that began_at is taken in the Adapter frontend is a bit harder. I couldn't think of a regression test for this (I'm open to ideas), but I did some tests with extra code that is not possible to merge:
    • If I put a panic! in the else branch of the let began_at = if let Some(lifecycle_timestamps) = lifecycle_timestamps, the panic doesn't happen while running statement-logging.td or when running slts or when running queries manually.
    • I've put a sleep at the beginning of the Some(FrontendMessage::Execute { match arm in protocol.rs, and verified that finished_at - began_at includes that sleep on the PR, but not on main. (Note that Testdrive submits queries using the Extended Query flow.)

Motivation

Tips for reviewer

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

@ggevay ggevay force-pushed the lifecycle-parsing branch from c6cbb55 to 789bcc8 Compare June 4, 2025 10:28
@ggevay ggevay marked this pull request as ready for review June 4, 2025 10:34
@ggevay ggevay requested a review from a team as a code owner June 4, 2025 10:34
@ggevay ggevay requested a review from aljoscha June 4, 2025 10:34
@ggevay ggevay force-pushed the lifecycle-parsing branch from 789bcc8 to e477cd5 Compare June 4, 2025 10:45
Copy link
Contributor

@aljoscha aljoscha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, and I think we don't need to contort ourselves and try and find a way to make this testable. Ultimately, we'll notice pretty quickly when this doesn't deliver the right timestamps, I would think.


impl LifecycleTimestamps {
/// Creates a new `LifecycleTimestamps`.
pub fn new(received: EpochMillis) -> Self {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not something you have to change now, but I think this new() will become unwieldy in the future, when the parameters are just a bunch of EpochMillis. Might be better to just always initialize using LifecycleTimestamps { this_timestamp, that_timestamp, etc. }.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll keep this in mind when adding more fields!

@ggevay ggevay merged commit 756eabe into MaterializeInc:main Jun 4, 2025
82 checks passed
@ggevay
Copy link
Contributor Author

ggevay commented Jun 4, 2025

Thanks for the quick review!

@ggevay ggevay added the A-ADAPTER Topics related to the ADAPTER layer label Jun 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ADAPTER Topics related to the ADAPTER layer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants