Skip to content

Make Prefect an Extra of Cascade#49

Open
jhamet93 wants to merge 4 commits intomainfrom
jhamet/Make_Prefect_Optional
Open

Make Prefect an Extra of Cascade#49
jhamet93 wants to merge 4 commits intomainfrom
jhamet/Make_Prefect_Optional

Conversation

@jhamet93
Copy link
Copy Markdown
Collaborator

At the moment, cascade is coupled to prefect even though there is no fundamental reason that requires this. Users should be able to use this library without prefect. Therefore, this change keeps the existing functionality but requires the user to install prefect in their environment if they wish for the code path of automatic inference to be activated.

Therefore this change introduces the following:

  • prefect is now an extra
  • prefect<2 support is dropped
  • A Context is introduced to represent the execution environment in which different implementations can obtain values that may be needed for remote submission of a job.
  • Remove top level prefect imports from the common entrypoints

@jhamet93 jhamet93 force-pushed the jhamet/Make_Prefect_Optional branch from b5fb8b2 to f96124b Compare March 13, 2025 15:39
Copy link
Copy Markdown
Collaborator

@AsherWright AsherWright left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@gracevivi523
Copy link
Copy Markdown

Overall looks great to me, just a question from my side regarding to Context, would a Context for databricks is necessary or not in addition to gcp and prefect context?

@jhamet93
Copy link
Copy Markdown
Collaborator Author

Overall looks great to me, just a question from my side regarding to Context, would a Context for databricks is necessary or not in addition to gcp and prefect context?

Internally, databricks has only been the destination for an execution, never a source (i.e. cascade decorated executions are not submitted from Databricks).

@jhamet93 jhamet93 force-pushed the jhamet/Make_Prefect_Optional branch from 2490ea8 to 2be0815 Compare July 7, 2025 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants