Skip to content

feat: add skip_optimize model config to opt out of post-materialization OPTIMIZE#1485

Open
moomindani wants to merge 1 commit into
databricks:mainfrom
moomindani:feat/skip-optimize-config
Open

feat: add skip_optimize model config to opt out of post-materialization OPTIMIZE#1485
moomindani wants to merge 1 commit into
databricks:mainfrom
moomindani:feat/skip-optimize-config

Conversation

@moomindani
Copy link
Copy Markdown

@moomindani moomindani commented May 27, 2026

Resolves #703

Description

Adds a skip_optimize model config that lets users opt out of the post-materialization OPTIMIZE call without removing zorder / liquid_clustered_by / auto_liquid_cluster from the table definition.

Motivation: the existing opt-out today is the run-wide DATABRICKS_SKIP_OPTIMIZE var, which forces an all-or-nothing decision for the entire invocation. Several users in #703 asked for a config-level opt-out so they can:

  • delegate OPTIMIZE to Predictive Optimization while keeping auto_liquid_cluster=true on the table
  • skip OPTIMIZE only for specific high-churn models and let it run for the rest
  • schedule OPTIMIZE out of band (workflow / job) instead of on the dbt critical path

Behavior:

  • New model config skip_optimize (bool, default false)
  • When truthy, databricks__optimize short-circuits to a no-op even if zorder / liquid_clustered_by / auto_liquid_cluster is set on the model — the clustering declaration remains in the table DDL, only the OPTIMIZE SQL emission is suppressed
  • Inherits via standard dbt config resolution: project → folder → model (more specific wins). Example:
    # dbt_project.yml
    models:
      my_project:
        +skip_optimize: true
        high_read_models:
          +skip_optimize: false
  • DATABRICKS_SKIP_OPTIMIZE var is unchanged (still skips run-wide)

Docs follow-up

User-facing config reference lives in dbt-labs/docs.getdbt.com (databricks-configs.md). A companion docs PR will be opened there to document skip_optimize alongside the existing zorder / liquid_clustered_by / auto_liquid_cluster entries.

Checklist

  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

…tion OPTIMIZE

Adds a `skip_optimize` model config that lets users opt out of the
post-materialization OPTIMIZE call without removing
zorder/liquid_clustered_by/auto_liquid_cluster from the table
definition. Complements the existing run-wide DATABRICKS_SKIP_OPTIMIZE
var by allowing project-, folder-, or model-level opt-out via standard
dbt config inheritance.

Resolves databricks#703

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow users to opt out of optimize calls on liquid-clustering/zordering

1 participant