Skip to content

Fix partitioned asset events incorrectly triggering non-partition-aware Dags#63848

Merged
Lee-W merged 10 commits intoapache:mainfrom
ShubhamGondane:fix/63734-partitioned-asset-non-partitioned-dag
Mar 24, 2026
Merged

Fix partitioned asset events incorrectly triggering non-partition-aware Dags#63848
Lee-W merged 10 commits intoapache:mainfrom
ShubhamGondane:fix/63734-partitioned-asset-non-partitioned-dag

Conversation

@ShubhamGondane
Copy link
Copy Markdown
Contributor

Partitioned asset events (from DAGs using CronPartitionTimetable) were incorrectly
queuing non-partition-aware DAGs in AssetDagRunQueue, causing spurious
ASSET_TRIGGERED runs. Fixed by skipping the non-partitioned queue path when
partition_key is set in AssetManager._queue_dagruns.

closes: #63734


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Sonnet 4.6

Generated-by: Claude Sonnet 4.6 following the guidelines

Comment thread airflow-core/src/airflow/assets/manager.py
Comment thread airflow-core/tests/unit/assets/test_manager.py Outdated
Comment thread airflow-core/newsfragments/63848.bugfix.rst Outdated
@Lee-W
Copy link
Copy Markdown
Member

Lee-W commented Mar 20, 2026

also, we'll need the CI to be fixed

@Lee-W Lee-W requested review from dstandish and uranusjr March 23, 2026 07:32
@Lee-W Lee-W added this to the Airflow 3.2.0 milestone Mar 23, 2026
@Lee-W Lee-W added the AIP-76 label Mar 23, 2026
@Lee-W
Copy link
Copy Markdown
Member

Lee-W commented Mar 23, 2026

I like this idea, but it seems we might have some concerns about whether we want to design it this way. Thus, I'll keep it open for gathering more idea and consensus

@uranusjr
Copy link
Copy Markdown
Member

I have a use case question. Say I have a downstream workflow that wants to react to partitioned upstreams, but doesn’t really care about the particular partitions (maybe to do some global calculations or cache busting etc). Is there a way to do that, or is the downstream forced to have some kind of partition even if its tasks (or even further downstream workflows that depend on its asset events) just ignore it?

Or maybe the design question is more fundamental. An asset event means the asset has changed in some way, and there are really two ways you may want to be notified when change happens: Only on a “full” change (e.g. file replacement, deletion), or on any change. The former kind shouldn’t trigger on a partitioned update, but the latter would want to. Is there a way to distinguish the two? When should this distinguishing logic happen? On the upstream (when an asset event is emitted), or the downstream (when listening to an asset)?

@Lee-W
Copy link
Copy Markdown
Member

Lee-W commented Mar 24, 2026

I have a use case question. Say I have a downstream workflow that wants to react to partitioned upstreams, but doesn’t really care about the particular partitions (maybe to do some global calculations or cache busting etc). Is there a way to do that, or is the downstream forced to have some kind of partition even if its tasks (or even further downstream workflows that depend on its asset events) just ignore it?

I think we'll need some kind of partition for that

Or maybe the design question is more fundamental. An asset event means the asset has changed in some way, and there are really two ways you may want to be notified when change happens: Only on a “full” change (e.g. file replacement, deletion), or on any change. The former kind shouldn’t trigger on a partitioned update, but the latter would want to. Is there a way to distinguish the two?

In the current design, we only have full/non-partitioned and partitioned. We do not yet have a way to listen to both.

When should this distinguishing logic happen? On the upstream (when an asset event is emitted), or the downstream (when listening to an asset)?

If we were to have this, I think it should be handled in the downstream instead.

@Lee-W Lee-W removed the AIP-76 label Mar 24, 2026
@Lee-W
Copy link
Copy Markdown
Member

Lee-W commented Mar 24, 2026

@vatsrahul1001 's going to release rc1, and asking us to merge 3.2.0 PRs. For this issue, I think we need to have 1 fix. If it's not the desired fix, we can revert it and create a follow-up PR with the desired fix

@Lee-W Lee-W changed the title Fix partitioned asset events incorrectly triggering non-partition-aware DAGs Fix partitioned asset events incorrectly triggering non-partition-aware Dags Mar 24, 2026
@Lee-W Lee-W merged commit 20bb220 into apache:main Mar 24, 2026
101 checks passed
vatsrahul1001 pushed a commit that referenced this pull request Mar 25, 2026
…re Dags (#63848)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Mar 30, 2026
…re Dags (apache#63848)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request Mar 30, 2026
…re Dags (#63848)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
(cherry picked from commit 20bb220)
Suraj-kumar00 pushed a commit to Suraj-kumar00/airflow that referenced this pull request Apr 7, 2026
…re Dags (apache#63848)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
abhijeets25012-tech pushed a commit to abhijeets25012-tech/airflow that referenced this pull request Apr 9, 2026
…re Dags (apache#63848)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Asset updates with a partition key run Dags scheduled based on the non-partitioned asset

4 participants