Fix partitioned asset events incorrectly triggering non-partition-aware Dags#63848
Conversation
|
also, we'll need the CI to be fixed |
|
I like this idea, but it seems we might have some concerns about whether we want to design it this way. Thus, I'll keep it open for gathering more idea and consensus |
|
I have a use case question. Say I have a downstream workflow that wants to react to partitioned upstreams, but doesn’t really care about the particular partitions (maybe to do some global calculations or cache busting etc). Is there a way to do that, or is the downstream forced to have some kind of partition even if its tasks (or even further downstream workflows that depend on its asset events) just ignore it? Or maybe the design question is more fundamental. An asset event means the asset has changed in some way, and there are really two ways you may want to be notified when change happens: Only on a “full” change (e.g. file replacement, deletion), or on any change. The former kind shouldn’t trigger on a partitioned update, but the latter would want to. Is there a way to distinguish the two? When should this distinguishing logic happen? On the upstream (when an asset event is emitted), or the downstream (when listening to an asset)? |
I think we'll need some kind of partition for that
In the current design, we only have full/non-partitioned and partitioned. We do not yet have a way to listen to both.
If we were to have this, I think it should be handled in the downstream instead. |
|
@vatsrahul1001 's going to release rc1, and asking us to merge 3.2.0 PRs. For this issue, I think we need to have 1 fix. If it's not the desired fix, we can revert it and create a follow-up PR with the desired fix |
…re Dags (#63848) Co-authored-by: Wei Lee <weilee.rx@gmail.com>
…re Dags (apache#63848) Co-authored-by: Wei Lee <weilee.rx@gmail.com>
…re Dags (apache#63848) Co-authored-by: Wei Lee <weilee.rx@gmail.com>
…re Dags (apache#63848) Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Partitioned asset events (from DAGs using
CronPartitionTimetable) were incorrectlyqueuing non-partition-aware DAGs in
AssetDagRunQueue, causing spuriousASSET_TRIGGEREDruns. Fixed by skipping the non-partitioned queue path whenpartition_keyis set inAssetManager._queue_dagruns.closes: #63734
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Sonnet 4.6 following the guidelines