Deterministic non-streaming lifecycle ordering for concurrent workflow branches

[davidahmann]

Problem observed

In fan-out workflows, executor lifecycle events can arrive in scheduler-dependent order. That means equivalent runs may produce different non-streaming traces, even when outputs are the same. The behavior is logically valid but operationally noisy: replay comparisons and incident triage need deterministic receipts, not ordering drift from concurrent completion timing.

Why it matters operationally

Agent Framework treats tracing/debugging as a first-class experience. When event ordering shifts between equivalent runs, teams have to hand-normalize traces before they can reason about regressions. This is especially painful in workflows with parent/tool/subworkflow branches, where lifecycle events are the core timeline for “what ran when.” Deterministic ordering in non-streaming results makes audit and diff workflows substantially more reliable without changing executor semantics.

Minimal repro

cd python
uv run pytest packages/core/tests/workflow/test_workflow.py -k 'fan_out_multiple_completed_events or parallel_executor_lifecycle_order_is_deterministic_in_run_results'
uv run pytest packages/core/tests/workflow/test_workflow.py

Fix approach

I added a small normalization step in non-streaming finalization:

• keep non-lifecycle event positions intact
• within each superstep window, sort lifecycle events (executor_invoked, executor_completed, executor_failed) by executor_id per lifecycle type

This yields stable non-streaming traces for equivalent runs while avoiding changes to executor behavior or streaming transport behavior.

Validation evidence

• Targeted workflow tests for fan-out and deterministic lifecycle ordering passed
• Full packages/core/tests/workflow/test_workflow.py passed
• Ruff checks on touched workflow runtime/test files passed

Open follow-up question for maintainers

Do you want the same normalization contract extended to streaming mode, or should deterministic ordering remain a non-streaming receipt guarantee only?

Inspired by research context: CAISI publishes independent, reproducible AI agent governance research: https://caisi.dev