Add exception propagation for deferred and async execution by rostan-t · Pull Request #6210 · NVIDIA/DALI

rostan-t · 2026-02-13T14:56:24Z

Category:

New feature (non-breaking change which adds functionality)

Description:

When using asynchronous execution, exceptions are raised when evaluating rather than directly when calling functions. This makes it harder to trace back exception to their actual source.

This PR modified the raised exception to fix this issue. The stack trace is captured when calling an operator and used to create a new exception with a proper traceback when necessary.

Additional information:

Affected modules and functionalities:

Dynamic mode

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

Implements new requirements
Affects existing requirements
N/A

REQ IDs: N/A

JIRA TASK: DALI-4599

greptile-apps · 2026-02-13T14:59:19Z

Greptile Summary

This PR adds exception propagation for deferred and async execution in DALI's dynamic mode. When operators fail during asynchronous or deferred evaluation, the error was previously raised at the point of evaluation (e.g., .evaluate()), making it hard to trace back to the original operator call. This PR captures the call stack at operator invocation time and uses it to construct a synthetic traceback that points back to the original call site.

Adds _exceptions.py with capture_stack() to capture frames at invocation time and rethrow_exception() to re-raise exceptions with a synthetic traceback pointing to the original call site, chained via raise ... from ...
Modifies Invocation.__init__ to capture the call stack when in deferred or eager mode (skipped for sync modes where the natural traceback is sufficient)
Threads the captured call stack through _AsyncExecutor.submit() and _Future so async exceptions also get the origin traceback
Adds caller_depth parameter to Invocation to accommodate different call depths from _op_builder.py (default 4) vs _batch2tensor.py (explicit 3)
Adds comprehensive tests in test_exceptions.py that verify exception chaining works correctly for both deferred and eager modes
Removes glob message matching from some existing assert_raises calls since the exception messages are now wrapped by the rethrow mechanism

Confidence Score: 4/5

This PR is well-structured and safe to merge with minor considerations around test assertion specificity.
The implementation is sound and well-guarded: stack capture is only done for modes that need it (deferred/eager), exception rethrow is gated by both _call_stack and _eval_mode being non-None, and the synthetic frame construction uses documented Python APIs. The exception recreation via type(old_exception)(message) is safe because DALI's C++ backend only surfaces standard Python exception types. New tests cover the main scenarios. Minor concern: some existing tests had their error message assertions weakened.
_exceptions.py and _invocation.py are the core logic files — pay attention to the interaction between _eval_mode assignment timing in apply_eval_policy and the rethrow guard in run().

Important Files Changed

Filename	Overview
dali/python/nvidia/dali/experimental/dynamic/_exceptions.py	New module providing call stack capture and synthetic traceback construction for re-raising exceptions with origin context. Implementation is sound — uses documented Python APIs for frame/traceback synthesis.
dali/python/nvidia/dali/experimental/dynamic/_invocation.py	Integrates call stack capture and exception rethrow into operator invocations. Stack is captured only for deferred/eager modes. Exception rethrow is guarded by both `_call_stack` and `_eval_mode` being non-None.
dali/python/nvidia/dali/experimental/dynamic/_async.py	Threads call stack through `_Future` and `_AsyncExecutor.submit()`. On future wait, exceptions are rethrown with the captured origin traceback. Clean integration with existing async machinery.
dali/python/nvidia/dali/experimental/dynamic/_batch2tensor.py	Adds explicit `caller_depth=3` to Invocation constructor call, reflecting the shallower call stack for this pseudo-operator compared to the default depth of 4.
dali/test/python/experimental_mode/test_exceptions.py	New test file with an `exception_tester` decorator that verifies exceptions are properly propagated with correct cause chaining for both deferred and eager modes. Tests multiple error scenarios.
dali/test/python/experimental_mode/test_async.py	Removes `test_exception_propagation` which is now covered by the more comprehensive `test_exceptions.py`. Removes unused `raises` import.
dali/test/python/experimental_mode/test_interop.py	Adds missing license header, removes `glob` parameter from `assert_raises` since the error message may now be wrapped by the rethrow mechanism.
dali/test/python/experimental_mode/test_tensor.py	Removes `glob` parameter from `assert_raises` in `test_batch_to_tensor_no_pad_error` to accommodate the new exception wrapping behavior.

Sequence Diagram

sequenceDiagram
    participant User as User Code
    participant Op as Operator.__call__
    participant Inv as Invocation
    participant Exc as _exceptions
    participant Async as _AsyncExecutor
    participant Worker as Worker Thread

    User->>Op: call operator (e.g., ndd.resize)
    Op->>Inv: Invocation.__init__(caller_depth=4)
    Inv->>Exc: capture_stack(depth) [deferred/eager only]
    Exc-->>Inv: CallStack (code objects + line numbers)
    
    alt Eager Mode
        Inv->>Async: submit(callable, call_stack)
        Async->>Worker: queue task with call_stack
        Worker->>Worker: _run_impl() → exception!
        Worker-->>Async: store exception in _Future
        User->>Inv: evaluate() / .cpu() / etc.
        Inv->>Async: future.wait()
        Async->>Exc: rethrow_exception(exc, call_stack, eager)
        Exc->>Exc: _make_traceback(call_stack)
        Exc-->>User: raise new_exc from original_with_synthetic_tb
    else Deferred Mode
        Note over Inv: No immediate execution
        User->>Inv: evaluate() / .cpu() / etc.
        Inv->>Inv: _run_impl() → exception!
        Inv->>Exc: rethrow_exception(exc, call_stack, deferred)
        Exc->>Exc: _make_traceback(call_stack)
        Exc-->>User: raise new_exc from original_with_synthetic_tb
    else Sync Mode
        Inv->>Inv: _run_impl() [no stack capture]
        Note over Inv: Natural traceback suffices
    end

_{Last reviewed commit: baf619a}

greptile-apps

_{5 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

Signed-off-by: Rostan Tabet <rtabet@nvidia.com>

greptile-apps

_{5 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

rostan-t · 2026-02-13T15:07:23Z

!build

dali-automaton · 2026-02-13T15:10:48Z

CI MESSAGE: [43977528]: BUILD STARTED

dali-automaton · 2026-02-13T17:00:18Z

CI MESSAGE: [43977528]: BUILD FAILED

Signed-off-by: Rostan Tabet <rtabet@nvidia.com>

klecki · 2026-02-13T18:56:45Z

+    while frame is not None:
+        if limit is not None and n >= limit:
+            break
+        stack.append((frame.f_code, frame.f_lineno))


As we are keeping the reference? to the frame's code, could this keep the frame alive?
I think that that's the whole reason for the FrameSummary objects, to remove any references and not prolong the lifetime in unintended way.

We are only keeping a reference to the code object, which shouldn't be referencing the frame at all.

>>> def foo(): ... return 1 + 2 ... >>> import gc >>> gc.get_referents(foo.__code__) []

Signed-off-by: Rostan Tabet <rtabet@nvidia.com>

rostan-t · 2026-02-16T09:08:14Z

!build

dali-automaton · 2026-02-16T09:10:31Z

CI MESSAGE: [44144024]: BUILD STARTED

greptile-apps

_{8 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

mzient · 2026-02-16T09:28:07Z

 def test_ragged_batch_to_torch_error(device: str):
    batch = ndd.batch([[1, 2, 3], [4, 5], [6]])
-    with assert_raises(ValueError, glob="non-uniform"):
+    with assert_raises(ValueError, glob="non-uniform"), ndd.EvalMode.sync_cpu:


Why? Don't we have to synchronize anyway to get a torch tensor?

Yes, we do. However, the exception happens before the synchronization: as_tensor calls batch_to_tensor and the exception is raised here, resulting in a AsynchronousExecutionError.

baf619a makes the re-raised exception type the same as the initial one. We just need to remove the message check.

Signed-off-by: Rostan Tabet <rtabet@nvidia.com>

greptile-apps

_{8 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

dali-automaton · 2026-02-16T11:31:51Z

CI MESSAGE: [44144024]: BUILD PASSED

rostan-t · 2026-02-16T11:43:48Z

!build

dali-automaton · 2026-02-16T11:45:40Z

CI MESSAGE: [44150629]: BUILD STARTED

mzient · 2026-02-16T12:54:26Z

 def test_ragged_batch_to_torch_error(device: str):
    batch = ndd.batch([[1, 2, 3], [4, 5], [6]])
-    with assert_raises(ValueError, glob="non-uniform"):
+    with assert_raises(ValueError):


Doesn't the original message show up when the error is rethrown?

assert_raises performs the check only on the error message of the rethrown exception.

The original message shows up but in the cause. It looks something like this:

Traceback (most recent call last): <synthetic traceback> ValueError: <intial message> The above exception was the direct cause of the following exception: Traceback (most recent call last): <initial traceback> ValueError: An error happened during asynchronous execution

dali-automaton · 2026-02-16T14:05:02Z

CI MESSAGE: [44150629]: BUILD PASSED

* Add exception propagation for deferred and async execution * Remove test_async.test_exception_propagation * Move the stack capture to the invocation. Fix it for the batch2tensor op * Add missing header in test_interop.py * Re-raise exceptions with the same type --------- Signed-off-by: Rostan Tabet <rtabet@nvidia.com>

rostan-t added the Dynamic Mode label Feb 13, 2026

github-advanced-security AI found potential problems Feb 13, 2026

View reviewed changes

Comment thread dali/test/python/experimental_mode/test_exceptions.py Fixed

greptile-apps Bot reviewed Feb 13, 2026

View reviewed changes

Comment thread dali/python/nvidia/dali/experimental/dynamic/_invocation.py Outdated

Comment thread dali/python/nvidia/dali/experimental/dynamic/_exceptions.py Outdated

Comment thread dali/python/nvidia/dali/experimental/dynamic/_exceptions.py Outdated

Add exception propagation for deferred and async execution

530be68

Signed-off-by: Rostan Tabet <rtabet@nvidia.com>

rostan-t force-pushed the ndd-better-exceptions branch from 87bbd68 to 530be68 Compare February 13, 2026 15:03

greptile-apps Bot reviewed Feb 13, 2026

View reviewed changes

Comment thread dali/python/nvidia/dali/experimental/dynamic/_invocation.py

Remove test_async.test_exception_propagation

7c1f784

Signed-off-by: Rostan Tabet <rtabet@nvidia.com>

klecki reviewed Feb 13, 2026

View reviewed changes

Move the stack capture to the invocation. Fix it for the batch2tensor op

ca63019

Signed-off-by: Rostan Tabet <rtabet@nvidia.com>

greptile-apps Bot reviewed Feb 16, 2026

View reviewed changes

Comment thread dali/python/nvidia/dali/experimental/dynamic/_exceptions.py

mzient reviewed Feb 16, 2026

View reviewed changes

rostan-t added 2 commits February 16, 2026 09:49

Add missing header in test_interop.py

187aa2f

Signed-off-by: Rostan Tabet <rtabet@nvidia.com>

Re-raise exceptions with the same type

baf619a

Signed-off-by: Rostan Tabet <rtabet@nvidia.com>

greptile-apps Bot reviewed Feb 16, 2026

View reviewed changes

dali-automaton assigned banasraf and klecki Feb 16, 2026

klecki approved these changes Feb 16, 2026

View reviewed changes

mzient reviewed Feb 16, 2026

View reviewed changes

mzient approved these changes Feb 16, 2026

View reviewed changes

rostan-t merged commit c443bf9 into NVIDIA:main Feb 16, 2026
7 checks passed

rostan-t deleted the ndd-better-exceptions branch February 16, 2026 17:09

Conversation

rostan-t commented Feb 13, 2026

Category:

Description:

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

Uh oh!

Uh oh!

greptile-apps Bot commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rostan-t commented Feb 13, 2026

Uh oh!

dali-automaton commented Feb 13, 2026

Uh oh!

dali-automaton commented Feb 13, 2026

Uh oh!

klecki Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

rostan-t Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

rostan-t Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rostan-t commented Feb 16, 2026

Uh oh!

dali-automaton commented Feb 16, 2026

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mzient Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rostan-t Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rostan-t Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

dali-automaton commented Feb 16, 2026

Uh oh!

rostan-t commented Feb 16, 2026

Uh oh!

dali-automaton commented Feb 16, 2026

Uh oh!

mzient Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

rostan-t Feb 16, 2026

Choose a reason for hiding this comment

greptile-apps Bot commented Feb 13, 2026 •

edited

Loading

rostan-t Feb 16, 2026 •

edited

Loading

mzient Feb 16, 2026 •

edited

Loading

rostan-t Feb 16, 2026 •

edited

Loading