Skip to content

[DRAFT][STACKED on #206] Add Dep marker for explicit container dependencies in @Flow.model#213

Draft
NeejWeej wants to merge 16 commits into
Point72:mainfrom
NeejWeej:nk/explicit-deps-containers
Draft

[DRAFT][STACKED on #206] Add Dep marker for explicit container dependencies in @Flow.model#213
NeejWeej wants to merge 16 commits into
Point72:mainfrom
NeejWeej:nk/explicit-deps-containers

Conversation

@NeejWeej
Copy link
Copy Markdown

PR Summary: nk/explicit-deps-containers

Stacked PR Note

This builds on the @Flow.model PR: #206.
That PR should be reviewed/merged first. This PR has to target main
because GitHub cannot target the fork-only parent branch, please review this as
the follow-on Dep[...] container-dependency layer rather than re-reviewing
the base @Flow.model work.

TL;DR

This lets a flow consume a dynamic collection of upstream model outputs without
hard-coding every branch into the model signature. A fan-out step can produce
many sibling models, and a fan-in step can accept them as list[Dep[T]],
tuple[...], or dict[..., Dep[T]] while still receiving plain resolved
values inside the function. The marker keeps that power explicit: normal
containers stay literal unless the exact nested dependency slot is marked.

Tiny example:

@Flow.model
def load_one(shard: str, date: FromContext[str]) -> int:
    ...


@Flow.model
def combine(rows: list[Dep[int]]) -> int:
    return sum(rows)


def build_pipeline(shards: list[str]):
    # Fan-out: build one upstream model per shard.
    branch_models = [load_one(shard=shard) for shard in shards]

    # Fan-in: pass the branch models as one regular container input.
    return combine(rows=branch_models)

Without Dep, combine(rows=branch_models) would be rejected because regular
containers stay literal by default. With list[Dep[int]], each list item may be
either a literal int or a model producing an int, and combine() still sees
plain list[int] at execution time.

What Changed

This PR adds Dep[T], a narrow @Flow.model annotation marker for explicit
dependency leaves inside regular container inputs.

Regular parameters already accepted either a literal value or a direct
CallableModel supplying the whole parameter. That behavior is unchanged.
Dep[T] only adds marked nested slots inside literal containers:

from ccflow import Dep, Flow, FromContext


@Flow.model
def source(value: FromContext[int], offset: int) -> int:
    return value + offset


@Flow.model
def total(values: list[Dep[int]]) -> int:
    return sum(values)


model = total(values=[source(offset=1), 2, 3])
assert model.flow.compute(value=10).value == 16

The function body receives list[int]; generated @Flow.model code resolves
the marked model leaves before calling the function.

Semantics

  • list[int] still accepts a literal list[int] or a direct model returning
    list[int].
  • list[Dep[int]] additionally accepts model leaves inside the literal list.
  • Dep[T] means "literal T or CallableModel whose unwrapped result
    validates as T" at that exact annotation position.
  • Dep[...] is interpreted only by generated @Flow.model regular parameters.
  • Dep[...] is supported inside list, tuple, and dict values.
  • Top-level Dep[...] is rejected because direct whole-parameter dependencies
    already cover that case.
  • Union annotations may appear inside the marked slot, such as
    list[Dep[int | None]].
  • Union annotations may not wrap a Dep marker, including optional container
    forms like list[Dep[int]] | None.
  • Dict keys cannot be Dep[...].
  • Nested Dep[...] markers are rejected.
  • Dep[...] does not add automatic behavior to handwritten CallableModel
    fields; there it behaves like ordinary Annotated metadata.

Why This Shape

This keeps automatic dependency discovery simple and explicit. We do not scan
every container for models. Instead, users mark the exact nested positions where
model leaves are allowed. That preserves existing literal-container behavior
while supporting generic fan-in patterns such as lists of branch results.

Implementation Notes

  • Adds Dep as an Annotated[T, _DepMarker()] marker.
  • Extends generated model validation, execution, __deps__, and effective
    identity to walk only marked container positions.
  • Keeps registry alias fallback and serialized-model fallback available at
    marked dependency leaves.
  • Preserves other Annotated metadata when removing the Dep marker, so
    constraints such as Field(gt=0) still apply to literals and dependency
    results.
  • Updates Flow model examples to show explicit Dep container inputs.

Tests

Covered by ccflow/tests/test_flow_model.py:

  • marked model leaves inside list[Dep[int]] and list[Dep[list[int]]];
  • existing whole-parameter dependency behavior;
  • registry aliases and serialized model references at marked leaves;
  • pickle/cloudpickle and JSON/BaseModel roundtrips for models with marked
    dependency leaves;
  • graph dependency discovery and effective cache identity;
  • preservation of non-Dep Annotated metadata;
  • rejection of unmarked nested models, top-level Dep, nested Dep, dict-key
    Dep, and set usage;
  • handwritten CallableModel fields ignoring Dep as normal annotation
    metadata.

NeejWeej added 14 commits May 4, 2026 04:40
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
… @Flow.model

Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
…-containers

Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 17, 2026

Codecov Report

❌ Patch coverage is 86.71875% with 187 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.81%. Comparing base (3c8fd19) to head (6938b95).

Files with missing lines Patch % Lines
...amples/flow_model/flow_model_hydra_builder_demo.py 0.00% 65 Missing ⚠️
ccflow/examples/flow_model/flow_model_example.py 0.00% 57 Missing ⚠️
ccflow/_flow_model_binding.py 93.49% 20 Missing and 12 partials ⚠️
ccflow/tests/test_callable.py 91.01% 15 Missing ⚠️
ccflow/tests/test_flow_context.py 97.61% 7 Missing ⚠️
ccflow/evaluators/common.py 93.02% 4 Missing and 2 partials ⚠️
ccflow/context.py 93.10% 1 Missing and 1 partial ⚠️
ccflow/tests/flow_model_hydra_fixtures.py 91.30% 1 Missing and 1 partial ⚠️
ccflow/tests/evaluators/test_common.py 98.70% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #213      +/-   ##
==========================================
- Coverage   95.37%   93.81%   -1.56%     
==========================================
  Files         142      151       +9     
  Lines       11404    17614    +6210     
  Branches      620     1156     +536     
==========================================
+ Hits        10876    16525    +5649     
- Misses        399      865     +466     
- Partials      129      224      +95     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
@NeejWeej NeejWeej force-pushed the nk/explicit-deps-containers branch from 85c61fa to 7b76d13 Compare May 17, 2026 23:19
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
@NeejWeej NeejWeej force-pushed the nk/explicit-deps-containers branch from f864ab1 to 6938b95 Compare May 18, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant