feat: v1 semantic convention

[FBumann]

Draft PR

The strict v1 semantic convention for linopy — predictable coordinate alignment and NaN handling.

This is the master PR for the new semantic convention in linopy. It starts with our Design & transitioning goals, which is carried out in our New Semantics spec. Both files are tracked in this branch. WHat you read is the current state.

Both might change until this PR is merged

Scope

The convention ships behind linopy.options["semantic_convention"] — v1 opt-in, legacy the default. This PR carries the design, spec, tests and implementation; documentation notebooks follow separately.

Testing

All tests in linopy will be executed for both semantics.
Differing behaviour will be tested using pytest.markers.

This will increase ci time temporarily until v1 is released.

Defered to a follow up PR

Docs: Documentation, Migration guide, Release notes

[FBumann]

Coming next, in order:

• Slice F — §11 auxiliary-coordinate conflicts. Raise on non-dimension-coord conflict during alignment (covers Auxiliary non-dimension coordinates leak into expressions and break alignment #295). Small.
• Slice G — §13 reductions audit. Most reductions already use skipna=True; need a focused audit + tests for sum / mean / groupby / resample / coarsen and the objective. Likely mostly tests.
• Slice P — linopy/piecewise.py and linopy/sos_reformulation.py. Internal callers build expressions by .isel(...)-slicing along a piece dim and comparing the two slices (delta_hi <= delta_lo); the slices share the dim with different coords, which v1 §8 rejects. Until this lands, test_piecewise_constraints.py + test_piecewise_feasibility.py carry a module-level pytestmark = pytest.mark.legacy and two SOS2 tests are method-marked legacy. Fixing piecewise removes those marks.

Then a final pass on docs (user-facing migration / rollout — deferred so far).

[MaykThewessen]

Thanks @FBumann — read goals.md + convention.md + the [#714] catalogue. Priority ordering (no-silent-wrong-answers > algebraic-laws > absence-first-class > least-surprise) reads exactly right for an optimisation library. Notes from the PyPSA-Eur side, ordered by leverage:

Highest leverage: lock the §5 / [#627] question before v1 GAs

§5 ("user NaN raises") is the safe default, but if [#627] later flips to "user NaN means absent", code written defensively for strict-v1 — peppering fillna(...) at construction sites — will quietly behave differently than the same code under v1-with-#627. Two semantic flips inside the v1 lifecycle costs trust. Worth deciding §5 before v1 promotes to default, even at the cost of slipping the schedule. My vote: keep raise — in PyPSA-Eur, NaN in user data is almost always an index-mismatch bug, and goal 1 is the whole point.

Biggest migration cost is §8, not §5

PyPSA-Eur builds expressions against n.snapshots[mask] subsets and currently relies on size-coincidence positional behaviour. To make the legacy → v1 step survivable on a 50k-LOC codebase, the warning needs to identify the exact mismatch: (dim, missing_labels[:5]), not just "shared dim mismatch". And — bigger ask — a linopy.diagnose_alignment(model) helper that walks every stored expression and lists every site v1 would raise, runnable under legacy, would let users patch one site at a time before flipping the switch.

Happy to PR this helper once the spec stabilises.

§10 needs an operator-syntax escape hatch

join= lives on named methods only. A scoped context manager — with linopy.alignment(join="override"): ... — would let people refactor incrementally without rewriting every a + b into a.add(b, join="override"). Especially useful where the positional override is intentional (some legacy PyPSA stats code). Scoped, not global; threadlocal.

Happy to PR this too.

Spec gaps worth nailing down

• §13 — mean over an all-absent dim is 0/0. Pick one: result is absent (NaN propagates) or raise. My vote: absent, consistent with §6's "absence absorbs".
• §5 — does the rule cover bounds and RHS, not just coeffs / const? add_variables(lower=series_with_nan) today silently broadcasts. Under v1 I assume it raises; spec should say so. Same for constraint RHS.
• §11 — error message on aux-coord conflict should print both values (or .head() thereof), not just the coord name. We've been bitten by [Auxiliary non-dimension coordinates leak into expressions and break alignment #295] in PyPSA-Eur post-processing where a carrier aux coord silently disappears after expr.sum("snapshot"); the diagnostic matters.

On my #715

Once v1 is opt-in, the six xfail-strict tests flip to XPASS under options["semantic_convention"]="v1". Happy to parametrize the file over legacy / v1 so it tracks the rollout schedule rather than only gating on the eventual default — turns #715 into a v1 conformance harness.

[MaykThewessen]

Lovely commit @FBumann — two upgrades I didn't ask for that I should have:

• Per-op message split (+/- → 0, * → 0, / → 1) puts the NaN in user-supplied data is silently filled instead of surfaced #713 asymmetric fill in the warning text. Users can grep CI logs by op_kind.
• _legacy_masked_variable_message in Variable.to_linexpr catches the 2 * x + y (masked y, no NaN constant) case where no other warn site fires. That's the most common PyPSA-Eur divergence and would have migrated silently. Excellent.

Three follow-up suggestions, ordered by leverage:

1. Include (dim, missing_labels[:5]) in coord-mismatch text. _legacy_coord_mismatch_message("merge along dim 'time'") names the dim but not the diff. In a large codebase the dim alone isn't enough to locate the wrong reindex. The _shared_dim_mismatch_message v1-raise formatter already prints both sides — could share it between v1-raise and legacy-warn so the strings stay aligned.
2. Stacklevel sanity check. Current mix is 3/4/5 across call sites. Worth a test that pins the user-visible frame to the caller's source, not linopy internals — IDE jump-to-source on the warning depends on it.
3. diagnose_alignment(model) audit helper still distinct from warnings. Warnings fire only when ops run; the helper walks every stored expression in a built model and lists every v1-raise site, runnable under legacy. Different shape — lets users audit before opting in. Happy to PR if you want it; or if you'd rather own it, glad to leave it to you.

[FBumann]

@MaykThewessen Im not sure how such a diagnose_alignment(model) would work, as it might not be clear from a build constraint if the creation path that lead to it would have produced a different constraint in the new semantic convention.
However, feel free to PR this.

[MaykThewessen]

Full line-level pass on the implementation files (semantics.py, expressions.py, variables.py, config.py, piecewise.py, sos_reformulation.py) against the §1–§13 spec in convention.md. The spec ↔ code alignment is strong overall — almost every section has a direct implementation site, and the is_v1() switch is consistently placed. Findings below are not blockers to opt-in landing, but most should resolve before v1 promotes to default, because they constitute silent-divergence paths the rollout warnings cannot catch.

High-impact (silent §-violations under v1)

1. §8 / §11 enforcement gated behind if join is None: in both merge (line 2573) and _align_constant (line 580). User-supplied join= bypasses aux-coord conflict check — the #295 bug v1 is supposed to close still reproduces with join="override".
2. §6 absence propagation gap on quadratics (line 2633–2634): the FACTOR_DIM branch's .prod(...) uses xarray default skipna=True. The TERM_DIM branch right above correctly does skipna = not is_v1(). expr * expr with one absent factor silently collapses absence to identity 1.
3. §5 not enforced on Variable.to_linexpr(coefficient) (variables.py:312): array coefficients carrying user NaN flow through with no check_user_nan_array call.
4. §1/§2 storage invariant not constructor-enforced (expressions.py:1328 / __init__): no absorb_absence call inside LinearExpression.__init__. A bypass-constructed expression can carry "half-absent" rows; isnull() then misreports.

Migration ergonomics

5. stacklevel lands inside linopy internals on every call site I traced (semantics.py:170 default =3, lines 200 / 207 default =5). IDE jump-to-source on the legacy warnings points at __add__/merge/to_linexpr, not at the user's call site — defeats the actionable-warning goal #2.
6. Legacy RHS-NaN warning collapses coord-mismatch and data-error into one message (expressions.py:1292). User gets fillna(...) advice when the underlying cause was a reindex artifact.
7. override join semantic drift (line 609): docs and _shared_dim_mismatch_message advertise "positional alignment, made explicit". Implementation is assign_coords, which is coord-renaming with no shape guarantee.

Robustness

8. Brittle xarray exception string-match (line 624): if "exact" in str(e): will silently break on any upstream xarray rephrase. Structural check via merge_shared_user_coord_mismatch is safer.
9. op_kind derived from fill_value magic value (line 749) — fragile coupling between numeric fill and message routing.

Spec/doc gaps

10. §11 asymmetric-presence behaviour (semantics.py:255) is implemented (propagate from the one operand that has it) but not stated in convention.md §11. One sentence closes it.
11. §13 / mean of all-absent dim still open — I raised this above; no impl decision visible yet.

Performance (deferrable)

The _align_constant / merge hot paths run aux-coord checks on every v1 binary op without identity short-circuits. Likely measurable in PyPSA-Eur-scale models (thousands of add_constraints calls). Cheap to add if a is b: return False before the equals-check. Worth a profile pass once functionality is locked.

What's working well

• The §12 raise plumbing for to_constraint is clean — self.sub(rhs, join=join) reuses the §8 path correctly.
• piecewise.py / sos_reformulation.py minimal drop=True / assign_coords edits are correct under both semantics; no findings.
• Per-op message split + masked-variable warn site (commit 9455683) catches the most common PyPSA-Eur divergence at its origin. That's the highest-leverage migration affordance in the PR.

Happy to PR fixes for any subset (especially #1, #2, #3, #5, #8) once the design questions above are settled. Treat all line comments as suggestions, not blockers.

[MaykThewessen]

override semantics drift. _shared_dim_mismatch_message (and convention.md §10) document override as "positional alignment, made explicit". The implementation here is other.assign_coords(coords=self.coords) — that's coord-renaming, not positional alignment, and has no shape/size guarantee.

If other.sizes != self.const.sizes on a shared dim, assign_coords will silently broadcast (or raise an opaque xarray error). The legacy positional path at least gated on other.sizes == self.const.sizes (line ~593) before doing the rename. The v1 override branch should either keep that gate or rename itself to relabel/coerce to be honest about what it does.

[MaykThewessen]

You're right, @FBumann — diagnose_alignment(model) walking the built artifact is the wrong shape. Once a constraint is in model.constraints, the legacy-vs-v1 divergence has already collapsed: a silently-positionally-aligned x + y and an explicitly label-aligned one produce identical Datasets. The build path is the only place the difference is visible.

Pivot proposal: capture at op time, not audit time. The warnings from commit 9455683 already fire at every divergence site — turn them into a structured trace rather than stderr lines.

with linopy.alignment_trace() as trace:
    m = build_my_model()

trace.report()   # pandas.DataFrame
#   file              line  op_kind  divergence              dim    n_unmatched
#   pypsa/optimize.py  412   add      coord_mismatch          time   17
#   pypsa/optimize.py  588   div      nan_user_fill           -      3
#   my_constraints.py  102   mul      masked_var_arithmetic   bus    2

Implementation sketch: context manager installs a warnings.catch_warnings() filter for LinopySemanticsWarning, parses each into a structured record (each _legacy_*_message already has a canonical shape — divergence class, op_kind, dim, fix-hint). On exit, returns a Trace object with .to_df() / .summarise() / .first_n_per_site(n=5).

Two payoffs over raw warnings:

• Deduplicates: 50,000 identical warnings from a tight loop collapse to one row with count.
• Survives filterwarnings('once') / 'ignore' — runs under its own catch context, independent of the user's global warning config.

Net: it's the warning machinery you already shipped, with structured capture and reporting on top. No new emission sites, no instrumentation in the hot path. PR-ready once the legacy-warn message format stabilises.

I'll open the PR once the spec-drift items from my review are settled — they affect what divergence classes the trace needs to recognise (e.g. if _align_constant's override semantics get tightened, that's a new divergence class to surface).

[FBumann]

@MaykThewessen Seems like a usefull addition!

[FBumann]

@MaykThewessen
Feel free to PR a linopy.alignment_trace() context manager. Although if its not to complex i would prefer to have it as a documented helper in the docs instead of inside the codebase. But only if thats possible...

Also feel free to PR a context manager like 'with linopy.alignment(join="override")', although i would argue that you should wait until this PR is reviewed and merged.

I would defer performance polishes like the proposed short-circuits until we merge this, maybe even until we drop the legacy convention. THis will keep the convention more focused and readable as long as its not 100% proven.