perf: default integer arrays to int32 for ~25% memory reduction#566
Draft
FBumann wants to merge 5 commits intoPyPSA:masterfrom
Draft
perf: default integer arrays to int32 for ~25% memory reduction#566FBumann wants to merge 5 commits intoPyPSA:masterfrom
FBumann wants to merge 5 commits intoPyPSA:masterfrom
Conversation
linopy/constants.py — Added DEFAULT_LABEL_DTYPE = np.int32 linopy/model.py — Variable and constraint label assignment now uses np.arange(..., dtype=DEFAULT_LABEL_DTYPE) with overflow guards that raise ValueError if labels exceed int32 max. linopy/expressions.py — _term coord assignment and all .astype(int) for vars arrays now use DEFAULT_LABEL_DTYPE (int32). linopy/common.py — fill_missing_coords uses np.arange(..., dtype=DEFAULT_LABEL_DTYPE). Polars schema inference now checks array.dtype.itemsize instead of the old OS/numpy-version hack. test/test_constraints.py — Updated 2 dtype assertions to use np.issubdtype instead of == int. test/test_dtypes.py (new) — 7 tests covering int32 labels, expression vars, solve correctness, and overflow guards.
…k to int64 via astype(int), now use DEFAULT_LABEL_DTYPE. Also Variables.to_dataframe arange for map_labels. - linopy/constraints.py: Constraints.to_dataframe arange for map_labels. - linopy/common.py: save_join outer-join fallback was casting to int64.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes proposed in this Pull Request
Cut memory for internal integer arrays (labels, vars indices,
_termcoords) by ~25% and improve build speed by ~10-35% by defaulting toint32instead ofint64.What changed
linopy/constants.py: AddedDEFAULT_LABEL_DTYPE = np.int32linopy/model.py: Variable and constraint label assignment usesnp.arange(..., dtype=DEFAULT_LABEL_DTYPE)with overflow guard that raisesValueErrorif labels exceed int32 max (~2.1 billion)linopy/expressions.py:_termcoord assignment and.astype(int)for vars arrays now useDEFAULT_LABEL_DTYPElinopy/variables.py:ffill,bfill,sanitizeuseDEFAULT_LABEL_DTYPEinstead ofastype(int)(which widened labels back to int64);Variables.to_dataframearange uses int32linopy/constraints.py:Constraints.to_dataframearange usesDEFAULT_LABEL_DTYPElinopy/common.py:fill_missing_coordsuses int32 arange;save_joinouter-join fallback usesDEFAULT_LABEL_DTYPEinstead ofastype(int); polars schema infersInt32/Int64based on actual array dtypetest/test_constraints.py: Updated dtype assertions to usenp.issubdtype(compatible with both int32 and int64)test/test_dtypes.py(new): Tests for int32 labels, expression vars, solve correctness, and overflow guarddev-scripts/benchmark_lp_writer.py(new): Benchmark script supporting--phase memory|build|lp_writewith--plotcomparison modeBenchmark results
Reproduce with:
Memory (dataset
.nbytes)Consistent 1.25x reduction across all problem sizes (e.g. 640 MB → 512 MB at 8M vars). The
labelsandvarsarrays shrink 50% (int64 → int32) whilelower/upper/coeffs/rhsstay float64.Build speed
Consistently ~1.1-1.35x faster across all sizes (30 iterations with GC, tight error bars). 10-20% for large models (170ms → 153ms at 8M vars), and up to 35% for small/medium models where the fixed overhead of array allocation matters more relative to total time.
Similar results on real PyPSA model.
No influence on lp-write
Checklist
doc.doc/release_notes.rstof the upcoming release is included.