depletion_results.h5 storage efficiency; float32, gzip#3896
Open
yrrepy wants to merge 4 commits intoopenmc-dev:developfrom
Open
depletion_results.h5 storage efficiency; float32, gzip#3896yrrepy wants to merge 4 commits intoopenmc-dev:developfrom
yrrepy wants to merge 4 commits intoopenmc-dev:developfrom
Conversation
Thread hdf5_dtype ('float32' or 'float64', default 'float64') from
Integrator/SIIntegrator through StepResult.save() to the HDF5 writer.
Affects only the saved number and reaction rate datasets.
np.asarray(n0, dtype=np.float64) replaces n0.copy() so the working vector is float64 regardless of input dtype. Guards against silent precision loss on restart when initial compositions are read from a float32 HDF5 dataset
yrrepy
changed the title
Allow optional gzip or lzf compression on number and reaction rate datasets in depletion_results.h5. Default is None (no compression) for backward compatibility. Compression is applied at write time via h5py's chunk-level filters — readers auto-decompress with no code changes. - Thread hdf5_compression through Integrator, SIIntegrator, and StepResult.save() to create_dataset() calls - Validate at construction time to fail fast before transport solve - Propagate hdf5_dtype and hdf5_compression in distribute()
yrrepy
changed the title
Contributor
Author
|
Was going to do the gzip as a follow-up but recanted and thought better bundled. The HDF5 gzip internally compresses number and reaction rate datasets. For a modest mesh of:
|
yrrepy
changed the title
yrrepy
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This commit enables the user to choose float32 as the dtype for number and reaction rates datasets saved to the depletion_results.h5 file. It is optional and defaults to float64.
integrator.integrate(path='results.h5', write_rates=True, hdf5_dtype='float32')The intent is to reduce disk space used by depletion files and/or enable greater fidelity in calculations.
Essentially, at fixed file size it allows greater fidelity in the mesh (or reactions or nuclides, etc), in lieu of precision on the reaction rates and atom density.
e.g. for fixed chain, nuclides, timesteps, etc and fixed h5 file size, by going from float64 to float32 a user can now have 200,000 depleted materials instead of 100,000 (or 200,000 mesh voxels instead of 100,000). A significant boon to geometry fidelity.
per Claude;
XS have approx. 0.1-10% uncertainty, these uncertainties mean the underlying physics supports at most 2-4 significant digits. float32 provides ~7 significant digits.
The CRAM solver itself still operates in float64, and a guard has been added for a route that would have allowed float32 to get in.
Eigenvalues, time, source_rate, and depletion_time datasets remain float64 regardless of this setting.
Checklist