feat(config): add checkpoint_on_improvement option to save only on new best#461
Open
mvanhorn wants to merge 1 commit intoalgorithmicsuperintelligence:mainfrom
Open
Conversation
…w best Adds a new config flag, checkpoint_on_improvement, that triggers a checkpoint callback whenever a new-best program is found, in addition to the existing checkpoint_interval gate. Default False preserves existing behavior. When the optimization is making slow progress, checkpoint_interval saves work that the next interval would overwrite without any new best. This option lets users say 'only checkpoint when there's actually something new to save.' Wires the flag through the worker config dict (process_parallel.py:386) and adds a unit test that verifies the callback fires for a new-best run and not for a not-best run when checkpoint_interval is set high. Fixes algorithmicsuperintelligence#434
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Optimization runs that span many iterations without improvement currently produce checkpoints anyway, since
checkpoint_intervalfires on a fixed cadence. This PR adds an opt-incheckpoint_on_improvementconfig flag that triggers the checkpoint callback when a new best program is found, in addition to the existing interval-based trigger.Why this matters
enhancement, help wanted). Reporter notes that interval-only checkpointing wastes I/O when slow-progress runs save the same best program over and over.Changes
openevolve/config.py:407addscheckpoint_on_improvement: bool = Falsenext tocheckpoint_interval. Default preserves existing behavior.openevolve/process_parallel.py:387wires the new field into the worker config dict so worker processes see it.openevolve/process_parallel.py:670splits the trigger intointerval_hitandimprovement_hitand fires the checkpoint callback when either is true. Logs distinguish "Checkpoint interval reached" vs "Checkpointing new best solution".tests/test_process_parallel.py:156addstest_checkpoint_on_improvement_only_fires_for_new_best. Setscheckpoint_on_improvement=Trueandcheckpoint_interval=10000so only the improvement path can fire, then drives one new-best iteration and one not-best iteration. Assertscheckpoint_calls == [1].Testing
Full test suite passes locally on Python 3.10 via uv:
Default
checkpoint_on_improvement=Falsekeeps existing runs identical; the new path only activates when users opt in.Fixes #434
AI was used for assistance.