feat: modified SWEBench modal harness to use modal experimental snapshots and processing TestOutput#665
Open
clee-codegen wants to merge 23 commits intodevelopfrom
Open
feat: modified SWEBench modal harness to use modal experimental snapshots and processing TestOutput#665clee-codegen wants to merge 23 commits intodevelopfrom
clee-codegen wants to merge 23 commits intodevelopfrom
Conversation
clee-codegen
force-pushed
the
swebench-sandbox-snapshots
branch
from
8444ca6 to
7a46611
Compare
clee-codegen
changed the title
clee-codegen
force-pushed
the
swebench-sandbox-snapshots
branch
from
7a46611 to
7cf4f81
Compare
clee-codegen
force-pushed
the
swebench-sandbox-snapshots
branch
from
2195fa5 to
b5f1828
Compare
Contributor
jemeza-codegen
left a comment
jemeza-codegen
left a comment
There was a problem hiding this comment.
Looks like a good start!
codegen-examples/examples/swebench_agent_run/run_swebench_modal_harness.py
Outdated
Show resolved
Hide resolved
codegen-examples/examples/swebench_agent_run/snapshot_manager.py
Outdated
Show resolved
Hide resolved
| raise NotImplementedError("Not implemented") | ||
|
|
||
|
|
||
| class VolumeSnapshotManager(SnapshotManager): |
Contributor
There was a problem hiding this comment.
would we still need this?
Contributor
Author
There was a problem hiding this comment.
Nah, we'd only use the ModalDict implementation. I left in here if for whatever reason modal Dict didn't work out. I saw in the documentation that its not a very stable dict in that it could get lost.
Codecov ReportAll modified and coverable lines are covered by tests ✅ ✅ All tests successful. No failed tests found. Additional details and impacted files |
# Motivation The **Codegen on OSS** package provides a pipeline that: - **Collects repository URLs** from different sources (e.g., CSV files or GitHub searches). - **Parses repositories** using the codegen tool. - **Profiles performance** and logs metrics for each parsing run. - **Logs errors** to help pinpoint parsing failures or performance bottlenecks. <!-- Why is this change necessary? --> # Content <!-- Please include a summary of the change --> see [codegen-on-oss/README.md](https://github.com/codegen-sh/codegen-sdk/blob/acfe3dc07b65670af33b977fa1e7bc8627fd714e/codegen-on-oss/README.md) # Testing <!-- How was the change tested? --> `uv run modal run modal_run.py` No unit tests yet 😿 # Please check the following before marking your PR as ready for review - [ ] I have added tests for my changes - [x] I have updated the documentation or added new documentation as needed
clee-codegen
force-pushed
the
swebench-sandbox-snapshots
branch
from
b78b6b5 to
01236e5
Compare
clee-codegen
force-pushed
the
swebench-sandbox-snapshots
branch
from
0cb107a to
60fed54
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Content
Testing
Please check the following before marking your PR as ready for review