Skip to content

Commit f72717a

Browse files
sjarmakclaude
andcommitted
fix: curl-security-review-001 MCP verifier — wrong mirror URL in clone manifest
The Dockerfile.sg_only clone manifest had "sg-evals/curl/curl" (invalid 3-segment GitHub path) instead of "sg-evals/curl--09e25b9d" (the actual mirror). This caused the clone-at-verify step to fail, inject_defects.sh to crash on the empty workspace, and set -e to propagate the failure — killing test.sh before writing reward.txt. Every MCP run hit RewardFileNotFoundError. Validated fix: MCP rerun scores 0.51. Also added defensive error handling around inject_defects.sh in the verifier wrapper so future clone failures degrade gracefully instead of crashing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 87f1a5d commit f72717a

File tree

2 files changed

+6
-3
lines changed

2 files changed

+6
-3
lines changed

benchmarks/ccb_test/curl-security-review-001/environment/Dockerfile.sg_only

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ RUN chmod +x /tmp/inject_defects.sh
3636
RUN mkdir -p /workspace/tests /logs/verifier /logs/agent
3737

3838
# Clone manifest for verifier (clone-at-verify strategy)
39-
RUN echo '{"workdir":"/workspace","repos":[{"mirror":"sg-evals/curl/curl","target_dir":"."}],"inject_defects":"/tmp/inject_defects.sh"}' > /tmp/.sg_only_clone_manifest.json
39+
RUN echo '{"workdir":"/workspace","repos":[{"mirror":"sg-evals/curl--09e25b9d","target_dir":"."}],"inject_defects":"/tmp/inject_defects.sh"}' > /tmp/.sg_only_clone_manifest.json
4040

4141
# Mark sg_only mode
4242
RUN touch /tmp/.sg_only_mode

benchmarks/ccb_test/curl-security-review-001/tests/sgonly_verifier_wrapper.sh

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -151,8 +151,11 @@ if [ -f "$MANIFEST" ]; then
151151
echo "[sg_only_verifier] Running defect injection: $INJECT_SCRIPT"
152152
cd "$WORKDIR"
153153
chmod +x "$INJECT_SCRIPT"
154-
bash "$INJECT_SCRIPT"
155-
echo "[sg_only_verifier] Defect injection complete"
154+
if bash "$INJECT_SCRIPT"; then
155+
echo "[sg_only_verifier] Defect injection complete"
156+
else
157+
echo "[sg_only_verifier] WARNING: Defect injection failed (exit $?) — scoring will proceed with degraded accuracy"
158+
fi
156159
fi
157160

158161
# 4. Overlay agent changes

0 commit comments

Comments
 (0)