bug: claude-cli provider crashes after ~4 tests in combined eval run (SessionStart hook error)

[christso]

Summary

When running multiple eval files in a single agentv eval run invocation against the claude-cli target, the provider crashes after ~4 tests with a SessionStart hook error. All subsequent tests fail with the same error.

Reproduction

# This fails after ~4 tests:
agentv eval run --target claude-cli --workers 1 "evals/hivespec/*.eval.yaml"

# This works — each invocation gets a fresh session:
agentv eval run --target claude-cli --workers 1 evals/hivespec/hs-claim.eval.yaml
agentv eval run --target claude-cli --workers 1 evals/hivespec/hs-explore.eval.yaml
# etc.

Error

{"type":"system","subtype":"hook_started","hook_id":"...","hook_name":"SessionStart:startup","hook_event":"SessionStart","session_id":"..."}

Claude CLI exits with code 1. The error is a raw JSON event from the session hook, not a proper error message.

Observed behavior

• Tests 1-4: all PASS (1.000)
• Tests 5-17: all ERROR with the same SessionStart hook failure
• When the same 17 tests are run as 5 separate invocations (one per eval file), all complete successfully

Expected behavior

All 17 tests should complete when run in a single invocation. The claude-cli provider should clean up sessions between tests and handle hook errors gracefully.

Investigation starting points

1. Claude CLI provider: `packages/core/src/evaluation/providers/` — find the claude-cli provider implementation
2. Check how sessions are created and torn down between test cases
3. Check if there's a session cleanup or process kill between tests
4. Compare with pi-cli provider which handles 17 sequential tests without issues

Acceptance signals

• `agentv eval run --target claude-cli --workers 1 "evals/hivespec/*.eval.yaml"` completes all 17 tests without SessionStart errors
• Each test gets a fresh Claude CLI session (no state leakage between tests)
• If a SessionStart hook fails, the provider retries with a fresh session before marking the test as an execution error

Non-goals

• Parallel Claude CLI sessions (workers > 1) — that's a separate concern
• Changing how Claude CLI hooks work — the fix should be in agentv's provider layer

[christso]

Verification attempt (2026-03-29)

Ran all 17 HiveSpec evals against claude-cli target with --workers 1 in a single invocation — no crash. All 17 tests completed (14 passed, 3 quality failures, 0.912 mean score).

Environment: Claude Code 2.1.87, Braintrust trace plugin 1.4.2, Linux arm64

The SessionStart hook error is not currently reproducible. This may have been fixed in a Claude Code or plugin update since the original report.

Current state of PR #840:

• --no-session-persistence — kept as a defensive measure (prevents session state leakage between tests)
• TRACE_TO_BRAINTRUST=false — kept as a defensive measure against the non-atomic cache write race condition, though the race was not triggered in this run

Upstream: Filed braintrustdata/braintrust-claude-plugin#19 with atomic write fix PR braintrustdata/braintrust-claude-plugin#20.

[christso]

Cannot reproduce — all 17 HiveSpec evals completed against claude-cli without crash (Claude Code 2.1.87, Braintrust plugin 1.4.2). Stripping defensive workaround from PR #840. Will reopen if it resurfaces.