Skip to content

Commit 58df5ae

Browse files
sjarmakclaude
sjarmak
and
claude
committed
Add 14 onboard-search ground truth (381/381 complete) + remove skip filter
- Convert 14 ccx-onboard-search-* tasks from function_id schema to standard files/symbols format for IR evaluation (ground_truth.json preserved for verifier) - Promote oracle_answer_agent.json for all 14 tasks - Remove SKIP_TASK_PATTERNS for onboard-search in daytona_curator_runner.py - Total coverage: 381/381 tasks (160 SDLC + 221 Org) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 2e52ab4 commit 58df5ae

File tree

57 files changed

+1318
-3
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

57 files changed

+1318
-3
lines changed

‎benchmarks/csb_org_onboarding/ccx-onboard-search-201/tests/ground_truth_agent.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"files": [
3+
"pkg/scheduler/schedule_one.go"
4+
],
5+
"expected_edit_files": [],
6+
"symbols": [
7+
{
8+
"file": "pkg/scheduler/schedule_one.go",
9+
"symbol": "findNodesThatPassFilters",
10+
"repo": null
11+
}
12+
],
13+
"chunks": [
14+
{
15+
"file": "pkg/scheduler/schedule_one.go",
16+
"line_start": 1,
17+
"line_end": 50,
18+
"annotation": "Target function: findNodesThatPassFilters"
19+
}
20+
],
21+
"text": "Semantic function retrieval task. Target: pkg/scheduler/schedule_one.go::findNodesThatPassFilters"
22+
}

‎benchmarks/csb_org_onboarding/ccx-onboard-search-201/tests/ground_truth_meta.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"has_ground_truth": true,
3+
"has_chunk_ground_truth": true,
4+
"ground_truth_source": "converted_from_function_id_schema",
5+
"ground_truth_confidence": "high",
6+
"task_name": "ccx-onboard-search-201",
7+
"curator_agent_version": "2.0",
8+
"model": "manual",
9+
"backend": "manual",
10+
"timestamp": "2026-03-06T19:47:38.687273+00:00",
11+
"files_count": 1,
12+
"edit_files_count": 0,
13+
"chunks_count": 1,
14+
"symbols_count": 1,
15+
"cost_usd": 0
16+
}

‎benchmarks/csb_org_onboarding/ccx-onboard-search-201/tests/oracle_answer.json‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"files": [
3+
"pkg/scheduler/schedule_one.go"
4+
],
5+
"expected_edit_files": [],
6+
"symbols": [
7+
{
8+
"file": "pkg/scheduler/schedule_one.go",
9+
"symbol": "findNodesThatPassFilters",
10+
"repo": null
11+
}
12+
],
13+
"chunks": [
14+
{
15+
"file": "pkg/scheduler/schedule_one.go",
16+
"line_start": 1,
17+
"line_end": 50,
18+
"annotation": "Target function: findNodesThatPassFilters"
19+
}
20+
],
21+
"text": "Semantic function retrieval task. Target: pkg/scheduler/schedule_one.go::findNodesThatPassFilters",
22+
"_metadata": {
23+
"model": "manual",
24+
"backend": "manual",
25+
"ground_truth_source": "converted_from_function_id_schema",
26+
"timestamp": "2026-03-06T19:47:38.687166+00:00"
27+
}
28+
}

‎benchmarks/csb_org_onboarding/ccx-onboard-search-201/tests/oracle_answer_agent.json‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"files": [
3+
"pkg/scheduler/schedule_one.go"
4+
],
5+
"expected_edit_files": [],
6+
"symbols": [
7+
{
8+
"file": "pkg/scheduler/schedule_one.go",
9+
"symbol": "findNodesThatPassFilters",
10+
"repo": null
11+
}
12+
],
13+
"chunks": [
14+
{
15+
"file": "pkg/scheduler/schedule_one.go",
16+
"line_start": 1,
17+
"line_end": 50,
18+
"annotation": "Target function: findNodesThatPassFilters"
19+
}
20+
],
21+
"text": "Semantic function retrieval task. Target: pkg/scheduler/schedule_one.go::findNodesThatPassFilters",
22+
"_metadata": {
23+
"model": "manual",
24+
"backend": "manual",
25+
"ground_truth_source": "converted_from_function_id_schema",
26+
"timestamp": "2026-03-06T19:47:38.687166+00:00"
27+
}
28+
}

‎benchmarks/csb_org_onboarding/ccx-onboard-search-202/tests/ground_truth_agent.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"files": [
3+
"pkg/kubelet/eviction/eviction_manager.go"
4+
],
5+
"expected_edit_files": [],
6+
"symbols": [
7+
{
8+
"file": "pkg/kubelet/eviction/eviction_manager.go",
9+
"symbol": "synchronize",
10+
"repo": null
11+
}
12+
],
13+
"chunks": [
14+
{
15+
"file": "pkg/kubelet/eviction/eviction_manager.go",
16+
"line_start": 1,
17+
"line_end": 50,
18+
"annotation": "Target function: synchronize"
19+
}
20+
],
21+
"text": "Semantic function retrieval task. Target: pkg/kubelet/eviction/eviction_manager.go::synchronize"
22+
}

‎benchmarks/csb_org_onboarding/ccx-onboard-search-202/tests/ground_truth_meta.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"has_ground_truth": true,
3+
"has_chunk_ground_truth": true,
4+
"ground_truth_source": "converted_from_function_id_schema",
5+
"ground_truth_confidence": "high",
6+
"task_name": "ccx-onboard-search-202",
7+
"curator_agent_version": "2.0",
8+
"model": "manual",
9+
"backend": "manual",
10+
"timestamp": "2026-03-06T19:47:38.687549+00:00",
11+
"files_count": 1,
12+
"edit_files_count": 0,
13+
"chunks_count": 1,
14+
"symbols_count": 1,
15+
"cost_usd": 0
16+
}

‎benchmarks/csb_org_onboarding/ccx-onboard-search-202/tests/oracle_answer.json‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"files": [
3+
"pkg/kubelet/eviction/eviction_manager.go"
4+
],
5+
"expected_edit_files": [],
6+
"symbols": [
7+
{
8+
"file": "pkg/kubelet/eviction/eviction_manager.go",
9+
"symbol": "synchronize",
10+
"repo": null
11+
}
12+
],
13+
"chunks": [
14+
{
15+
"file": "pkg/kubelet/eviction/eviction_manager.go",
16+
"line_start": 1,
17+
"line_end": 50,
18+
"annotation": "Target function: synchronize"
19+
}
20+
],
21+
"text": "Semantic function retrieval task. Target: pkg/kubelet/eviction/eviction_manager.go::synchronize",
22+
"_metadata": {
23+
"model": "manual",
24+
"backend": "manual",
25+
"ground_truth_source": "converted_from_function_id_schema",
26+
"timestamp": "2026-03-06T19:47:38.687467+00:00"
27+
}
28+
}

‎benchmarks/csb_org_onboarding/ccx-onboard-search-202/tests/oracle_answer_agent.json‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"files": [
3+
"pkg/kubelet/eviction/eviction_manager.go"
4+
],
5+
"expected_edit_files": [],
6+
"symbols": [
7+
{
8+
"file": "pkg/kubelet/eviction/eviction_manager.go",
9+
"symbol": "synchronize",
10+
"repo": null
11+
}
12+
],
13+
"chunks": [
14+
{
15+
"file": "pkg/kubelet/eviction/eviction_manager.go",
16+
"line_start": 1,
17+
"line_end": 50,
18+
"annotation": "Target function: synchronize"
19+
}
20+
],
21+
"text": "Semantic function retrieval task. Target: pkg/kubelet/eviction/eviction_manager.go::synchronize",
22+
"_metadata": {
23+
"model": "manual",
24+
"backend": "manual",
25+
"ground_truth_source": "converted_from_function_id_schema",
26+
"timestamp": "2026-03-06T19:47:38.687467+00:00"
27+
}
28+
}

‎benchmarks/csb_org_onboarding/ccx-onboard-search-203/tests/ground_truth_agent.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"files": [
3+
"clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java"
4+
],
5+
"expected_edit_files": [],
6+
"symbols": [
7+
{
8+
"file": "clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java",
9+
"symbol": "drainBatchesForOneNode",
10+
"repo": null
11+
}
12+
],
13+
"chunks": [
14+
{
15+
"file": "clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java",
16+
"line_start": 1,
17+
"line_end": 50,
18+
"annotation": "Target function: drainBatchesForOneNode"
19+
}
20+
],
21+
"text": "Semantic function retrieval task. Target: clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java::drainBatchesForOneNode"
22+
}

‎benchmarks/csb_org_onboarding/ccx-onboard-search-203/tests/ground_truth_meta.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"has_ground_truth": true,
3+
"has_chunk_ground_truth": true,
4+
"ground_truth_source": "converted_from_function_id_schema",
5+
"ground_truth_confidence": "high",
6+
"task_name": "ccx-onboard-search-203",
7+
"curator_agent_version": "2.0",
8+
"model": "manual",
9+
"backend": "manual",
10+
"timestamp": "2026-03-06T19:47:38.687819+00:00",
11+
"files_count": 1,
12+
"edit_files_count": 0,
13+
"chunks_count": 1,
14+
"symbols_count": 1,
15+
"cost_usd": 0
16+
}

0 commit comments

Comments
 (0)