Skip to content

Commit 26f793e

Browse files
sjarmakclaude
sjarmak
and
claude
committed
chore: commit 13 ground truth files from sequential curator run
Phase1 prompt batch (sequential, killed at 18/74 to switch to Daytona). All use Opus 4.6 hybrid backend, avg $0.66/task. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent f3657e6 commit 26f793e

File tree

26 files changed

+740
-0
lines changed

26 files changed

+740
-0
lines changed

‎benchmarks/csb_sdlc_fix/flipt-cockroachdb-backend-fix-001/tests/ground_truth.json‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
{
2+
"files": [
3+
"internal/config/database.go",
4+
"internal/storage/sql/db.go",
5+
"internal/storage/sql/migrator.go",
6+
"cmd/flipt/main.go",
7+
"internal/storage/sql/postgres/postgres.go",
8+
"go.mod"
9+
],
10+
"symbols": [
11+
{
12+
"file": "internal/config/database.go",
13+
"symbol": "DatabaseProtocol",
14+
"repo": null
15+
},
16+
{
17+
"file": "internal/config/database.go",
18+
"symbol": "DatabasePostgres",
19+
"repo": null
20+
},
21+
{
22+
"file": "internal/config/database.go",
23+
"symbol": "databaseProtocolToString",
24+
"repo": null
25+
},
26+
{
27+
"file": "internal/config/database.go",
28+
"symbol": "stringToDatabaseProtocol",
29+
"repo": null
30+
},
31+
{
32+
"file": "internal/storage/sql/db.go",
33+
"symbol": "Driver",
34+
"repo": null
35+
},
36+
{
37+
"file": "internal/storage/sql/db.go",
38+
"symbol": "Postgres",
39+
"repo": null
40+
},
41+
{
42+
"file": "internal/storage/sql/db.go",
43+
"symbol": "driverToString",
44+
"repo": null
45+
},
46+
{
47+
"file": "internal/storage/sql/db.go",
48+
"symbol": "stringToDriver",
49+
"repo": null
50+
},
51+
{
52+
"file": "internal/storage/sql/db.go",
53+
"symbol": "open",
54+
"repo": null
55+
},
56+
{
57+
"file": "internal/storage/sql/db.go",
58+
"symbol": "parse",
59+
"repo": null
60+
},
61+
{
62+
"file": "internal/storage/sql/migrator.go",
63+
"symbol": "NewMigrator",
64+
"repo": null
65+
},
66+
{
67+
"file": "internal/storage/sql/migrator.go",
68+
"symbol": "expectedVersions",
69+
"repo": null
70+
},
71+
{
72+
"file": "cmd/flipt/main.go",
73+
"symbol": "run",
74+
"repo": null
75+
}
76+
]
77+
}

‎benchmarks/csb_sdlc_fix/flipt-cockroachdb-backend-fix-001/tests/ground_truth_meta.json‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"has_ground_truth": true,
3+
"has_chunk_ground_truth": false,
4+
"ground_truth_source": "curator_agent",
5+
"ground_truth_confidence": "medium",
6+
"task_name": "flipt-cockroachdb-backend-fix-001",
7+
"curator_agent_version": "2.0",
8+
"model": "claude-opus-4-6",
9+
"backend": "hybrid",
10+
"timestamp": "2026-03-03T15:45:20.640497+00:00",
11+
"files_count": 6,
12+
"edit_files_count": 0,
13+
"chunks_count": 0,
14+
"symbols_count": 13,
15+
"cost_usd": 1.7157,
16+
"elapsed_sec": 190.6,
17+
"exploration_notes": "At pinned commit 2d0ff0c, the task requires adding CockroachDB as a database backend across 6 key files:\n\n1. **internal/config/database.go** \u2014 Defines `DatabaseProtocol` enum (iota-based uint8) with constants `DatabaseSQLite`, `DatabasePostgres`, `DatabaseMySQL`. Contains two maps: `databaseProtocolToString` and `stringToDatabaseProtocol` that map between protocol constants and string identifiers. A new `DatabaseCockroachDB` constant and `cockroachdb`/`cockroach` string mappings must be added he"
18+
}

‎benchmarks/csb_sdlc_fix/flipt-ecr-auth-oci-fix-001/tests/ground_truth.json‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
{
2+
"files": [
3+
"internal/config/credentials.go",
4+
"internal/credentials/credentials.go",
5+
"config/flipt.schema.cue",
6+
"internal/config/storage.go",
7+
"internal/storage/environments/environments.go",
8+
"internal/config/config.go"
9+
],
10+
"symbols": [
11+
{
12+
"file": "internal/config/credentials.go",
13+
"symbol": "CredentialType",
14+
"repo": null
15+
},
16+
{
17+
"file": "internal/config/credentials.go",
18+
"symbol": "CredentialConfig",
19+
"repo": null
20+
},
21+
{
22+
"file": "internal/credentials/credentials.go",
23+
"symbol": "GitAuthentication",
24+
"repo": null
25+
},
26+
{
27+
"file": "internal/credentials/credentials.go",
28+
"symbol": "APIAuthentication",
29+
"repo": null
30+
},
31+
{
32+
"file": "internal/credentials/credentials.go",
33+
"symbol": "CredentialSource",
34+
"repo": null
35+
},
36+
{
37+
"file": "internal/storage/environments/environments.go",
38+
"symbol": "GetOrCreate",
39+
"repo": null
40+
},
41+
{
42+
"file": "internal/config/storage.go",
43+
"symbol": "StorageConfig",
44+
"repo": null
45+
}
46+
]
47+
}

‎benchmarks/csb_sdlc_fix/flipt-ecr-auth-oci-fix-001/tests/ground_truth_meta.json‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"has_ground_truth": true,
3+
"has_chunk_ground_truth": false,
4+
"ground_truth_source": "curator_agent",
5+
"ground_truth_confidence": "medium",
6+
"task_name": "flipt-ecr-auth-oci-fix-001",
7+
"curator_agent_version": "2.0",
8+
"model": "claude-opus-4-6",
9+
"backend": "hybrid",
10+
"timestamp": "2026-03-03T15:47:29.022495+00:00",
11+
"files_count": 6,
12+
"edit_files_count": 0,
13+
"chunks_count": 0,
14+
"symbols_count": 7,
15+
"cost_usd": 0.9889,
16+
"elapsed_sec": 128.4,
17+
"exploration_notes": "The flipt codebase has NO existing OCI storage or `internal/oci/` directory \u2014 this is entirely new functionality. The current storage is Git-based with credentials defined in `internal/config/credentials.go` (4 types: basic, ssh, access_token, github_app) and implemented in `internal/credentials/credentials.go`. To add AWS ECR authentication: (1) `internal/config/credentials.go` must be modified to add a new CredentialType for AWS ECR with its config struct and validation, (2) `internal/credenti"
18+
}

‎benchmarks/csb_sdlc_fix/flipt-eval-latency-fix-001/tests/ground_truth.json‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
{
2+
"files": [
3+
"internal/server/evaluation/server.go",
4+
"internal/server/evaluation/evaluation.go",
5+
"internal/server/evaluation/legacy_evaluator.go"
6+
],
7+
"symbols": [
8+
{
9+
"file": "internal/server/evaluation/server.go",
10+
"symbol": "Server",
11+
"repo": null
12+
},
13+
{
14+
"file": "internal/server/evaluation/server.go",
15+
"symbol": "New",
16+
"repo": null
17+
},
18+
{
19+
"file": "internal/server/evaluation/evaluation.go",
20+
"symbol": "Variant",
21+
"repo": null
22+
},
23+
{
24+
"file": "internal/server/evaluation/evaluation.go",
25+
"symbol": "Boolean",
26+
"repo": null
27+
},
28+
{
29+
"file": "internal/server/evaluation/evaluation.go",
30+
"symbol": "Batch",
31+
"repo": null
32+
},
33+
{
34+
"file": "internal/server/evaluation/evaluation.go",
35+
"symbol": "variant",
36+
"repo": null
37+
},
38+
{
39+
"file": "internal/server/evaluation/evaluation.go",
40+
"symbol": "boolean",
41+
"repo": null
42+
},
43+
{
44+
"file": "internal/server/evaluation/legacy_evaluator.go",
45+
"symbol": "Evaluator",
46+
"repo": null
47+
},
48+
{
49+
"file": "internal/server/evaluation/legacy_evaluator.go",
50+
"symbol": "Evaluate",
51+
"repo": null
52+
}
53+
],
54+
"expected_edit_files": [
55+
"internal/server/evaluation/server.go",
56+
"internal/server/evaluation/server.go",
57+
"internal/server/evaluation/evaluation.go",
58+
"internal/server/evaluation/evaluation.go"
59+
],
60+
"expected_edit_files_source": "curator_agent_verifier_analysis",
61+
"expected_edit_files_confidence": "high"
62+
}

‎benchmarks/csb_sdlc_fix/flipt-eval-latency-fix-001/tests/ground_truth_meta.json‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"has_ground_truth": true,
3+
"has_chunk_ground_truth": false,
4+
"ground_truth_source": "curator_agent",
5+
"ground_truth_confidence": "medium",
6+
"task_name": "flipt-eval-latency-fix-001",
7+
"curator_agent_version": "2.0",
8+
"model": "claude-opus-4-6",
9+
"backend": "hybrid",
10+
"timestamp": "2026-03-03T15:48:09.296515+00:00",
11+
"files_count": 3,
12+
"edit_files_count": 4,
13+
"chunks_count": 0,
14+
"symbols_count": 9,
15+
"cost_usd": 0.3488,
16+
"elapsed_sec": 40.3,
17+
"exploration_notes": "The task requires: (1) Creating a new duration tracking file in `internal/server/evaluation/` with a thread-safe struct that records durations and computes count/average/P99. (2) Modifying `server.go` to add the tracker as a field on the `Server` struct and initialize it in `New()`. (3) Modifying `evaluation.go` to wire duration recording into the `Variant()`, `Boolean()`, and `Batch()` methods. The existing codebase already uses OTel metrics for latency (`metrics.EvaluationLatency`) in `boolean"
18+
}

‎benchmarks/csb_sdlc_fix/flipt-otlp-exporter-fix-001/tests/ground_truth.json‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
{
2+
"files": [
3+
"internal/config/tracing.go",
4+
"internal/otel/traces/tracing.go",
5+
"internal/cmd/grpc.go",
6+
"config/flipt.schema.cue",
7+
"config/flipt.schema.json"
8+
],
9+
"symbols": [
10+
{
11+
"file": "internal/config/tracing.go",
12+
"symbol": "TracingConfig",
13+
"repo": null
14+
},
15+
{
16+
"file": "internal/otel/traces/tracing.go",
17+
"symbol": "GetExporter",
18+
"repo": null
19+
},
20+
{
21+
"file": "internal/config/metrics.go",
22+
"symbol": "MetricsExporter",
23+
"repo": null
24+
},
25+
{
26+
"file": "internal/config/metrics.go",
27+
"symbol": "MetricsConfig",
28+
"repo": null
29+
}
30+
]
31+
}

‎benchmarks/csb_sdlc_fix/flipt-otlp-exporter-fix-001/tests/ground_truth_meta.json‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"has_ground_truth": true,
3+
"has_chunk_ground_truth": false,
4+
"ground_truth_source": "curator_agent",
5+
"ground_truth_confidence": "medium",
6+
"task_name": "flipt-otlp-exporter-fix-001",
7+
"curator_agent_version": "2.0",
8+
"model": "claude-opus-4-6",
9+
"backend": "hybrid",
10+
"timestamp": "2026-03-03T15:49:03.399565+00:00",
11+
"files_count": 5,
12+
"edit_files_count": 0,
13+
"chunks_count": 0,
14+
"symbols_count": 4,
15+
"cost_usd": 0.5029,
16+
"elapsed_sec": 54.1,
17+
"exploration_notes": "The task requires adding OTLP as a tracing exporter option alongside the existing autoexport-based approach. The core files are:\n\n1. **internal/config/tracing.go** \u2014 Currently `TracingConfig` only has an `Enabled` bool. Needs an `Exporter` field (like `MetricsConfig` has) with `jaeger`/`zipkin`/`otlp` enum values, and needs `setDefaults` to default to `jaeger`.\n\n2. **internal/otel/traces/tracing.go** \u2014 `GetExporter()` currently delegates to `autoexport.NewSpanExporter()` with no config awareness"
18+
}

‎benchmarks/csb_sdlc_fix/flipt-trace-sampling-fix-001/tests/ground_truth.json‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
{
2+
"files": [
3+
"internal/config/tracing.go",
4+
"internal/cmd/grpc.go",
5+
"internal/config/config.go"
6+
],
7+
"symbols": [
8+
{
9+
"file": "internal/config/tracing.go",
10+
"symbol": "TracingConfig",
11+
"repo": null
12+
},
13+
{
14+
"file": "internal/config/tracing.go",
15+
"symbol": "setDefaults",
16+
"repo": null
17+
},
18+
{
19+
"file": "internal/cmd/grpc.go",
20+
"symbol": "NewGRPCServer",
21+
"repo": null
22+
},
23+
{
24+
"file": "internal/config/config.go",
25+
"symbol": "Default",
26+
"repo": null
27+
}
28+
]
29+
}

‎benchmarks/csb_sdlc_fix/flipt-trace-sampling-fix-001/tests/ground_truth_meta.json‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"has_ground_truth": true,
3+
"has_chunk_ground_truth": false,
4+
"ground_truth_source": "curator_agent",
5+
"ground_truth_confidence": "medium",
6+
"task_name": "flipt-trace-sampling-fix-001",
7+
"curator_agent_version": "2.0",
8+
"model": "claude-opus-4-6",
9+
"backend": "hybrid",
10+
"timestamp": "2026-03-03T15:49:46.652507+00:00",
11+
"files_count": 3,
12+
"edit_files_count": 0,
13+
"chunks_count": 0,
14+
"symbols_count": 4,
15+
"cost_usd": 0.4172,
16+
"elapsed_sec": 43.3,
17+
"exploration_notes": "The task requires modifying 3 files:\n\n1. **`internal/config/tracing.go`** \u2014 The core config struct. Currently `TracingConfig` only has an `Enabled` bool field with a simple `setDefaults()`. This file needs: (a) a `SamplingRate float64` field with default 1.0, (b) a `Propagators []string` field with default propagators, (c) a `validate()` method to check sampling rate is in [0,1] and propagators are from a supported set, using the existing `errFieldWrap`/`errString` error helpers from `errors.go`"
18+
}

0 commit comments

Comments
 (0)