Skip to content

Commit fe1d8a1

Browse files
suryaiyer95claude
suryaiyer95
and
claude
authored
feat: [AI-5975] add sql_quality telemetry for issue prevention metrics (#446)
* feat: [AI-5975] add `sql_quality` telemetry for issue prevention metrics Add a new `sql_quality` telemetry event that fires whenever tools successfully detect SQL issues — turning findings into measurable "issues prevented" data in App Insights. Architecture: - New `sql_quality` event type in `Telemetry.Event` with `finding_count`, `by_severity`, `by_category`, `has_schema`, `dialect`, `duration_ms` - New `Telemetry.Finding` interface and `aggregateFindings()` helper - Centralized emission in `tool.ts` — checks `metadata.findings` array after any tool completes, aggregates counts, emits event - Tools populate `metadata.findings` with `{category, severity}` pairs: - `sql_analyze`: issue type + severity from lint/semantic/safety analysis - `altimate_core_validate`: classified validation errors (missing_table, missing_column, syntax_error, type_mismatch) - `altimate_core_semantics`: rule/type + severity from semantic checks - `altimate_core_fix`: fix_applied / unfixable_error categories - `altimate_core_correct`: correction_applied findings - `altimate_core_equivalence`: equivalence_difference findings PII-safe: only category names and severity levels flow to telemetry, never SQL content. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: [AI-5975] drop `by_severity` from `sql_quality` telemetry, address PR review - Remove `severity` from `Finding` interface — only `category` matters - Drop `by_severity` from `sql_quality` event type and `aggregateFindings` - Gate `sql_quality` emission on `!isSoftFailure` to avoid double-counting with `core_failure` events (Copilot review feedback) - Simplify semantics tool: use fixed `"semantic_issue"` category instead of dead `issue.rule ?? issue.type` fallback chain (CodeRabbit review feedback) - Update test header to accurately describe what tests cover - Fix sql_analyze test to use honest coarse categories (`"lint"`, `"safety"`) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: [AI-5975] drop `by_severity` from `sql_quality` telemetry, address PR review - Remove `severity` from `Finding` interface — only `category` matters - Drop `by_severity` from `sql_quality` event type and `aggregateFindings` - Gate `sql_quality` emission on `!isSoftFailure` to avoid double-counting with `core_failure` events (Copilot review feedback) - Simplify semantics tool: use fixed `"semantic_issue"` category instead of dead `issue.rule ?? issue.type` fallback chain (CodeRabbit review feedback) - Update test header to accurately describe what tests cover - Fix sql_analyze test to use honest coarse categories (`"lint"`, `"safety"`) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: [AI-5975] decouple `metadata.success` from domain outcomes in finding tools Five tools were suppressing `sql_quality` telemetry because their `metadata.success` tracked domain outcomes (SQL invalid, policy violated, queries not equivalent) rather than engine execution success. `tool.ts` gate: `!isSoftFailure && findings.length > 0` - `isSoftFailure = metadata.success === false` - Tools that found issues had `success: false` → findings suppressed Fix: set `success: true` when the engine ran (even if it found problems). Domain outcomes remain in dedicated fields (`valid`, `pass`, `equivalent`, `fixed`). Only catch blocks set `success: false` (real engine crashes). Affected tools: - `altimate_core_validate` — validation errors now emit `sql_quality` - `altimate_core_semantics` — semantic issues now emit `sql_quality` - `altimate_core_policy` — policy violations now emit `sql_quality` - `altimate_core_equivalence` — differences now emit `sql_quality` - `altimate_core_fix` — unfixable errors now emit `sql_quality` Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: [AI-5975] remove hardcoded `dialect: "snowflake"` from core tools - Remove `dialect` from metadata in 8 altimate-core/impact tools that don't accept a dialect parameter (it was always hardcoded to "snowflake") - Make `dialect` optional in `sql_quality` telemetry event type - Only emit `dialect` when the tool actually provides it (sql-analyze, sql-optimize, schema-diff still do via `args.dialect`) - Tracked as #455 for adding proper dialect parameter support later Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: [AI-5975] guard finding arrays with `Array.isArray` for defensive safety If a dispatcher returns a non-array for `errors`, `violations`, `issues`, or `changes`, the `?? []` fallback handles null/undefined but not other types. `Array.isArray` prevents `.map()` from throwing on unexpected payloads. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 7a85ad8 commit fe1d8a1

16 files changed

+572
-24
lines changed

‎packages/opencode/src/altimate/native/sql/register.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ register("sql.analyze", async (params) => {
4545
for (const f of lint.findings ?? []) {
4646
issues.push({
4747
type: "lint",
48+
rule: f.rule,
4849
severity: f.severity ?? "warning",
4950
message: f.message ?? f.rule ?? "",
5051
recommendation: f.suggestion ?? "",

‎packages/opencode/src/altimate/native/types.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ export interface SqlAnalyzeParams {
3030

3131
export interface SqlAnalyzeIssue {
3232
type: string
33+
rule?: string
3334
severity: string
3435
message: string
3536
recommendation: string

‎packages/opencode/src/altimate/telemetry/index.ts‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -405,6 +405,21 @@ export namespace Telemetry {
405405
masked_args?: string
406406
duration_ms: number
407407
}
408+
// altimate_change start — sql quality telemetry for issue prevention metrics
409+
| {
410+
type: "sql_quality"
411+
timestamp: number
412+
session_id: string
413+
tool_name: string
414+
tool_category: string
415+
finding_count: number
416+
/** JSON-encoded Record<string, number> — count per issue category */
417+
by_category: string
418+
has_schema: boolean
419+
dialect?: string
420+
duration_ms: number
421+
}
422+
// altimate_change end
408423

409424
const ERROR_PATTERNS: Array<{
410425
class: Telemetry.Event & { type: "core_failure" } extends { error_class: infer C } ? C : never
@@ -774,6 +789,22 @@ export namespace Telemetry {
774789
}
775790
}
776791

792+
// altimate_change start — sql quality telemetry types
793+
/** Lightweight finding record for quality telemetry. Only category — never SQL content. */
794+
export interface Finding {
795+
category: string
796+
}
797+
798+
/** Aggregate an array of findings into category counts suitable for the sql_quality event. */
799+
export function aggregateFindings(findings: Finding[]): Record<string, number> {
800+
const by_category: Record<string, number> = {}
801+
for (const f of findings) {
802+
by_category[f.category] = (by_category[f.category] ?? 0) + 1
803+
}
804+
return by_category
805+
}
806+
// altimate_change end
807+
777808
export async function shutdown() {
778809
// Wait for init to complete so we know whether telemetry is enabled
779810
// and have a valid endpoint to flush to. init() is fire-and-forget

‎packages/opencode/src/altimate/tools/altimate-core-check.ts‎

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import z from "zod"
22
import { Tool } from "../../tool/tool"
33
import { Dispatcher } from "../native"
4+
import type { Telemetry } from "../telemetry"
45

56
export const AltimateCoreCheckTool = Tool.define("altimate_core_check", {
67
description:
@@ -11,6 +12,7 @@ export const AltimateCoreCheckTool = Tool.define("altimate_core_check", {
1112
schema_context: z.record(z.string(), z.any()).optional().describe("Inline schema definition"),
1213
}),
1314
async execute(args, ctx) {
15+
const hasSchema = !!(args.schema_path || (args.schema_context && Object.keys(args.schema_context).length > 0))
1416
try {
1517
const result = await Dispatcher.call("altimate_core.check", {
1618
sql: args.sql,
@@ -19,14 +21,34 @@ export const AltimateCoreCheckTool = Tool.define("altimate_core_check", {
1921
})
2022
const data = (result.data ?? {}) as Record<string, any>
2123
const error = result.error ?? data.error
24+
// altimate_change start — sql quality findings for telemetry
25+
const findings: Telemetry.Finding[] = []
26+
for (const err of data.validation?.errors ?? []) {
27+
findings.push({ category: "validation_error" })
28+
}
29+
for (const f of data.lint?.findings ?? []) {
30+
findings.push({ category: f.rule ?? "lint" })
31+
}
32+
for (const t of data.safety?.threats ?? []) {
33+
findings.push({ category: t.type ?? "safety_threat" })
34+
}
35+
for (const p of data.pii?.findings ?? []) {
36+
findings.push({ category: "pii_detected" })
37+
}
38+
// altimate_change end
2239
return {
2340
title: `Check: ${formatCheckTitle(data)}`,
24-
metadata: { success: result.success, ...(error && { error }) },
41+
metadata: {
42+
success: result.success,
43+
has_schema: hasSchema,
44+
...(error && { error }),
45+
...(findings.length > 0 && { findings }),
46+
},
2547
output: formatCheck(data),
2648
}
2749
} catch (e) {
2850
const msg = e instanceof Error ? e.message : String(e)
29-
return { title: "Check: ERROR", metadata: { success: false, error: msg }, output: `Failed: ${msg}` }
51+
return { title: "Check: ERROR", metadata: { success: false, has_schema: hasSchema, error: msg }, output: `Failed: ${msg}` }
3052
}
3153
},
3254
})

‎packages/opencode/src/altimate/tools/altimate-core-correct.ts‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import z from "zod"
22
import { Tool } from "../../tool/tool"
33
import { Dispatcher } from "../native"
4+
import type { Telemetry } from "../telemetry"
45

56
export const AltimateCoreCorrectTool = Tool.define("altimate_core_correct", {
67
description:
@@ -11,6 +12,7 @@ export const AltimateCoreCorrectTool = Tool.define("altimate_core_correct", {
1112
schema_context: z.record(z.string(), z.any()).optional().describe("Inline schema definition"),
1213
}),
1314
async execute(args, ctx) {
15+
const hasSchema = !!(args.schema_path || (args.schema_context && Object.keys(args.schema_context).length > 0))
1416
try {
1517
const result = await Dispatcher.call("altimate_core.correct", {
1618
sql: args.sql,
@@ -19,14 +21,26 @@ export const AltimateCoreCorrectTool = Tool.define("altimate_core_correct", {
1921
})
2022
const data = (result.data ?? {}) as Record<string, any>
2123
const error = result.error ?? data.error ?? extractCorrectErrors(data)
24+
// altimate_change start — sql quality findings for telemetry
25+
const changes = Array.isArray(data.changes) ? data.changes : []
26+
const findings: Telemetry.Finding[] = changes.map(() => ({
27+
category: "correction_applied",
28+
}))
29+
// altimate_change end
2230
return {
2331
title: `Correct: ${data.success ? "CORRECTED" : "COULD NOT CORRECT"}`,
24-
metadata: { success: result.success, iterations: data.iterations, ...(error && { error }) },
32+
metadata: {
33+
success: result.success,
34+
iterations: data.iterations,
35+
has_schema: hasSchema,
36+
...(error && { error }),
37+
...(findings.length > 0 && { findings }),
38+
},
2539
output: formatCorrect(data),
2640
}
2741
} catch (e) {
2842
const msg = e instanceof Error ? e.message : String(e)
29-
return { title: "Correct: ERROR", metadata: { success: false, iterations: 0, error: msg }, output: `Failed: ${msg}` }
43+
return { title: "Correct: ERROR", metadata: { success: false, iterations: 0, has_schema: hasSchema, error: msg }, output: `Failed: ${msg}` }
3044
}
3145
},
3246
})

‎packages/opencode/src/altimate/tools/altimate-core-equivalence.ts‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import z from "zod"
22
import { Tool } from "../../tool/tool"
33
import { Dispatcher } from "../native"
4+
import type { Telemetry } from "../telemetry"
45

56
export const AltimateCoreEquivalenceTool = Tool.define("altimate_core_equivalence", {
67
description:
@@ -12,9 +13,10 @@ export const AltimateCoreEquivalenceTool = Tool.define("altimate_core_equivalenc
1213
schema_context: z.record(z.string(), z.any()).optional().describe("Inline schema definition"),
1314
}),
1415
async execute(args, ctx) {
15-
if (!args.schema_path && (!args.schema_context || Object.keys(args.schema_context).length === 0)) {
16+
const hasSchema = !!(args.schema_path || (args.schema_context && Object.keys(args.schema_context).length > 0))
17+
if (!hasSchema) {
1618
const error = "No schema provided. Provide schema_context or schema_path so table/column references can be resolved."
17-
return { title: "Equivalence: NO SCHEMA", metadata: { success: false, equivalent: false, error }, output: `Error: ${error}` }
19+
return { title: "Equivalence: NO SCHEMA", metadata: { success: false, equivalent: false, has_schema: false, error }, output: `Error: ${error}` }
1820
}
1921
try {
2022
const result = await Dispatcher.call("altimate_core.equivalence", {
@@ -28,14 +30,28 @@ export const AltimateCoreEquivalenceTool = Tool.define("altimate_core_equivalenc
2830
// "Not equivalent" is a valid analysis result, not a failure.
2931
// Only treat it as failure when there's an actual error.
3032
const isRealFailure = !!error
33+
// altimate_change start — sql quality findings for telemetry
34+
const findings: Telemetry.Finding[] = []
35+
if (!data.equivalent && data.differences?.length) {
36+
for (const d of data.differences) {
37+
findings.push({ category: "equivalence_difference" })
38+
}
39+
}
40+
// altimate_change end
3141
return {
3242
title: isRealFailure ? "Equivalence: ERROR" : `Equivalence: ${data.equivalent ? "EQUIVALENT" : "DIFFERENT"}`,
33-
metadata: { success: !isRealFailure, equivalent: data.equivalent, ...(error && { error }) },
43+
metadata: {
44+
success: !isRealFailure,
45+
equivalent: data.equivalent,
46+
has_schema: hasSchema,
47+
...(error && { error }),
48+
...(findings.length > 0 && { findings }),
49+
},
3450
output: formatEquivalence(data),
3551
}
3652
} catch (e) {
3753
const msg = e instanceof Error ? e.message : String(e)
38-
return { title: "Equivalence: ERROR", metadata: { success: false, equivalent: false, error: msg }, output: `Failed: ${msg}` }
54+
return { title: "Equivalence: ERROR", metadata: { success: false, equivalent: false, has_schema: hasSchema, error: msg }, output: `Failed: ${msg}` }
3955
}
4056
},
4157
})

‎packages/opencode/src/altimate/tools/altimate-core-fix.ts‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import z from "zod"
22
import { Tool } from "../../tool/tool"
33
import { Dispatcher } from "../native"
4+
import type { Telemetry } from "../telemetry"
45

56
export const AltimateCoreFixTool = Tool.define("altimate_core_fix", {
67
description:
@@ -12,6 +13,7 @@ export const AltimateCoreFixTool = Tool.define("altimate_core_fix", {
1213
max_iterations: z.number().optional().describe("Maximum fix iterations (default: 5)"),
1314
}),
1415
async execute(args, ctx) {
16+
const hasSchema = !!(args.schema_path || (args.schema_context && Object.keys(args.schema_context).length > 0))
1517
try {
1618
const result = await Dispatcher.call("altimate_core.fix", {
1719
sql: args.sql,
@@ -24,14 +26,29 @@ export const AltimateCoreFixTool = Tool.define("altimate_core_fix", {
2426
// post_fix_valid=true with no errors means SQL was already valid (nothing to fix)
2527
const alreadyValid = data.post_fix_valid && !error
2628
const success = result.success || alreadyValid
29+
// altimate_change start — sql quality findings for telemetry
30+
const findings: Telemetry.Finding[] = []
31+
for (const fix of data.fixes_applied ?? data.changes ?? []) {
32+
findings.push({ category: "fix_applied" })
33+
}
34+
for (const err of data.unfixable_errors ?? []) {
35+
findings.push({ category: "unfixable_error" })
36+
}
37+
// altimate_change end
2738
return {
2839
title: `Fix: ${alreadyValid ? "ALREADY VALID" : data.fixed ? "FIXED" : "COULD NOT FIX"}`,
29-
metadata: { success, fixed: !!data.fixed_sql, ...(error && { error }) },
40+
metadata: {
41+
success,
42+
fixed: !!data.fixed_sql,
43+
has_schema: hasSchema,
44+
...(error && { error }),
45+
...(findings.length > 0 && { findings }),
46+
},
3047
output: formatFix(data),
3148
}
3249
} catch (e) {
3350
const msg = e instanceof Error ? e.message : String(e)
34-
return { title: "Fix: ERROR", metadata: { success: false, fixed: false, error: msg }, output: `Failed: ${msg}` }
51+
return { title: "Fix: ERROR", metadata: { success: false, fixed: false, has_schema: hasSchema, error: msg }, output: `Failed: ${msg}` }
3552
}
3653
},
3754
})

‎packages/opencode/src/altimate/tools/altimate-core-policy.ts‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import z from "zod"
22
import { Tool } from "../../tool/tool"
33
import { Dispatcher } from "../native"
4+
import type { Telemetry } from "../telemetry"
45

56
export const AltimateCorePolicyTool = Tool.define("altimate_core_policy", {
67
description:
@@ -12,6 +13,7 @@ export const AltimateCorePolicyTool = Tool.define("altimate_core_policy", {
1213
schema_context: z.record(z.string(), z.any()).optional().describe("Inline schema definition"),
1314
}),
1415
async execute(args, ctx) {
16+
const hasSchema = !!(args.schema_path || (args.schema_context && Object.keys(args.schema_context).length > 0))
1517
try {
1618
const result = await Dispatcher.call("altimate_core.policy", {
1719
sql: args.sql,
@@ -21,14 +23,26 @@ export const AltimateCorePolicyTool = Tool.define("altimate_core_policy", {
2123
})
2224
const data = (result.data ?? {}) as Record<string, any>
2325
const error = result.error ?? data.error
26+
// altimate_change start — sql quality findings for telemetry
27+
const violations = Array.isArray(data.violations) ? data.violations : []
28+
const findings: Telemetry.Finding[] = violations.map((v: any) => ({
29+
category: v.rule ?? "policy_violation",
30+
}))
31+
// altimate_change end
2432
return {
2533
title: `Policy: ${data.pass ? "PASS" : "VIOLATIONS FOUND"}`,
26-
metadata: { success: result.success, pass: data.pass, ...(error && { error }) },
34+
metadata: {
35+
success: true, // engine ran — violations are findings, not failures
36+
pass: data.pass,
37+
has_schema: hasSchema,
38+
...(error && { error }),
39+
...(findings.length > 0 && { findings }),
40+
},
2741
output: formatPolicy(data),
2842
}
2943
} catch (e) {
3044
const msg = e instanceof Error ? e.message : String(e)
31-
return { title: "Policy: ERROR", metadata: { success: false, pass: false, error: msg }, output: `Failed: ${msg}` }
45+
return { title: "Policy: ERROR", metadata: { success: false, pass: false, has_schema: hasSchema, error: msg }, output: `Failed: ${msg}` }
3246
}
3347
},
3448
})

‎packages/opencode/src/altimate/tools/altimate-core-semantics.ts‎

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import z from "zod"
22
import { Tool } from "../../tool/tool"
33
import { Dispatcher } from "../native"
4+
import type { Telemetry } from "../telemetry"
45

56
export const AltimateCoreSemanticsTool = Tool.define("altimate_core_semantics", {
67
description:
@@ -11,9 +12,10 @@ export const AltimateCoreSemanticsTool = Tool.define("altimate_core_semantics",
1112
schema_context: z.record(z.string(), z.any()).optional().describe("Inline schema definition"),
1213
}),
1314
async execute(args, ctx) {
14-
if (!args.schema_path && (!args.schema_context || Object.keys(args.schema_context).length === 0)) {
15+
const hasSchema = !!(args.schema_path || (args.schema_context && Object.keys(args.schema_context).length > 0))
16+
if (!hasSchema) {
1517
const error = "No schema provided. Provide schema_context or schema_path so table/column references can be resolved."
16-
return { title: "Semantics: NO SCHEMA", metadata: { success: false, valid: false, issue_count: 0, error }, output: `Error: ${error}` }
18+
return { title: "Semantics: NO SCHEMA", metadata: { success: false, valid: false, issue_count: 0, has_schema: false, error }, output: `Error: ${error}` }
1719
}
1820
try {
1921
const result = await Dispatcher.call("altimate_core.semantics", {
@@ -25,14 +27,27 @@ export const AltimateCoreSemanticsTool = Tool.define("altimate_core_semantics",
2527
const issueCount = data.issues?.length ?? 0
2628
const error = result.error ?? data.error ?? extractSemanticsErrors(data)
2729
const hasError = Boolean(error)
30+
// altimate_change start — sql quality findings for telemetry
31+
const issues = Array.isArray(data.issues) ? data.issues : []
32+
const findings: Telemetry.Finding[] = issues.map(() => ({
33+
category: "semantic_issue",
34+
}))
35+
// altimate_change end
2836
return {
2937
title: hasError ? "Semantics: ERROR" : `Semantics: ${data.valid ? "VALID" : `${issueCount} issues`}`,
30-
metadata: { success: result.success, valid: data.valid, issue_count: issueCount, ...(error && { error }) },
38+
metadata: {
39+
success: true, // engine ran — semantic issues are findings, not failures
40+
valid: data.valid,
41+
issue_count: issueCount,
42+
has_schema: hasSchema,
43+
...(error && { error }),
44+
...(findings.length > 0 && { findings }),
45+
},
3146
output: formatSemantics(hasError ? { ...data, error } : data),
3247
}
3348
} catch (e) {
3449
const msg = e instanceof Error ? e.message : String(e)
35-
return { title: "Semantics: ERROR", metadata: { success: false, valid: false, issue_count: 0, error: msg }, output: `Failed: ${msg}` }
50+
return { title: "Semantics: ERROR", metadata: { success: false, valid: false, issue_count: 0, has_schema: hasSchema, error: msg }, output: `Failed: ${msg}` }
3651
}
3752
},
3853
})

0 commit comments

Comments
 (0)