feat(ai,ui): unify cost tracking and chart type taxonomy#401
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>
Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>
Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>
|
This PR is very large. Consider breaking it into smaller PRs for easier review. |
There was a problem hiding this comment.
Pull request overview
This PR unifies cost tracking across AI operations and creates a standardized chart type taxonomy shared by Dashboard and Report widgets.
Changes:
- Created standardized
TokenUsageSchemaandAIOperationCostSchemafor unified AI cost tracking, integrating them into 6 AI protocols (conversation, orchestration, nlq, predictive, rag-pipeline, and cost itself) - Created new
chart.zod.tswith 27 chart types organized by purpose (comparison, trend, distribution, relationship, composition, performance, geo, advanced, tabular), replacing fragmented chart type definitions in Dashboard (27 types) and Report (7 types) - Added 56 comprehensive tests (35 for cost tracking, 21 for chart taxonomy)
Reviewed changes
Copilot reviewed 39 out of 39 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/spec/src/ai/cost.zod.ts | Added TokenUsageSchema and AIOperationCostSchema for standardized cost tracking |
| packages/spec/src/ai/cost.test.ts | Added 35 tests for token usage and AI operation cost schemas |
| packages/spec/src/ai/conversation.zod.ts | Integrated unified TokenUsageSchema and added cost field to messages and sessions |
| packages/spec/src/ai/orchestration.zod.ts | Added tokens and cost fields to execution results |
| packages/spec/src/ai/nlq.zod.ts | Added cost tracking fields to NLQ response schema |
| packages/spec/src/ai/predictive.zod.ts | Added cost tracking fields to prediction results |
| packages/spec/src/ai/rag-pipeline.zod.ts | Added unified token/cost tracking with backward compatibility for deprecated tokensUsed |
| packages/spec/src/ui/chart.zod.ts | New file defining unified chart type taxonomy with 27 chart types |
| packages/spec/src/ui/chart.test.ts | Added 21 comprehensive tests for chart types and configurations |
| packages/spec/src/ui/dashboard.zod.ts | Removed inline chart type enum, now imports from chart.zod.ts |
| packages/spec/src/ui/dashboard.test.ts | Updated tests to use unified chart types, removed tests for obsolete types |
| packages/spec/src/ui/report.zod.ts | Extended ChartConfigSchema for report-specific chart configuration |
| packages/spec/src/ui/report.test.ts | Updated tests to validate new chart types |
| packages/spec/src/ui/index.ts | Added export for chart schemas |
| packages/spec/json-schema/** | Generated JSON schemas for new TypeScript schemas |
| content/docs/references/** | Updated documentation to reflect new schemas and chart types |
| /** Cost Tracking */ | ||
| tokens: TokenUsageSchema.optional().describe('Token usage for this query'), | ||
| cost: z.number().nonnegative().optional().describe('Cost for this query in USD'), |
There was a problem hiding this comment.
The new tokens and cost fields added to NLQResponseSchema lack test coverage. Consider adding tests to the nlq test file to validate these new fields, ensuring proper validation for the TokenUsageSchema structure and non-negative cost values.
| tokens: TokenUsageSchema.optional().describe('Token usage for this prediction (if AI-powered)'), | ||
| cost: z.number().nonnegative().optional().describe('Cost for this prediction in USD'), |
There was a problem hiding this comment.
The new tokens and cost fields added to PredictionResultSchema lack test coverage. Consider adding tests to the predictive test file to validate these new fields, especially for the case where predictions are AI-powered and include token usage and cost information.
| tokens: TokenUsageSchema.optional().describe('Token usage for this query'), | ||
| cost: z.number().nonnegative().optional().describe('Cost for this query in USD'), | ||
| tokensUsed: z.number().int().optional().describe('Deprecated: use tokens.total instead'), |
There was a problem hiding this comment.
The new tokens and cost fields added to RAGQueryResponseSchema lack test coverage. Consider adding tests to the rag-pipeline test file to validate these new fields. Also note that the deprecated tokensUsed field should have tests to ensure backward compatibility while guiding users to migrate to the new tokens.total field.
| tokens: TokenUsageSchema.optional().describe('Token usage for this message'), | ||
| cost: z.number().nonnegative().optional().describe('Cost for this message in USD'), |
There was a problem hiding this comment.
The new cost field added to ConversationMessageSchema lacks test coverage. The existing test file conversation.test.ts contains tests for the tokens field but does not validate the new cost field. Consider adding tests to ensure the cost field validates correctly with positive numbers, rejects negative values, and works properly when optional.
| totalTokens: TokenUsageSchema.optional().describe('Total tokens across all messages'), | ||
| totalCost: z.number().nonnegative().optional().describe('Total cost for this session in USD'), |
There was a problem hiding this comment.
The new totalTokens and totalCost fields added to ConversationSessionSchema lack test coverage. The existing test file conversation.test.ts does not include validation tests for these new fields. Consider adding tests to ensure these fields work correctly, especially for validating that totalCost only accepts non-negative numbers and totalTokens follows the TokenUsageSchema structure.
| tokens: TokenUsageSchema.optional().describe('Total token usage for this execution'), | ||
| cost: z.number().nonnegative().optional().describe('Total cost for this execution in USD'), |
There was a problem hiding this comment.
The new tokens and cost fields added to AIOrchestrationExecutionResultSchema lack test coverage. Consider adding tests to the orchestration test file to validate these new fields, including checking that tokens follows the TokenUsageSchema structure and cost only accepts non-negative numbers.
Problem
Cost tracking fragmented across AI protocols (conversation, orchestration, NLQ, RAG, predictive). Dashboard declares 27 chart types, Report declares 7 - no shared taxonomy.
Changes
AI Cost Tracking Unification
Created standardized schemas:
TokenUsageSchema:{prompt, completion, total}AIOperationCostSchema: operation type, model ID, tokens, cost, timestampIntegrated into 6 AI protocols:
conversation.zod.ts: Addedtokensandcostto messages and sessionsorchestration.zod.ts: Added to execution resultsnlq.zod.ts,predictive.zod.ts,rag-pipeline.zod.ts: Added to response schemasChart Type Taxonomy
Created
ui/chart.zod.tswith 27 chart types organized by purpose:Updated both
dashboard.zod.tsandreport.zod.tsto import from sharedChartConfigSchema:Impact
Original prompt
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.