feat(llm-gateway): complete Cloudflare Worker implementation (Phases 1–7)

[iscekic]

Summary

• Implements the llm-gateway Cloudflare Worker — a transparent drop-in replacement for the Next.js /api/openrouter/[...path]/route.ts proxy
• Full Hono middleware chain matching the existing request flow: body parsing → auth → anonymous gate → rate limiting → provider resolution → balance/org checks → request transform → upstream proxy → background tasks
• 168 unit tests passing, typecheck clean

Phases

Phase	Commit	Description
1	5a6b119	Scaffolding: package.json, wrangler.jsonc, tsconfig, Hono stub
2	775f840	Request parsing, JWT auth, anonymous gate
3	7ce0844	Rate limiting (KV sliding window), provider resolution, BYOK
4	faeacf1	Balance/org checks, request validation, request transform
5	84202a8	Upstream proxy, response handling, custom LLM (Vercel AI SDK), abuse service
6	6121e81	Background tasks: usage accounting, API metrics, request logging, abuse cost
7	b31334a	168 unit tests covering every middleware and library module

Follow-up improvements (also in this PR)

• b07e629 — Replace all setTimeout with Workers-native scheduler.wait()
• b5eaf47 — Convert O11Y service binding from .fetch() HTTP to RPC (ingestApiMetrics method on WorkerEntrypoint), eliminating HTTP routing overhead and admin token auth for internal calls

Architecture

• Hono middleware chain — each concern is a named middleware using c.set()/c.get() on Hono context
• Cloudflare bindings: Hyperdrive (Postgres), KV (rate limiting), Service binding (O11Y via RPC), Secrets Store (all API keys)
• Platform mappings: after() → ctx.waitUntil(), response.clone() → ReadableStream.tee(), setTimeout → scheduler.wait(), Sentry → structured logging
• Smart Placement — colocated with DB, not at edge

Key files

llm-gateway/
  src/
    index.ts              # Hono app with full middleware chain
    middleware/            # 14 middleware files
    handler/proxy.ts      # Upstream fetch + ReadableStream.tee() + background tasks
    background/           # usage-accounting, api-metrics, request-logging, abuse-cost
    lib/                  # Pure functions: models, providers, BYOK, custom-llm, etc.
  test/unit/              # 24 test files, 168 tests
cloudflare-o11y/
  src/index.ts            # Added ingestApiMetrics RPC method

[kilo-code-bot]

Code Review Summary

Status: 2 New Issues Found (101 total) | Recommendation: Address before merge

Fix these issues in Kilo Cloud

Overview

Severity	Count (new)	Count (total)
CRITICAL	0	4
WARNING	2	82
SUGGESTION	0	15

New Issues in This Review Pass

Issue Details (click to expand)

WARNING

File	Line	Issue
llm-gateway/src/background/kilo-pass.ts	510	Triple as unknown as Tx cast violates project coding rules — appears 3 times (lines 510, 512, 516)
llm-gateway/src/background/kilo-pass.ts	312	new Date(nextYearlyIssueAtIso) bypasses parseIso validation — could produce Invalid Date and corrupt issue month to "NaN-NaN-01"

Prior Review Issues (99 comments)

Many issues from prior review passes have been fixed by the author, including:

• ✅ err.message exposed to client → now returns generic error
• ✅ Redundant userExistsWithCache DB query → single lookup
• ✅ null as unknown as string cast → removed
• ✅ Missing await on applyProviderSpecificLogic → fixed
• ✅ Dead src/types.ts file → deleted
• ✅ as ReadableStream<Uint8Array> cast → removed
• ✅ Null-forgiving ! operator → extracted local const
• ✅ Unsafe as cast on HTTP status → narrowed at source
• ✅ Race condition in KV rate limiter → replaced with Durable Object
• ✅ zai provider double config → restructured as if/else chain
• ✅ as cast in anonymous.ts → removed, uses flow-sensitive typing
• ✅ Uncaught JSON.parse in parseAwsCredentials → wrapped in try/catch
• ✅ Missing max_completion_tokens validation → now validated
• ✅ Missing org membership check for custom LLMs → added
• ✅ .tee() on free model path → replaced with buffer+replay
• ✅ Non-constant-time string comparison for pepper → uses timingSafeEqual
• ✅ NaN from parseFloat → guarded with Number.isNaN()
• ✅ Reader not cancelled in catch block → added reader.cancel()
• ✅ Error logging not in waitUntil → wrapped
• ✅ Dead abuse-cost.ts file → deleted
• ✅ Unused wrangler secrets → removed
• ✅ 402→503 early return without background tasks → now schedules tasks first

Several issues were acknowledged as matching the reference implementation (not regressions):

• JSON.parse without try/catch in SSE parser (matches reference)
• JSON.parse on tool call arguments (matches reference)
• new URL() without try/catch (matches reference)
• Internal error message exposed via e.message (matches reference)
• atob() without try/catch (matches reference, data from DB)
• toLowerCase() on model ID (matches reference)

Outstanding issues from prior passes that remain open:

• as ReadableStreamDefaultReader<Uint8Array> casts (3 locations)
• as casts in abuse-service.ts (response.json)
• as cast in custom-llm/index.ts line 162
• as cast in vercel-routing.ts line 48
• as cast in tool-calling.ts line 15
• as cast in index.ts line 25 (useWorkersLogger)
• Stale file header comments in usage-accounting.ts
• abuse_classification hardcoded to 0
• hasGenerationEndpoint: true for custom LLMs
• Free model list duplication between models.ts and providers.ts
• context_length mismatch for minimax model
• Anonymous user ID embeds raw IP address
• fetchWithBackoff retries on all 4xx (including auth failures)
• FIRST_TOPUP_BONUS hardcoded constant
• Inconsistent error response formats across middlewares
• select() with no columns fetches all columns
• Timestamp array scalability in RateLimitDO

Files Reviewed (110+ files)

New llm-gateway worker (core):

• llm-gateway/src/index.ts - Entry point, middleware chain
• llm-gateway/src/handler/proxy.ts - Core proxy handler
• llm-gateway/src/handler/background-tasks.ts - Background task orchestration

Middleware:

• llm-gateway/src/middleware/auth.ts
• llm-gateway/src/middleware/anonymous-gate.ts
• llm-gateway/src/middleware/balance-and-org.ts
• llm-gateway/src/middleware/extract-ip.ts
• llm-gateway/src/middleware/free-model-rate-limit.ts
• llm-gateway/src/middleware/log-free-model-usage.ts
• llm-gateway/src/middleware/parse-body.ts
• llm-gateway/src/middleware/promotion-limit.ts
• llm-gateway/src/middleware/provider-resolution.ts
• llm-gateway/src/middleware/request-timing.ts
• llm-gateway/src/middleware/request-transform.ts
• llm-gateway/src/middleware/request-validation.ts
• llm-gateway/src/middleware/resolve-auto-model.ts

Background tasks:

• llm-gateway/src/background/api-metrics.ts
• llm-gateway/src/background/kilo-pass.ts - 2 new issues
• llm-gateway/src/background/request-logging.ts
• llm-gateway/src/background/usage-accounting.ts

Libraries:

• llm-gateway/src/lib/abuse-service.ts
• llm-gateway/src/lib/anonymous.ts
• llm-gateway/src/lib/byok.ts
• llm-gateway/src/lib/custom-llm/index.ts
• llm-gateway/src/lib/extract-headers.ts
• llm-gateway/src/lib/feature-detection.ts
• llm-gateway/src/lib/jwt.ts
• llm-gateway/src/lib/models.ts
• llm-gateway/src/lib/org-restrictions.ts
• llm-gateway/src/lib/promotions.ts
• llm-gateway/src/lib/prompt-info.ts
• llm-gateway/src/lib/provider-hash.ts
• llm-gateway/src/lib/provider-specific.ts
• llm-gateway/src/lib/providers.ts
• llm-gateway/src/lib/rate-limit.ts
• llm-gateway/src/lib/response-helpers.ts
• llm-gateway/src/lib/rewrite-free-model-response.ts
• llm-gateway/src/lib/sentry.ts
• llm-gateway/src/lib/tool-calling.ts
• llm-gateway/src/lib/vercel-routing.ts

Durable Objects:

• llm-gateway/src/dos/RateLimitDO.ts

Shared packages:

• packages/worker-utils/src/o11y-schemas.ts
• packages/worker-utils/src/user-exists-cache.ts
• packages/worker-utils/src/index.ts

Other workers (refactored):

• cloudflare-o11y/src/index.ts
• cloudflare-o11y/src/api-metrics-routes.ts
• cloudflare-o11y/src/o11y-analytics.ts
• cloudflare-o11y/src/session-metrics-analytics.ts
• cloudflare-session-ingest/src/env.ts
• cloudflare-session-ingest/src/middleware/kilo-jwt-auth.ts
• cloudflare-session-ingest/src/o11y-binding.d.ts
• cloudflare-ai-attribution/src/ai-attribution.worker.ts
• cloudflare-webhook-agent-ingest/src/index.ts

Config & tests: 40+ test files, config files (eslint, vitest, wrangler, tsconfig, package.json)

[kilo-code-bot]

[WARNING]: Stale comments — the file header claims "No PostHog first-usage events" and "No KiloPass threshold check", but the code below actually implements both features (PostHog events around line 729+, KiloPass threshold check around line 545+). These misleading comments should be removed or updated to reflect the current implementation.

[kilo-code-bot]

[WARNING]: hasGenerationEndpoint: true is incorrect for custom LLMs.

Custom LLMs don't expose an OpenRouter-style /generation?id= endpoint. Setting this to true causes fetchGeneration in usage-accounting.ts (line 815) to make a spurious HTTP request to customLlmRow.base_url/generation?id=<messageId> after every successful custom LLM request.

While the call is wrapped in a try/catch (line 835) so it won't crash, it:

• Adds unnecessary latency to the background accounting task
• Sends unexpected requests to the customer's custom LLM endpoint
• Logs a warning on every custom LLM request, creating noise

Suggested change

	hasGenerationEndpoint: true,
	apiKey: customLlmRow.api_key,
	hasGenerationEndpoint: false,

[kilo-code-bot]

[WARNING]: writeApiMetricsDataPoint is not awaited, unlike the sibling ingestSessionMetrics method (line 30) which awaits writeSessionMetricsDataPoint.

writeApiMetricsDataPoint is synchronous for the Analytics Engine write but calls waitUntil for the Stream send. Since this is an RPC method called via service binding, the caller (sendApiMetrics in api-metrics.ts) awaits the result. The method will return before the Stream send completes, which is likely fine since waitUntil extends the execution context. However, the inconsistency with ingestSessionMetrics is worth noting — if writeApiMetricsDataPoint ever becomes async (e.g., for error handling), the missing await would silently swallow errors.

[kilo-code-bot]

[WARNING]: as Parameters<typeof app.use>[1] cast violates the project coding rule to "STRONGLY AVOID typescript's as operator".

Consider using a type-safe wrapper or satisfies instead. If the type mismatch is due to workers-tagged-logger returning a handler typed against an older Hono version, a thin adapter function would be safer:

const logger = useWorkersLogger('llm-gateway');
app.use('*', (c, next) => logger(c, next));

[kilo-code-bot]

[WARNING]: select() with no column arguments fetches ALL columns from kilocode_users. This includes potentially large/sensitive fields that are never used by the gateway (e.g., profile data). Consider selecting only the columns actually needed downstream:

.select({
  id: kilocode_users.id,
  api_token_pepper: kilocode_users.api_token_pepper,
  google_user_email: kilocode_users.google_user_email,
  microdollars_used: kilocode_users.microdollars_used,
  total_microdollars_acquired: kilocode_users.total_microdollars_acquired,
  is_admin: kilocode_users.is_admin,
})

This reduces data transfer from Postgres and avoids leaking unnecessary PII into the request context.

[kilo-code-bot]

[WARNING]: fraudHeaders is read from context here (line 76), but it was set by requestTransformMiddleware which runs before proxyHandler. However, isLegacyExtension checks fraudHeaders.http_user_agent — if requestTransformMiddleware ever fails to set fraudHeaders, this would throw a runtime error accessing .http_user_agent on undefined. The middleware chain ordering is correct today, but this implicit dependency is fragile. Consider adding a null check:

const isLegacyExtension = !!fraudHeaders?.http_user_agent?.startsWith('Kilo-Code/');

[kilo-code-bot]

[WARNING]: new Response(errorBodyBytes, response) passes the original response as the ResponseInit. The Response constructor accepts { status, statusText, headers } from the second argument. Since response is a full Response object (not a plain ResponseInit), this relies on the runtime extracting only the relevant fields. While this works in practice on Cloudflare Workers, it's an implicit behavior. Consider being explicit:

new Response(errorBodyBytes, { status: response.status, statusText: response.statusText, headers: response.headers })

[kilo-code-bot]

[WARNING]: as Array<Record<string, unknown>> cast violates the project coding rule to "STRONGLY AVOID typescript's as operator". Since content is already narrowed to Array by the Array.isArray(content) check, you can use a type guard in the filter predicate without the cast:

return content
  .filter(
    (part): part is { type: string; text: string } =>
      part !== null &&
      typeof part === 'object' &&
      ('type' in part) &&
      (part.type === 'input_text' || part.type === 'output_text') &&
      ('text' in part) &&
      typeof part.text === 'string'
  )
  .map(p => p.text);

[kilo-code-bot]

[WARNING]: as unknown as Array<{ gateway: string; errorRate: number }> is a double as cast that violates the project coding rules. The db.execute generic parameter already types the rows, so this cast is unnecessary. Consider:

const rows = result.rows;
return {
  openrouter: rows.find(r => r.gateway === 'openrouter')?.errorRate ?? 0,
  vercel: rows.find(r => r.gateway === 'vercel')?.errorRate ?? 0,
};

If the types don't align, use satisfies or a runtime check instead of as unknown as.

[kilo-code-bot]

[WARNING]: extraRequiredProviders is called with normalizedModelId (which strips the colon suffix via normalizeModelId), but the kiloFreeModelProviders map keys include colons (e.g., corethink:free, x-ai/grok-code-fast-1:optimized:free). This means the lookup will NEVER match for models like corethink:free (normalized to corethink) or x-ai/grok-code-fast-1:optimized:free (normalized to x-ai/grok-code-fast-1). The provider restriction check for enterprise orgs will silently skip these models.

While the previous reviewer noted this is "safe because the default behavior is permissive", this means enterprise orgs with a provider_allow_list that excludes corethink or stealth will NOT be blocked from using those free models — the restriction is silently bypassed. This could be a compliance issue for enterprise customers who expect their allow list to be enforced.

[kilo-code-bot]

[WARNING]: fetchWithBackoff retries on ALL 4xx responses (r.status >= 400), including 401 (auth failure) and 403 (forbidden). Auth failures will never succeed on retry. Consider only retrying on 404 and 5xx:

r => r.status === 404 || r.status >= 500

[kilo-code-bot]

[WARNING]: abuse_classification is hardcoded to 0 here, even though usageContext.abuse_request_id is available. The reference implementation stores the abuse classification request ID in this field. This means the abuse_classification column in microdollar_usage will always be 0, making it impossible to correlate usage records with abuse classifications in the database.

Suggested change

	abuse_classification: 0,
	abuse_classification: usageContext.abuse_request_id ?? 0,

[kilo-code-bot]

[WARNING]: shouldRouteToVercel is called with requestedModel (already lowercased by parseBodyMiddleware), but preferredModels in models.ts contains mixed-case entries like anthropic/claude-opus-4.6. The preferredModels.includes(requestedModel) check in vercel-routing.ts:109 will fail for any model whose preferredModels entry uses different casing than the lowercased requestedModel. Currently all entries happen to be lowercase, but this is fragile — adding a model with uppercase characters to preferredModels would silently break Vercel routing for it.

[kilo-code-bot]

[WARNING]: as Record<string, unknown> cast violates the project coding rules. This can be replaced with an in check:

Suggested change

	return msg.role === 'tool' && typeof (msg as Record<string, unknown>).tool_call_id === 'string';
	return msg.role === 'tool' && 'tool_call_id' in msg && typeof msg.tool_call_id === 'string';

[kilo-code-bot]

[WARNING]: context_length mismatch — minimax/minimax-m2.5:free is 204_800 here but 1_000_000 in providers.ts. The value here feeds getKiloFreeModelContextLength() → makeErrorReadable(), which will incorrectly report context-length errors at ~205K tokens when the model actually supports 1M tokens. This will cause false "context length exceeded" error messages for users with large prompts between 205K and 1M tokens.

Suggested change

	context_length: 204_800,
	context_length: 1_000_000,

[kilo-code-bot]

[SUGGESTION]: as cast violates the project coding rule to "STRONGLY AVOID typescript's as operator". Since err is already narrowed to Error by the instanceof check, you can use an in guard instead:

Suggested change

	if ((err as { code?: string }).code === 'ERR_JWT_EXPIRED') {
	if ('code' in err && err.code === 'ERR_JWT_EXPIRED') {

[kilo-code-bot]

[WARNING]: x-forwarded-for is trivially spoofable by clients. On Cloudflare Workers, CF-Connecting-IP is the only trustworthy source — it's set by Cloudflare's edge and cannot be forged. If CF-Connecting-IP is absent, the request likely didn't traverse Cloudflare's edge (e.g., direct-to-origin), so trusting x-forwarded-for means an attacker could spoof their IP to bypass rate limiting (freeModelRateLimitMiddleware, promotionLimitMiddleware) and abuse detection.

Consider either:

• Rejecting requests without CF-Connecting-IP (they shouldn't reach a Workers deployment normally)
• Or at minimum, logging a warning when falling back to x-forwarded-for so it's visible in monitoring

[kilo-code-bot]

[WARNING]: Triple as unknown as Tx cast violates the project coding rule to "STRONGLY AVOID typescript's as operator".

This pattern appears three times (lines 510, 512, 516). The Tx type is extracted via a conditional type from WorkerDb['transaction'], but the Drizzle transaction callback parameter should already be compatible. Consider defining Tx directly from the transaction callback's parameter type, or using a wrapper function that accepts the transaction and calls the inner functions, avoiding the cast entirely:

type Tx = Parameters<Parameters<WorkerDb['transaction']>[0]>[0];

If the types genuinely don't align, a single cast at the db.transaction call site (wrapping the callback) would be cleaner than casting on every usage.

[kilo-code-bot]

[WARNING]: new Date(nextYearlyIssueAtIso) bypasses the parseIso validation used on line 308.

If nextYearlyIssueAtIso is a truthy but malformed string (e.g., "not-a-date"), parseIso on line 308 returns null and anchor falls through to startedAtIso. But on line 311, the truthy check passes and new Date(nextYearlyIssueAtIso) produces Invalid Date, which propagates through addMonths → computeIssueMonth producing "NaN-NaN-01" as the issue month.

Use the already-validated result instead:

Suggested change

	? addMonths(new Date(nextYearlyIssueAtIso), -1)
	? addMonths(parseIso(nextYearlyIssueAtIso) ?? anchor, -1)

Or reuse the parseIso result from line 308 by extracting it into a local variable.