Feature Request: Persistent context slot (flat monthly fee)

[will83]

There's a gap between ephemeral prompt caching (5min/1h TTL) and fine-tuning. For apps with a large, stable system context (~50-100K tokens) and moderate but irregular traffic, neither option fits well:

• Prompt caching works during peak hours but the cache expires during off-hours, forcing repeated cold writes when traffic resumes
• Fine-tuning is overkill when the context is just static reference data, not behavioral changes
• Dedicated instances ($1K+/month) are way too expensive for small-to-medium apps

Example use case: A customer support agent for a SaaS product. The system context holds ~80K tokens of product docs, pricing rules, refund policies, and edge cases. Traffic is high during business hours (Europe), dead at night. Every morning the cache is cold again — full write cost, higher latency on first requests.

Proposal

A "persistent context slot" — a pre-loaded context that doesn't expire, for a flat monthly fee:

• Upload context once via API (or a dashboard)
• Choose which models to pre-load it on (Haiku, Sonnet, Opus — each separately)
• Send requests with just the user message + a context_id — no need to re-send the context every time
• No TTL, no cache misses, no cold starts

Suggested pricing (as a user, this is what I'd be willing to pay)

Context size	Haiku	Sonnet	Opus
50K tokens	$5/mo	$13/mo	$20/mo
100K tokens	$10/mo	$25/mo	$40/mo
250K tokens	$25/mo	$60/mo	$95/mo

Output tokens billed at normal per-request rates. The slot only covers the stored context.

Why this matters

Today, keeping 100K tokens warm on Opus with 1h cache costs ~$36/month in refreshes alone — plus $0.05 per request in cache hits. At 1000 req/day, that's ~$1,500/month just in cache hits. Most small-to-medium apps can't justify that.

A persistent slot at $10-40/month would unlock a whole category of apps that don't exist today because the economics don't work with ephemeral caching. It also creates predictable MRR (vs volatile pay-per-use) and natural lock-in (context becomes a hosted asset).

Happy to discuss further.

[xXMrNidaXx]

Strong +1 on this. The gap between ephemeral caching and fine-tuning is real and painful.

Additional use cases that would benefit:

1. RAG systems with large but stable corpora — Company wikis, legal docs, product catalogs. The corpus rarely changes but the queries are constant. Today you either:

   • Re-embed and send 50-100K tokens every request (expensive)
   • Run a keep-alive process to refresh cache (operational overhead + still paying write costs)

2. Multi-tenant SaaS with per-customer context — Each customer has their own ~20-50K token knowledge base. With ephemeral caching, you'd need N separate caches all being refreshed. With slots, you'd pay N × $5-20/month — predictable and sane.

3. Coding assistants with large codebases — The codebase context (file tree, key modules, conventions) is ~stable. Ephemeral cache expires between coding sessions, so every morning starts cold.

On pricing:

The proposed pricing seems reasonable. For comparison:

• Keeping 100K tokens warm on Sonnet with 1h cache = ~$18/month in refresh costs (assuming 720 refreshes/month)
• Your proposed slot = $25/month but zero operational overhead and no cold starts

The slot model also aligns better with how customers budget ($/month) vs the current cache model (hope your traffic pattern doesn't cause unexpected refreshes).

Alternative framing: Could this be implemented as "pinned cache" — same API as prompt caching but with cache_control: { type: 'pinned' } instead of ephemeral? Would require billing changes but keep the API surface minimal.

[will83]

Thanks for the additional use cases — they all fit the same pattern perfectly.

On the cache_control: { type: 'pinned' } framing: I see the appeal from an API surface standpoint, but it doesn't solve the core problem.

With a pinned cache, you'd still need to send the full context payload on every request. The server skips reprocessing it, but you're still transmitting 50-100K tokens of input each time — and paying input token costs on every call. For a 100K token context at 1000 req/day on Sonnet, that's still ~$150/day in input costs alone, even with a cache hit.

The whole point of a persistent slot is that the context becomes a hosted asset, referenced by ID:

{
  "context_id": "ctx_abc123",
  "messages": [{ "role": "user", "content": "..." }]
}

No retransmission. No input token cost on the context. Just the user message.

That's the architectural shift that makes the economics work for the use cases we're both describing.