Using Zerocache

The only change to your app is the base URL. Your key, your models, your SDK, unchanged — Zerocache forwards your key upstream on a miss and stores only a hash of it.

Run it

docker run -d --name zerocache -p 8080:8080 \
  -v zerocache-data:/data ghcr.io/shramanb113/zerocache:latest
# or:  cargo run -p zerocache-http
# semantic near-match tier:  ghcr.io/shramanb113/zerocache:semantic  (then set ZEROCACHE_SEMANTIC=1)

No provider key is needed to start it — you send one per request.

Point your client — one line

OpenAI Python SDK (chat + embeddings):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/openai/v1",
    api_key="sk-your-real-key",
)

OpenAI TS SDK:

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:8080/openai/v1",
  apiKey: process.env.OPENAI_API_KEY,
});

Anthropic Python SDK (/v1/messages):

from anthropic import Anthropic

client = Anthropic(
    base_url="http://localhost:8080/anthropic",
    api_key="sk-ant-your-real-key",
)

Swap openai for any of mistral, gemini,groq, deepseek, together, openrouter,xai, fireworks — each takes that provider's own key.

Chat completions

curl http://localhost:8080/openai/v1/chat/completions \
  -H "Authorization: Bearer sk-your-real-key" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}],"temperature":0}'
# send it twice — the second response has X-Zerocache-Completion-Hit: true, ~1 ms, $0

Only deterministic requests are cached — temperature: 0 or an explicit seed. stream: true works: buffered on a miss, replayed frame-by-frame on a hit. Full contract on the API page.

Embeddings

Same story on /{provider}/v1/embeddings — identical text (down to casing, Unicode form, and punctuation) stops costing a second call. input takes a string or an array. Image embeddings: POST /gemini/v1/images/embeddings withdata: URIs.

Anthropic Messages

POST /{provider}/v1/messages with Claude's native shape.Authorization: Bearer is rewritten to x-api-key upstream;temperature: 0 to be cacheable. anthropic-version andanthropic-beta fold into the cache key.

Watch it work

GET /dashboard — live hit rate, tokens not billed, dollars saved, per provider. Screenshot this mid-run.
GET /metrics — the Prometheus counters behind the dashboard.
Response headers: X-Zerocache-Hits / X-Zerocache-Misses(embeddings), X-Zerocache-Completion-Hit / -Hit-Kind /-Semantic-Score (chat + messages).

Deploy

Single instance: the default sled store + a persistent volume (-v zerocache-data:/data).
Multi-replica: ZEROCACHE_STORAGE_BACKEND=redis + ZEROCACHE_REDIS_URL so every pod shares one cache. Add ZEROCACHE_CROSS_REPLICA_COALESCING=1 to collapse concurrent identical single-key misses across replicas.
/health and /ready are your Kubernetes probes; scrape /metrics per pod and sum().

Invalidate

DELETE /{provider}/v1/{embeddings,images/embeddings,chat/completions,messages}
Authorization: Bearer <caller's own key>
<same body a matching POST would send>

→ { "deleted": <count> }

Owner-scoped, idempotent — the count is keys requested, not keys found.