CacheKey

Stop paying to re-run LLM calls you've already made.

Zerocache is a self-hosted caching proxy that sits between your app and every LLM / embedding provider you use. It intercepts /v1/chat/completions,/v1/messages, and /v1/embeddings, serves repeats from a local content-addressed store in ~1 ms, and forwards only misses upstream. Provider-side prompt caching discounts input tokens for a few minutes and still runs the model; Zerocache serves the whole response — 100% off input and output — for as long as you keep the entry, across runs and machines. No SDK, no framework plugin: point your existing client at a new base_url and bring your own key per request.

Try it →Read the architecture

How it works

Your client sends a normal POST /{provider}/v1/{chat/completions,messages,embeddings}request to Zerocache instead of the real provider.
Zerocache derives a content- and owner-addressed key from a hash of your forwarded API key, the provider, the endpoint identity, the model, and a canonicalization of the request.
A hit returns in ~1 ms — 100% off input and output. A miss is forwarded upstream with your key, then written back.
Only deterministic requests are cached (temperature: 0 or a seed); a non-2xx upstream response is forwarded and never stored.
Concurrent misses on the exact same key are coalesced into one upstream call — in-process, and across replicas on the redis backend.

Where the savings come from

A single monotonic agent run gets almost no completion-cache hits — every turn's prompt is strictly longer than the last. The savings are real, but they come from specific patterns, agreed before building rather than assumed:

Repeated runs — CI/eval loops, prompt tuning, re-asks. A full hit is 100% off; provider prompt caching is an input-only discount on a request that still executes.
Auxiliary LLM calls — the short "summarize this" / "classify that" calls agents fire constantly, which recur verbatim across runs.
Retry / flap storms — a burst of identical requests collapses to one upstream call via in-process coalescing.
Multi-agent fan-out — templated prompts across N workers.
RAG re-indexing — re-embedding a corpus after editing a few documents costs only the deltas.

Measured, not claimed

100%cache hits re-running a 3-ticket support-triage agent suite (10 model + 9 tool calls per run)

~1 mscache hit, vs. 200–800 ms for a cold embedding call and 1–20 s for a cold LLM call

9 + 7chat providers (OpenAI-wire) + embedding providers, plus Anthropic /v1/messages

Re-running that agent suite a second time was zero upstream calls, byte-identical resolutions, and ~5.1k prompt + ~370 completion tokens saved per suite — every intermediate tool-call turn included. The agent showcase ships a real feature with a multi-agent team, then re-runs the identical task served entirely from cache. Full methodology and the three independent consumer battle-tests on theBenchmarks page.

Try it

docker run -d -p 8080:8080 -v zerocache-data:/data ghcr.io/shramanb113/zerocache:latest

# send this twice — the second call is a ~1 ms cache hit, 100% off input AND output
curl http://localhost:8080/openai/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hello"}],"temperature":0}'