Architecture

Layers

What a hit vs. a miss actually costs

Design principles

Content-addressed keys

Embeddings: blake3(owner_id + provider + cache_scope + model + model_version + canonicalize_text(text)). Chat/messages: owner_id + provider + cache_scope + model + adapter_version + canonicalized request body, domain-separated per surface. Model identity is in the key deliberately, so a model or adapter upgrade can't silently return a stale-but-plausible response.

Owner-scoped, not globally shared

Two callers embedding identical text under the same model and same forwarded key share a hit for free; two different callers never do — cost fairness and to avoid a cache-timing existence-leak risk.

Cache scope

A hash-derived scope component so a self-hosted or region-specific upstream can never silently reuse a vector computed by a different upstream behind the same model string.

In-process request coalescing

Concurrent misses on the exact same key share one provider call instead of each paying for it independently — proven with 5 concurrent misses producing exactly 1 real provider call. Covers embeddings, images, and completions.

Cross-replica request coalescing (opt-in)

ZEROCACHE_CROSS_REPLICA_COALESCING=1 on the redis backend adds a Redis-lock single-flight so N replicas make one upstream call for the same single-key miss (any chat completion, or a one-input embedding). Multi-item batches and images stay in-process only. Any Redis error degrades to today's per-replica behaviour, never a failed request.

Completion & messages cache

POST /{provider}/v1/chat/completions and /v1/messages cache whole responses keyed by a canonicalized request body (an order-independent serialization with a denylist of non-output-affecting fields). Only deterministic requests — temperature 0 (or a seed, for chat). A hit is 100% off input and output. Streaming works on the chat surface: buffered on a miss, replayed frame-by-frame on a hit.

Semantic near-match tier (opt-in)

A --features semantic build plus ZEROCACHE_SEMANTIC=1 adds a local candle embedder (all-MiniLM-L6-v2, compiled in) + an in-memory HNSW index. On an exact-match miss, a hit needs both a coarse-key byte match of the rest of the request and cosine similarity above a conservative threshold. The threshold, not the embedder, bounds false positives. sled in-process, or across replicas on redis via a Redis Stream change-feed with a blocking (XREAD BLOCK) reader.

Roadmap

Budgets & rate limits

Per-key monthly spend caps (429 when exceeded) and per-key RPS limits, with a cost-by-team view in the dashboard.

Multi-provider failover

Retry a failed request on a second configured provider. The adapters already exist; only the routing policy is missing.

Request log + replay

Opt-in: persist request/response pairs, browse them, replay one, diff the result.

One-click deploy

A fly.toml / Deploy button, and a hosted free tier.

Anthropic /v1/messages streaming & semantic tier

The OpenAI-wire chat surface has streaming buffer-and-replay and the semantic near-match tier; the /v1/messages surface has neither yet — stream: true there is a raw passthrough.

Workspace crates