Engineering decisions
Zerocache's own internal spec keeps an explicit log of every place the shipped code diverges from the original plan, and why. This page distills that log — expand any item for the full reasoning.
Provider adapters take an explicit model parameterThe wire contract requires honoring whatever model the client sends per request.
Without it, one adapter instance could only ever serve a single hardcoded model.
Store and provider traits require Send + Synczerocache-http shares them across async axum handlers behind Arc<dyn Trait>.
A structural requirement of the concurrency model, not a stylistic choice.
Both port traits return Result, not bare valuesOriginally scaffolded without this, and it was a real gap.
Failures panicked instead of surfacing as HTTP errors — fixed once discovered.
The Redis storage adapter shipped ahead of the PRD's own scheduleKubernetes multi-replica deployment was a real near-term requirement, not a hypothetical.
Confirmed with the PRD's author before building. Sled remains the default; Redis is opt-in.
GET /metrics ships in v1, ahead of the full observability specPrometheus's scrape-and-sum() model is the only way per-pod counters mean anything once multiple replicas exist.
Deliberately scoped to just hit/miss/token counters — no per-consumer tagging or latency-saved-vs-baseline yet.
Per-request bring-your-own-key replaces one operator-configured providerAny of seven providers can be selected per request, using the caller's own forwarded key — never Zerocache's.
This also activated owner-scoping: the cache key now includes a hash of the caller's key, so no two callers ever share an entry.
EmbeddingProvider is asyncRemoved a real architectural wart — two runtimes and a thread-pool hop per request under the old blocking-inside-spawn_blocking design.
Each adapter also chunks input into batches of 100 and builds its HTTP client with a uniform 30s timeout.
Provider timeouts, graceful shutdown, /health + /ready, bounded store callsA hung upstream connection used to block a request forever.
Verified for real, not just read for correctness: built and ran on Linux via WSL2, sent a genuine SIGTERM, confirmed a clean shutdown with no dropped in-flight request.
input accepts a bare string, not just an arrayReal LangChain/TS battle-testing found embedQuery() sends the bare-string form.
A real RAG app's question-answering path failed 100% of the time against Zerocache until this fix — document ingestion (the array form) had worked fine the whole time, masking the bug.
Malformed-JSON and wrong-shape errors use the app's own error envelopeThey used to return axum's default plain-text body — a real inconsistency the LangChain battle-test flagged.
Same status code and message axum already computed, just wrapped in the app's own { "error": "..." } shape.
Concurrent misses on the exact same key are coalesced into one provider call5 concurrent misses on a never-before-seen text used to produce 5 real provider calls before this fix.
In-process only — two replicas behind a load balancer each keep their own independent in-flight map. Proven with a dedicated concurrency test.
Provider adapters retry transient failures with exponential backoffMotivated by a real 429 hit against Gemini during the Python/LlamaIndex battle-test, not a hypothetical.
Retries 5xx/408/429/connection errors, never other 4xx.
Full request tracing, OTLP export optionalNo collector required to run Zerocache locally, matching the existing pattern for optional env-gated behavior.
A trace shows the full HTTP handler → cache lookup → provider call → store write path when enabled.
Image embeddings are a separate port, implemented by Gemini onlyOpenAI's public API has no image-embedding endpoint at all, verified live at planning time.
A default method bolted onto EmbeddingProvider would force dead code onto every text-only adapter.
HuggingFace: a fourth text-only provider, model in the URL pathHuggingFace's unified router is chat-completions-only and explicitly excludes embeddings — a genuinely different wire shape.
Still fits the existing EmbeddingProvider trait unchanged, since model was already a per-call parameter.
Duplicate images in one batch are embedded onceCloses a gap the original image-embedding launch had deliberately deferred.
Mirrors the text path's existing within-batch dedup exactly.
Four env vars let adapters point at self-hosted, wire-compatible endpointsHelps only for services that speak the exact same wire protocol as the real provider.
Does not add support for Azure, Bedrock, or Vertex AI — none of those are a base-URL swap.
Azure, Bedrock, and Vertex AI ship via a shared adapter kit, not three more copiesEach cloud is one API in front of several independent vendor wire shapes.
Multi-vendor routing lives entirely in the caller's model string — no new wire field, since model was already free-form and already lands in the cache key. All three ship mock-only, with no live-key smoke test yet.
CacheKey gained a cache_scope component, invalidating every prior entry onceWithout it, two different upstreams behind the same model string could silently share a vector.
A wrong vector is silent in a way a cold cache start never is — judged the correct tradeoff.
Image embedding gets the same request coalescing as textCloses the last gap the original image-embedding launch had explicitly deferred.
A second, separate in-flight map — kept apart from the text path's so a burst of image traffic can never contend the text path's lock.
The completion cache caches whole chat completions, not embeddingsA full hit is 100% off input and output — versus provider-side prompt caching's input-only discount on a request that still runs the model.
The savings come from repeated runs (CI/eval loops, re-asks), the short auxiliary LLM calls agents fire constantly, retry storms, and multi-agent fan-out. Only deterministic requests (temperature 0 or a seed) are cached; the key is an order-independent canonicalization of the output-affecting fields.
One generic OpenAI-wire chat adapter behind nine built-in providersItem 21 shipped with one chat provider hardcoded; every OpenAI-wire endpoint is the same wire shape with a different base URL.
openai, mistral, gemini, groq, deepseek, together, openrouter, xai, fireworks register with zero config; ZEROCACHE_CHAT_PROVIDERS adds or repoints one. The configured value is the full prefix up to /chat/completions — not the embeddings 'bare origin' rule, because that can't express Gemini's /v1beta/openai compat prefix.
The live savings dashboard is a real framework SPA embedded in the binaryThe ask was for a real charting library, not hand-rolled HTML — and it must ship with the single static binary.
Astro + React islands + Recharts, built to a committed dist/ that include_dir! embeds at compile time, served at /dashboard on the same origin as /metrics. cargo build needs no Node; only rebuilding the dashboard does. Completion savings are exact (from the stored usage block); embedding savings are estimated.
The semantic near-match tier is a non-workspace crate, compile-time gatedcandle + a bundled model must never be in the path of cargo build --workspace / cargo test --workspace.
zerocache-semantic is excluded from the workspace; it builds only via zerocache-http --features semantic. The default image is byte-identical to before. Runtime opt-in on top of that (ZEROCACHE_SEMANTIC=1). The cosine threshold, not the embedder, bounds false positives — a small MiniLM-class model works for every chat provider, adds no second key, adds no per-request latency.
The multi-replica semantic index propagates over a Redis Stream, poll then pushThe tier shipped sled-only; on redis it used to fall back to exact-match. N replicas each keep an in-memory HNSW graph and need to learn each other's writes.
One global stream zerocache:semantic:events; each replica folds the backlog at boot, then a background task tails it. First via a polled changes_since, then (item 29) via a blocking XREAD BLOCK so propagation lag drops to one round-trip and an idle feed makes zero calls. A not-yet-propagated vector is just a semantic miss that falls through — absent, never wrong.
Cross-replica request coalescing is opt-in and single-key onlyIn-process coalescing still lets N replicas behind a load balancer each pay the provider once for the same miss.
ZEROCACHE_CROSS_REPLICA_COALESCING=1 on redis adds a SET NX PX lock + pub/sub wake so the same single CacheKey (any chat completion, a one-input embedding) resolves to one upstream call. Multi-item batches and images stay in-process. Any Redis error degrades to today's per-replica behaviour — never a failed request; a non-2xx upstream is never stored.
Streaming completions: buffer on a miss, replay SSE on a hitstream: true used to be parsed as JSON, fail, and get cached as an error object.
A miss streams upstream frames to the client live while a background task buffers, assembles, and (on a clean finish) stores the record — with raw_sse alongside the assembled body. A stream:true and stream:false request for the same deterministic body share one entry. A hit replays raw_sse frame-by-frame; a non-cacheable stream:true request is a pure passthrough.
DELETE /{provider}/v1/chat/completions is the eviction counterpart the completion cache lackedThe completion and messages caches could store but not explicitly evict a single entry — the embedding path already had DELETE.
Same body shape, same Authorization requirement, same 401/404/422 ladder as the chat POST; response { "deleted": 1 }, owner- and cache_scope-scoped, idempotent (count is keys requested, not found). One shared key-derivation helper now backs POST, streaming POST, and DELETE so they can't drift. With the semantic tier on, it also drops the request's vector record and HNSW node.
Anthropic /v1/messages is a native surface, not an OpenAI-compat shimClaude-based agents send Anthropic's wire shape; forcing it through the OpenAI path would lose fields and mangle the response.
New zerocache-adapters-anthropic + a messages orchestrator that reuses CompletionStore, the completion metrics, and the coalescing machinery unchanged. Auth is Bearer in / x-api-key upstream; anthropic-version and anthropic-beta fold into the cache key. Cache gate is temperature 0 only — Anthropic has no seed/n. stream: true is a raw passthrough; the semantic tier for this surface is deferred.