Cut LLM token spend and answer repeat AI requests from cache in under a millisecond while running a multi-threaded, Redis-compatible store.
HotKV sits in front of your LLM calls as one Redis-compatible engine. It recognizes prompts it has already answered, exactly or approximately, and gives your application the cached response instead of a new API call.
import hotkv client = hotkv.from_url("hotkv://localhost:6379")client.pcache_create("support", ttl=3600) def ask(prompt: str) -> str: # Asked before, word for word: no model call cached = client.pcache_get("support", prompt) if cached is not None: return cached.decode() answer = call_llm(prompt) client.pcache_set("support", prompt, answer, ttl=3600) return answerIllustrative result (example data)
Scenario: ask("How do I reset my password?") twice
Both calls return: "Use the password-reset link on the sign-in page."
First call: cache miss → call_llm → stored answer
Second call: cached answer → no model callWe sent the same 3,000 customer support questions to four current Claude models twice: once straight to the model, once with HotKV in front.
At 1 million requests a day, extrapolated from the measured cost per request and assuming the same traffic pattern. The four Claude models ran with short answers and Fable 5.1 and Opus 5.5 at their lowest reasoning effort, so longer or reasoning-heavy answers would make each avoided call worth more.
We ran Claude Fable 5.1, Opus 5.5, Sonnet 5.5 and Haiku 5.5 on the same 3,000 requests each, with and without HotKV. The provider's own prompt caching was on in every run, and tokens and costs are taken from each response's usage, thinking tokens included.
| Model | Tokens saved | Cost without | Cost with | Lower by | Model calls | Median response |
|---|---|---|---|---|---|---|
| Claude Fable 5.1 | 70% | $30.31 | $9.18 | 70% | 3,000 to 896 | 2.7 s to 127 ms |
| Claude Opus 5.5 | 70% | $14.60 | $4.50 | 69% | 3,000 to 898 | 2.9 s to 126 ms |
| Claude Sonnet 5.5 | 70% | $5.51 | $1.66 | 70% | 3,000 to 897 | 1.8 s to 128 ms |
| Claude Haiku 5.5 | 70% | $0.27 | $0.08 | 70% | 3,000 to 897 | 1.4 s to 125 ms |
For every reworded question served from cache, a different model compared the cached answer with the fresh answer the same model gave to that exact question, in random order. No model graded its own output. This is a language model judging, not human review, and its strictness differs between models, so compare cached with fresh only within one model.
| Model | Reworded questions judged | Cached answer fits | Fresh answer fits |
|---|---|---|---|
| Claude Fable 5.1 | 926 | 83.3% | 90.2% |
| Claude Opus 5.5 | 935 | 93.3% | 99.0% |
| Claude Sonnet 5.5 | 936 | 77.9% | 86.1% |
| Claude Haiku 5.5 | 936 | 67.9% | 78.6% |
Each row is a separate run of Claude Sonnet 5.5 on the same 3,000 requests (cost without HotKV: $5.51).
| Threshold | Cost | Lower by | Cached answer fits | Fresh answer fits |
|---|---|---|---|---|
| 0.81 | $1.66 | 70% | 77.9% | 86.1% |
| 0.85 | $1.88 | 66% | 80.3% | 85.6% |
| 0.88 | $1.99 | 64% | 89.5% | 83.8% |
| 0.90 | $2.13 | 61% | 87.9% | 86.5% |
| 0.95 | $2.22 | 60% | 92.0% | 90.0% |
3,000 customer-support questions drawn from the Banking77 test split with a fixed seed (intent popularity follows a Zipf distribution, s=1.0; questions are sampled uniformly within each intent, with repeats). Each question was sent to the model with the same support policy as the system prompt. Baseline: every request calls the model, with the provider's own prompt caching enabled. HotKV: exact-match lookup (GET on a SHA-256 key of the normalized question), then a semantic lookup (SCACHE.GET with an embedding of the question), and the model is called only on a miss. Token counts are taken from each API response's usage block; costs use list prices and include the embedding calls. The similarity threshold was chosen on the separate Banking77 train split as the lowest value giving at least 98% same-intent precision on semantic hits.
Embedding model: OpenAI text-embedding-3-small. Semantic match threshold: 0.81. Dataset: Banking77 (test split) by PolyAI, licensed CC BY 4.0. 3,000 requests across 77 intents.
Measured October 11, 2026. Savings depend on how often your traffic repeats itself. A workload with no repeated or similar questions saves nothing.
We loaded the same OpenAI embeddings into HotKV and Redis 8 and ran the same 1,000 queries against each, with 1 to 100 concurrent client connections, on one machine. Both held full-precision vectors, with the same index settings.
Both engines stored full-precision float32 vectors. After loading 999,000 vectors the HotKV process held 6.77 GB and Redis 7.01 GB.
Not measured: Redis's default 8-bit quantized mode, the Redis Query Engine, other datasets and other hardware. The HotKV build is a development build that has not been released. Redis was loaded pinned to four vCPUs and searched on a reloaded index (see the article).
| ef | HotKV recall | Redis recall | HotKV queries/s | Redis queries/s | Ratio |
|---|---|---|---|---|---|
| 40 | 0.940 | 0.923 | 63,897 | 8,061 | 7.9x |
| 80 | 0.970 | 0.962 | 37,508 | 6,825 | 5.5x |
| 150 | 0.987 | 0.980 | 21,912 | 5,241 | 4.2x |
| 300 | 0.993 | 0.990 | 12,393 | 3,454 | 3.6x |
| 500 | 0.997 | 0.994 | 7,688 | 2,483 | 3.1x |
HotKV build c8dafd37 and Redis 8.10.2, cosine similarity, HNSW with M=16 and ef_construction=200 on both, float32 on both (Redis NOQUANT). Recall is recall@10 against exact brute-force results. Median of 3 runs on Azure Standard_D160alds_v7, AMD EPYC 9V45, 160 vCPUs. The servers shared 128 vCPUs and the load generators ran on 32 others, using the same client code for both.
Dataset: dbpedia-entities-openai-1M embeddings (OpenAI text-embedding-ada-002, 1,536 dimensions) by Kumar Shivendu and Nirant Kasliwal, licensed MIT, plus 1,000 held-out queries. Full method, scripts and raw results. Measured October 14, 2026.
HotKV is commercial software. Let us run it for you, or run it yourself on infrastructure you control.
A fully managed HotKV instance, hosted and operated by us on Google Cloud across 42 regions. Opening in waves. Request early access.
Run HotKV on your own servers or your own cloud account, under a license file, with full control over data residency and network access.
Twelve official client libraries. HotKV also speaks RESP2 and RESP3, so your existing Redis clients and tools connect without changes beyond the hostname.
pip install hotkv-clientnpm install @hotkv/clientgo get github.com/hotkv-ltd/hotkv-go-clientcargo add hotkv-clientcom.hotkv:hotkv-clientdotnet add package HotKV.Clientgem install hotkvcomposer require hotkv/client{:hotkv, "~> 1.0"}from GitHub (hotkv-perl-client)libhotkv (CMake, GitHub release)hotkv-cpp (CMake, GitHub release)The Python and Node.js SDKs ship one-line integrations for the frameworks teams already build with. Separate open-source tools cover LangGraph, MCP and OpenAI-compatible APIs.
pip install "hotkv-client[langchain]"from hotkv.integrations.langchain import HotKVSemanticCache set_llm_cache(HotKVSemanticCache(client, embedding=embeddings))
Also includes HotKVCache for exact-match caching, plus HotKVVectorStore and HotKVChatMessageHistory.
pip install "hotkv-client[llamaindex]"from hotkv.integrations.llama_index import HotKVVectorStore
Also includes HotKVKVStore and HotKVChatStore.
pip install "hotkv-client[litellm]"from hotkv.integrations.litellm import HotKVSemanticCache
LiteLLM's built-in Redis cache already works with HotKV unchanged. Reach for this integration when you also want semantic caching.
import { hotkvCache } from "@hotkv/client/ai-sdk";
wrapLanguageModel({ model, middleware: hotkvCache({ client }) });Ships in the @hotkv/client npm package.
import { HotKVCache } from "@hotkv/client/langchain";Ships in the @hotkv/client npm package.
pip install langgraph-checkpoint-hotkvfrom langgraph.checkpoint.hotkv import HotKVSaver from langgraph.store.hotkv import HotKVStore
A checkpointer for graph state and a long-term memory store with semantic search. Needs no HotKV license.
Read the docspip install hotkv-mcpclaude mcp add hotkv -- uvx hotkv-mcp --url hotkv://localhost:6379
Lets Claude, Cursor and other assistants use keys, agent memory, prompts and document search. Read-only mode and namespaces keep agents in bounds.
Read the docsgo install github.com/hotkv-ltd/hotkv-gateway/cmd/hotkv-gateway@latestclient = OpenAI(base_url="http://localhost:8080/v1")
A drop-in proxy that caches OpenAI-compatible and Anthropic API calls, streaming included. Change one URL, no code.
Read the docsDig into what makes HotKV fast, where it fits, and how to get started.
Multi-threaded from day one. Throughput that scales with your core count, not against it.
Learn moreSemantic cache, RAG, agent memory, and LLM cost controls built into the engine.
Learn moreCaching, sessions, queues, leaderboards. The same workloads as Redis, at higher throughput.
Learn moreKey tagging, nanosecond TTLs, vector search, and built-in metrics, part of the core engine.
Learn moreA multi-threaded engine where every core reads, parses and executes commands in parallel.
Learn moreStarter, Performance, and Scale plans on HotKV Cloud, with bundled traffic and no annual lock-in.
Learn more