Cache (Simple & Semantic)

Speed up and save money on your LLM requests by storing past responses in the Portkey cache. There are 2 cache modes:

  • Simple: Matches requests verbatim. Perfect for repeated, identical prompts. Works on all models including image generation models.

  • Semantic: Matches responses for requests that are semantically similar. Ideal for denoising requests with extra prepositions, pronouns, etc. Works on any model available on /chat/completions or /completions routes.

Portkey cache serves requests upto 20x times faster and cheaper.

Enable Cache in the Config

To enable Portkey cache, just add the cache params to your config object.

Simple Cache

"cache": { "mode": "simple" }

How it works: Simple cache performs an exact match on the input prompts. If the exact same request is received again, Portkey retrieves the response directly from the cache, bypassing the model execution.

Semantic Cache

"cache": { "mode": "semantic" }

How it works: Semantic cache considers the contextual similarity between input requests. It uses cosine similarity to ascertain if the similarity between the input and a cached request exceeds a specific threshold. If the similarity threshold is met, Portkey retrieves the response from the cache, saving model execution time. Check out this blog for more details on how we do this.

Semantic cache is a "superset" of both caches. Setting cache mode to "semantic" will work for when there are simple cache hits as well.

To optimise for accurate cache hit rates, Semantic cache only works with requests with less than 8,191 input tokens, and with number of messages (human, assistant, system combined) less than or equal to 4.

Read more how to set cache in Configs.

Setting Cache Age

You can set the age (or "ttl") of your cached response with this setting. Cache age is also set in your Config object:

"cache": { 
    "mode": "semantic",
    "max_age": 60

In this example, your cache will automatically expire after 60 seconds. Cache age is set in seconds.

  • Minimum cache age is 60 seconds

  • Maximum cache age is 90 days (i.e. 7776000 seconds)

  • Default cache age is 7 days (i.e. 604800 seconds)

Force Refresh Cache

Ensure that a new response is fetched and stored in the cache even when there is an existing cached response for your request. Cache force refresh can only be done at the time of making a request, and it is not a part of your Config.

You can enable cache force refresh with this header:

"x-portkey-cache-force-refresh": "True"
curl \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-virtual-key: open-ai-xxx" \
  -H "x-portkey-config: cache-config-xxx" \
  -H "x-portkey-cache-force-refresh: true" \
  -d '{
    "messages": [{"role": "user","content": "Hello!"}]
  • Cache force refresh is only activated if a cache config is also passed along with your request. (setting cacheForceRefresh as true without passing the relevant cache config will not have any effect)

  • For requests that have previous semantic hits, force refresh is performed on ALL the semantic matches of your request.

Cache in Analytics

Portkey shows you powerful stats on cache usage on the Analytics page. Just head over to the Cache tab, and you will see:

  • Your raw number of cache hits as well as daily cache hit rate

  • Your average latency for delivering results from cache and how much time it saves you

  • How much money the cache saves you

Cache in Logs

On the Logs page, the cache status is updated on the Status column. You will see Cache Disabled when you are not using the cache, and any of Cache Miss, Cache Refreshed, Cache Hit, Cache Semantic Hit based on the cache hit status. Read more here.

For each request we also calculate and show the cache response time and how much money you saved with each hit.

How Cache works with Configs

You can set cache at two levels:

  • Top-level that works across all the targets.

  • Target-level that works when that specific target is triggered.

  "cache": {"mode": "semantic", "max_age": 60},
  "strategy": {"mode": "fallback"},
  "targets": [
    {"virtual_key": "openai-key-1"},
    {"virtual_key": "openai-key-2"}

You can also set cache at both levels (top & target).

In this case, the target-level cache setting will be given preference over the top-level cache setting. You should start getting cache hits from the second request onwards for that specific target.

If any of your targets have override_params then cache on that target will not work until that particular combination of params is also stored with the cache. If there are no override_params for that target, then cache will be active on that target even if it hasn't been triggered even once.

Last updated