Comment on page


Configs streamline the management and setting of all Portkey features, enabling you to programmatically control various aspects like fallbacks, load balancing, retries, caching, and more - all through a single interface. Access and adjust these settings via the Portkey UI and invoke them in your LLM code through the REST API. (SDK support is coming soon.)

Setting Configs

Navigate to the ‘Configs’ page in the Portkey app to set and manage your Configs. Start typing, and you will see type hints and suggestions for the keys.

There, you can set the following features:

Expected Value
Integer [0,5]
"simple", "semantic"
"single", "fallback", "loadbalance", "ab_test"
Array of LLM options
  • retry - Set the automatic retry count. Minimum retry count is 0, and maximum is 5.
  • cache - Enable caching. Portkey provides two types of caches - simple & semantic. (More on that here)
  • mode - Set the orchestrattion method for your reuqests.
    • "fallback": Enable model fallback for primary failure scenarios.
    • "loadbalance": Distribute your request load among multiple providers or accounts.
    • "ab_test": Run A/B tests programmatically on various model parameters.
    • "single": Opt for standard orchestration.
  • options - Required when the mode is other than "single", allowing you to define model logic for fallback, load balance, or A/B test.

In options, you can set,

Expected Value
"anthropic", "openai", "cohere", "anyscale", etc
Your provider API key
Virtual key for any provider from your Portkey dashboard
Required in case of "loadbalance" or "ab_test" mode to set the weight for that particular LLM. Should sum to 1.
To set model params like temperature, top_p, max_tokens, model name, etc. Expects a JSON
  • override_params accepts JSON, letting you override model parameters set during completion calls, crucial for optimizing outputs when orchestrating between distinct models like Claude-2 & GPT-3.5.

In override_params you can set,

Expected Value
Model name based on the provider, like "gpt-3.5-turbo", "gpt-4", "claude-2", etc
All other model params like top_p, temperature, max_tokens, etc
Refer to the provider spec for more info on the params
Messages array input for chat completion models
Text string input for completion models

Example Config

Here's an example Config putting everything together, that sets fallback between gpt 3.5 & gpt 4:
"mode": "fallback",
"options": [
"provider": "openai",
"virtual_key": "open-ai-key-aaa",
"override_params": {
"model": "gpt-3.5-turbo"
"provider": "openai",
"virtual_key": "open-ai-key-aaa",
"override_params": {
"model": "gpt-4"

Maximum Number of Options

You can add upto 25 options in your Portkey config. (Each option can have same or different providers, models, and other params)