Comment on page
📝
Configs
Configs streamline the management and setting of all Portkey features, enabling you to programmatically control various aspects like fallbacks, load balancing, retries, caching, and more - all through a single interface. Access and adjust these settings via the Portkey UI and invoke them in your LLM code through the REST API. (SDK support is coming soon.)
Navigate to the ‘Configs’ page in the Portkey app to set and manage your Configs. Start typing, and you will see type hints and suggestions for the keys.
Key | Expected Value |
|---|---|
retry | Integer [0,5] |
cache | "simple", "semantic" |
mode | "single", "fallback", "loadbalance", "ab_test" |
options | Array of LLM options |
retry- Set the automatic retry count. Minimum retry count is 0, and maximum is 5.cache- Enable caching. Portkey provides two types of caches - simple & semantic. (More on that here)mode- Set the orchestrattion method for your reuqests.- "fallback": Enable model fallback for primary failure scenarios.
- "loadbalance": Distribute your request load among multiple providers or accounts.
- "ab_test": Run A/B tests programmatically on various model parameters.
- "single": Opt for standard orchestration.
options- Required when the mode is other than "single", allowing you to define model logic for fallback, load balance, or A/B test.
Key | Expected Value |
|---|---|
provider | "anthropic", "openai", "cohere", "anyscale", etc |
api_key | Your provider API key |
virtual_key | Virtual key for any provider from your Portkey dashboard |
weight | Required in case of "loadbalance" or "ab_test" mode to set the weight for that particular LLM. Should sum to 1. |
override_params | To set model params like temperature, top_p, max_tokens, model name, etc. Expects a JSON |
override_paramsaccepts JSON, letting you override model parameters set during completion calls, crucial for optimizing outputs when orchestrating between distinct models like Claude-2 & GPT-3.5.
Key | Expected Value |
|---|---|
model | Model name based on the provider, like "gpt-3.5-turbo", "gpt-4", "claude-2", etc |
All other model params like top_p, temperature, max_tokens, etc | Refer to the provider spec for more info on the params |
messages | Messages array input for chat completion models |
prompt | Text string input for completion models |
Here's an example Config putting everything together, that sets fallback between gpt 3.5 & gpt 4:
{
"mode": "fallback",
"options": [
{
"provider": "openai",
"virtual_key": "open-ai-key-aaa",
"override_params": {
"model": "gpt-3.5-turbo"
}
},
{
"provider": "openai",
"virtual_key": "open-ai-key-aaa",
"override_params": {
"model": "gpt-4"
}
}
]
}
You can add upto 25 options in your Portkey config. (Each option can have same or different providers, models, and other params)
Last modified 24d ago