Comment on page
Configs streamline the management and setting of all Portkey features, enabling you to programmatically control various aspects like fallbacks, load balancing, retries, caching, and more - all through a single interface. Access and adjust these settings via the Portkey UI and invoke them in your LLM code through the REST API. (SDK support is coming soon.)
Navigate to the ‘Configs’ page in the Portkey app to set and manage your Configs. Start typing, and you will see type hints and suggestions for the keys.
retry- Set the automatic retry count. Minimum retry count is 0, and maximum is 5.
mode- Set the orchestrattion method for your reuqests.
- "fallback": Enable model fallback for primary failure scenarios.
- "loadbalance": Distribute your request load among multiple providers or accounts.
- "ab_test": Run A/B tests programmatically on various model parameters.
- "single": Opt for standard orchestration.
options- Required when the mode is other than "single", allowing you to define model logic for fallback, load balance, or A/B test.
override_paramsaccepts JSON, letting you override model parameters set during completion calls, crucial for optimizing outputs when orchestrating between distinct models like Claude-2 & GPT-3.5.
Here's an example Config putting everything together, that sets fallback between gpt 3.5 & gpt 4:
You can add upto 25 options in your Portkey config. (Each option can have same or different providers, models, and other params)