Links

Automatic Retries

LLM APIs often have inexplicable failures. With Portkey, you can rescue a substantial number of your requests with our in-built automatic retries feature.
  • Automatic retries are triggered up to 5 times
  • Retries can also be triggered only on specific error codes
  • And each subsequent retry attempt follows exponential backoff strategy to prevent network overload

Enabling Retries

To enable retry, just add the retry param to your config object.

Retry with 5 attempts

{
"retry": {
"attempts": 5
},
"virtual_key": "virtual-key-xxx"
}

Retry only on specific error codes

By default, Portkey triggers retries on the following error codes: [429, 500, 502, 503, 504]
You can change this behaviour by setting the optional on_status_codes param in your retry config and manually inputting the error codes on which rety will be triggered.
{
"retry": {
"attempts": 3,
"on_status_codes": [ 408, 429, 401 ]
},
"virtual_key": "virtual-key-xxx"
}
If the on_status_codes param is present, retries will be triggered only on the error codes specified in that Config and not on Portkey's default error codes for retries (i.e. [429, 500, 502, 503, 504])

Exponential backoff strategy

Here's how Portkey triggers retries following exponential backoff:
Attempt
Time out between requests
Initial Call
Immediately
Retry 1st attempt
1 second
Retry 2nd attempt
2 seconds
Retry 3rd attempt
4 seconds
Retry 4th attempt
8 seconds
Retry 5th attempt
16 seconds