Guardrails
Ship to production more confidently with Portkey Guardrails on your requests & responses
This feature is in beta and available to select users. To enable it for your org, ping us on the Portkey Discord.
LLMs are brittle - not just in API uptimes or their inexplicable 400
/500
errors, but also in their core behavior. You can get a response with a 200
status code that completely errors out for your app's pipeline due to mismatched output. With Portkey's Guardrails, we now help you enforce LLM behavior in real-time with our Guardrails on the Gateway pattern.
Using Portkey's Guardrail platform, you can now verify your LLM inputs AND outputs to be adhering to your specifed checks; and since Guardrails are built on top of our Gateway, you can orchestrate your request exactly the way you want - with actions ranging from denying the request, logging the guardrail result, creating an evals dataset, falling back to another LLM or prompt, retrying the request, and more.
Examples of Guardrails Portkey offers:
Regex match - Check if the request or response text matches a regex pattern
JSON Schema - Check if the response JSON matches a JSON schema
Contains Code - Checks if the content contains code of format SQL, Python, TypeScript, etc.
Custom guardrail - If you are running a custom guardrail currently, you can also integrate it with Portkey
...and many more.
Portkey currently offers 20+ deterministic guardrails like the ones described above as well as LLM-based guardrails like Detect Gibberish
, Scan for prompt injection
, and more. These guardrails serve as protective barriers that help mitigate risks associated with Gen AI, ensuring its responsible and ethical deployment within organizations.
Portkey also integrates with your favourite Guardrail platforms like Aporia, SydeLabs, Pillar Security and more. Just add their API keys to Portkey and you can enable their guardrails policies on your Portkey calls! More details on Guardrail Partners here.
Using Guardrails
Putting Portkey Guardrails in production is just a 4-step process:
Create Guardrail Checks
Create Guardrail Actions
Enable Guardrail through Configs
Attach the Config to a Request
This flowchart shows how Portkey processes a Guardrails request:
Let's see in detail below:
1. Create a New Guardrail & Add Checks
On the "Guardrails" page, click on Create
and add your preferred Guardrail checks from the right sidebar.
On Portkey, you can configure Guardrails to be run on either the INPUT
(i.e. PROMPT
) or the OUTPUT
.
Hence, for the Guardrail you create, make sure your Guardrail is only validating ONLY ONE OF the Input or the Output.
Each Guardrail Check has a custom input field based on its usecase — just add the relevant details to the form and save your check.
A check is an individual function that assesses the input prompt or output response against predefined conditions.
You can add as many checks as you want to a single Guardrail.
A check ONLY returns a boolean (
Yes
/No
) verdict.
Here is a list of all the Guardrail checks available on Portkey and what they do.
2. Add Guardrail Actions
This is where you will define a basic orchestration logic for your Guardrail.
Guardrail is created to validate ONLY ONE OF the Input
or the Output
. The Actions set here will also apply only to either the request
or the response
.
There are 6 Types of Guardrail Actions
Action | State | Description | Impact |
---|---|---|---|
Async |
| Run the Guardrail checks asynchronously along with the LLM request. |
|
Async |
|
|
|
Deny |
|
|
|
Deny |
|
|
|
On Success |
| If all of the Guardrail checks |
|
On Failure |
| If any of the Guardrail checks |
|
Set the relevant actions you want with your checks, name your Guardrail and save it! When you save the Guardrail, you will get an associated $Guardrail_ID
that you can then add to your request.
3. "Enable" the Guardrails through Configs
This is where Portkey's magic comes into play. The Guardrail you created above is yet not an Active
guardrail because it is not attached to any request.
Configs is one of Portkey's most powerful features and is used to define all kinds of request orchestration - everything from caching, retries, fallbacks, timeouts, to load balancing.
Now, you can use Configs to add Guardrail checks & actions to your request.
Add Guardrail ID before the request
OR after the request
before the request
OR after the request
Type | Config Key | Value | Description |
---|---|---|---|
Before Request Hook |
| [ | This key is used to run Guardrail |
After Request Hook |
| [ | This key is used to run Guardrail |
Example Config with Guardrails
Guardrail Behaviour on the Gateway
For asynchronous guardrails (async=
TRUE
), Portkey returns the standard, default status codes from the LLM providers — this is because the Guardrails verdict is not affecting how you orchestrate your requests. Portkey will only log the Guardrail result for you.
But for synchronous requests (async=
FALSE
), Portkey can orchestrate your requests based on the Guardrail verdict. The behaviour is dependent on the following:
Guardrail Check Verdict (
PASS
orFAIL
) ANDGuardrail Action — DENY Setting (
TRUE
orFALSE
)
Portkey sends different request status codes
corresponding to your set Guardrail behaviour.
For requests where async=
FALSE
Guardrail Verdict | DENY Setting | Returned Status Code | Description |
---|---|---|---|
|
|
| Guardrails have passed, request will be processed regardless |
|
|
| Guardrails have passed, request will be processed regardless |
|
|
| Guardrails have failed, but the request should still be processed. Portkey introduces a new Status code to indicate this state. |
|
|
| Guardrails have failed, and the request should not be processed. Portkey introduces a new Status code to indicate this state. |
Example Config Using the New 246
& 446
Status Codes
246
& 446
Status CodesYou can create these Configs in Portkey UI, save them, and get an associated Config ID you can attach to your requests. More here.
4. Final Step - Attach Config to Request
Now, while instantiating your Portkey client or while sending headers, just pass the Config ID.
For more, refer to the Config documentation here.
Viewing Guardrail Results in Portkey Logs
Portkey Logs will show you detailed information about Guardrail results for each request.
On the Feedback & Guardrails
tab on the log drawer, you can see
Feedback & Guardrails
tab on the log drawer, you can seeGuardrail Details
Overview: How many checks
passed
and how manyfailed
Verdict: Guardrail verdict for each of the checks in your Guardrail
Latency: Round trip time for each check in your Guardrail
Feedback Details
Portkey will also show the feedback object logged for each request
Value
: The numerical feedback value you passedWeight
: The numerical feedback weightMetadata Key & Value
: Any custom metadata sent with the feedbacksuccessfulChecks
: Which checks associated with this requestpassed
failedChecks
: Which checks associated with this requestfailed
erroredChecks
: If there were any checks that errored out along the way
Defining Guardrails Directly in JSON
On Portkey, you can also create the Guardrails in code and add them to your Configs. Read more about this here:
Creating Raw Guardrails (in JSON)Bring Your Own Guardrails
If you already have a custom guardrail pipeline where you send your inputs/outputs for evaluation, you can also integrate it with Portkey using a modular, custom webhook! Read more here:
Bring Your Own GuardrailsExamples of When to Deny Requests with Guardrails
Prompt Injection Checks: Preventing inputs that could alter the behavior of the AI model or manipulate its responses.
Moderation Checks: Ensuring responses do not contain offensive, harmful, or inappropriate content.
Compliance Checks: Verifying that inputs and outputs comply with regulatory requirements or organizational policies.
Security Checks: Blocking requests that contain potentially harmful content, such as SQL injection attempts or cross-site scripting (XSS) payloads.
By appropriately configuring Guardrail Actions, you can maintain the integrity and reliability of your AI app, ensuring that only safe and compliant requests are processed.
Last updated