Guardrails

LLMs are brittle - not just in API uptimes or their inexplicable 400/500 errors, but also in their core behavior. You can get a response with a 200 status code that completely errors out for your app's pipeline due to mismatched output. With Portkey's Guardrails, we now help you enforce LLM behavior in real-time with our Guardrails on the Gateway pattern.

Using Portkey's Guardrail platform, you can now verify your LLM inputs AND outputs to be adhering to your specifed checks; and since Guardrails are built on top of our Gateway, you can orchestrate your request exactly the way you want - with actions ranging from denying the request, logging the guardrail result, creating an evals dataset, falling back to another LLM or prompt, retrying the request, and more.

Examples of Guardrails Portkey offers:

• Regex match - Check if the request or response text matches a regex pattern

• JSON Schema - Check if the response JSON matches a JSON schema

• Contains Code - Checks if the content contains code of format SQL, Python, TypeScript, etc.

• Custom guardrail - If you are running a custom guardrail currently, you can also integrate it with Portkey

• ...and many more.

Portkey currently offers 20+ deterministic guardrails like the ones described above as well as LLM-based guardrails like Detect Gibberish, Scan for prompt injection, and more. These guardrails serve as protective barriers that help mitigate risks associated with Gen AI, ensuring its responsible and ethical deployment within organizations.

Browse the full list of supported Guardrail checks here.

Using Guardrails

Putting Portkey Guardrails in production is just a 4-step process:

1. Create Guardrail Checks

2. Create Guardrail Actions

3. Enable Guardrail through Configs

4. Attach the Config to a Request

This flowchart shows how Portkey processes a Guardrails request:

Let's see in detail below:

1. Create a New Guardrail & Add Checks

On the "Guardrails" page, click on Create and add your preferred Guardrail checks from the right sidebar.

Each Guardrail Check has a custom input field based on its usecase — just add the relevant details to the form and save your check.

Here is a list of all the Guardrail checks available on Portkey and what they do.

2. Add Guardrail Actions

This is where you will define a basic orchestration logic for your Guardrail.

There are 6 Types of Guardrail Actions

Set the relevant actions you want with your checks, name your Guardrail and save it! When you save the Guardrail, you will get an associated $Guardrail_ID that you can then add to your request.

3. "Enable" the Guardrails through Configs

This is where Portkey's magic comes into play. The Guardrail you created above is yet not an Active guardrail because it is not attached to any request.

Configs is one of Portkey's most powerful features and is used to define all kinds of request orchestration - everything from caching, retries, fallbacks, timeouts, to load balancing.

Add Guardrail ID before the request OR after the request

Example Config with Guardrails

{
	"retry": {
		"attempts": 3
	},
	"cache": {
		"mode": "simple"
	},
	"virtual_key":"openai-xxx",
	"before_request_hooks": [{
		"id": "input-guardrail-id-xx"
	}],
	"after_request_hooks": [{
		"id": "output-guardrail-id-xx"
	}]
}

Guardrail Behaviour on the Gateway

For asynchronous guardrails (async=TRUE), Portkey returns the standard, default status codes from the LLM providers — this is because the Guardrails verdict is not affecting how you orchestrate your requests. Portkey will only log the Guardrail result for you.

But for synchronous requests (async=FALSE), Portkey can orchestrate your requests based on the Guardrail verdict. The behaviour is dependent on the following:

• Guardrail Check Verdict (PASS or FAIL) AND

• Guardrail Action — DENY Setting (TRUE or FALSE)

Portkey sends different request status codes corresponding to your set Guardrail behaviour.

Example Config Using the New 246 & 446 Status Codes

{
	"strategy": {
		"mode": "fallback",
		"on_status_codes": [246,446]
	},
	"targets": [
		{"virtual_key": "openai-key-xxx"},
		{"virtual_key": "anthropic-key-xxx"}
	],
	"before_request_hooks": [
		{"id": "guardrails-id-xxx"}
	]
}

{
	"retry": {
		"on_status_codes": [246],
		"attempts": 5
	},
	"after_request_hooks": [
		{"id": "guardrails-id"}
	]
}

You can create these Configs in Portkey UI, save them, and get an associated Config ID you can attach to your requests. More here.

4. Final Step - Attach Config to Request

Now, while instantiating your Portkey client or while sending headers, just pass the Config ID.

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    config: "pc-***" // Supports a string config id or a config object
});

const portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    config="pc-***" # Supports a string config id or a config object
)

const openai = new OpenAI({
  apiKey: 'OPENAI_API_KEY',
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY",
    config: "CONFIG_ID"
  })
});

client = OpenAI(
    api_key="OPENAI_API_KEY", # defaults to os.environ.get("OPENAI_API_KEY")
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="openai",
        api_key="PORTKEY_API_KEY", # defaults to os.environ.get("PORTKEY_API_KEY")
        config="CONFIG_ID"
    )
)

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-config: $CONFIG_ID" \ 
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{
        "role": "user",
        "content": "Hello!"
      }]
  }'

For more, refer to the Config documentation here.

Viewing Guardrail Results in Portkey Logs

Portkey Logs will show you detailed information about Guardrail results for each request.

On the Feedback & Guardrails tab on the log drawer, you can see

Guardrail Details

• Overview: How many checks passed and how many failed

• Verdict: Guardrail verdict for each of the checks in your Guardrail

• Latency: Round trip time for each check in your Guardrail

Feedback Details

Portkey will also show the feedback object logged for each request

• Value: The numerical feedback value you passed

• Weight: The numerical feedback weight

• Metadata Key & Value: Any custom metadata sent with the feedback

• successfulChecks: Which checks associated with this request passed

• failedChecks: Which checks associated with this request failed

• erroredChecks: If there were any checks that errored out along the way

Defining Guardrails Directly in JSON

On Portkey, you can also create the Guardrails in code and add them to your Configs. Read more about this here:

Bring Your Own Guardrails

If you already have a custom guardrail pipeline where you send your inputs/outputs for evaluation, you can also integrate it with Portkey using a modular, custom webhook! Read more here:

Examples of When to Deny Requests with Guardrails

1. Prompt Injection Checks: Preventing inputs that could alter the behavior of the AI model or manipulate its responses.

2. Moderation Checks: Ensuring responses do not contain offensive, harmful, or inappropriate content.

3. Compliance Checks: Verifying that inputs and outputs comply with regulatory requirements or organizational policies.

4. Security Checks: Blocking requests that contain potentially harmful content, such as SQL injection attempts or cross-site scripting (XSS) payloads.

By appropriately configuring Guardrail Actions, you can maintain the integrity and reliability of your AI app, ensuring that only safe and compliant requests are processed.