AWS
This enterprise-focused document provides comprehensive instructions for deploying the Portkey software on AWS, tailored to meet the needs of large-scale, mission-critical applications.
It includes specific recommendations for component sizing, high availability, disaster recovery, and integration with monitoring systems.
Components and Sizing Recommendations
Component | Options | Sizing Recommendations |
---|---|---|
AI Gateway | Deploy as a Docker container in your Kubernetes cluster using Helm Charts | AWS EC2 t4g.medium instance, with at least 4GiB of memory and two vCPUs For high reliability, deploy across multiple Availability Zones. |
Logs Store (optional) | Hosted MongoDB, Document DB or AWS S3 | Each log document is ~10kb in size (uncompressed) |
Cache (Prompts, Configs & Virtual Keys) | Elasticache or self-hosted Redis | Deploy in the same VPC as the Portkey Gateway. |
Deployment Steps
Prerequisites
Ensure the following tools are installed:
- Docker
- Kubectl
- Helm (v3 or above)
Step 1: Clone the Portkey repo containing helm chart
git clone https://github.com/Portkey-AI/helm-chart
Step 2: Update values.yaml for Helm
Modify the values.yaml file in the Helm chart directory to include the Docker registry credentials and necessary environment variables. You can find the sample file at ./helm-chart/helm/enterprise/values.yaml
Image Credentials Configuration
imageCredentials:
name: portkey-enterprise-registry-credentials
create: true
registry: kubernetes.io/dockerconfigjson
username:
password:
The Portkey team will share the credentials for your image
Environment Variables Configuration
environment:
...
data:
SERVICE_NAME:
LOG_STORE:
MONGO_DB_CONNECTION_URL:
MONGO_DATABASE:
MONGO_COLLECTION_NAME:
LOG_STORE_REGION:
LOG_STORE_ACCESS_KEY:
LOG_STORE_SECRET_KEY:
LOG_STORE_GENERATIONS_BUCKET:
ANALYTICS_STORE:
ANALYTICS_STORE_ENDPOINT:
ANALYTICS_STORE_USER:
ANALYTICS_STORE_PASSWORD:
ANALYTICS_LOG_TABLE:
ANALYTICS_FEEDBACK_TABLE:
CACHE_STORE:
REDIS_URL:
REDIS_TLS_ENABLED:
PORTKEY_CLIENT_AUTH:
ORGANISATIONS_TO_SYNC:
Notes on the Log Store LOG_STORE
can be
- an S3 compatible store (AWS S3
s3
, GCSgcs
, Wasabiwasabi
) - or a MongoDB store (Hosted MongoDB
mongo
, AWS DocumentDBmongo
)
If the LOG_STORE
is mongo
, the following environment variables are needed
MONGO_DB_CONNECTION_URL:
MONGO_DATABASE:
MONGO_COLLECTION_NAME:
If the LOG_STORE
is s3
or wasabi
or gcs
, the following values are mandatory
LOG_STORE_REGION:
LOG_STORE_ACCESS_KEY:
LOG_STORE_SECRET_KEY:
LOG_STORE_GENERATIONS_BUCKET:
All the above mentioned are S3 Compatible document storages and interoperable with S3 API. You need to generate the Access Key
and Secret Key
from the respective providers.
Notes on Cache If CACHE_STORE
is set as redis
, a redis instance will also get deployed in the cluster. If you are using custom redis, then leave it blank.
The following values are mandatory
REDIS_URL:
REDIS_TLS_ENABLED:
REDIS_URL
defaults to redis://redis:6379
and REDIS_TLS_ENABLED
defaults to false
.
Notes on Analytics Store This is hosted in Portkey’s control plane and these credentials will be shared by the Portkey team.
The following are mandatory and are shared by the Portkey Team.
PORTKEY_CLIENT_AUTH:
ORGANISATIONS_TO_SYNC:
Step 3: Deploy using Helm Charts
Navigate to the directory containing your Helm chart and run the following command to deploy the application:
helm install portkey-gateway ./helm/enterprise --namespace portkeyai --create-namespace
This command installs the Helm chart into the portkeyai
namespace.
Step 4: Verify the Deployment
Check the status of your deployment to ensure everything is running correctly:
kubectl get pods -n portkeyai
Step 5: Port Forwarding (Optional)
To access the service over internet, use port forwarding:
kubectl port-forward -n portkeyai 443:8787
Replace <pod-name>
with the name of your pod.
Uninstalling the Deployment
If you need to remove the deployment, run:
helm uninstall portkey-app --namespace portkeyai
This command will uninstall the Helm release and clean up the resources.
Network Configuration
Step 1: Allow access to the service
To make the service accessible from outside the cluster, define a Service of type LoadBalancer in your values.yaml or Helm templates. Specify the desired port for external access.
service:
type: LoadBalancer
port:
targetPort: 8787
Replace <desired_port>
with the port number for external access with the port the application listens on internally.
Step 2: Ensure Outbound Network Access
By default, Kubernetes allows full outbound access, but if your cluster has NetworkPolicies that restrict egress, configure them to allow outbound traffic.
Example NetworkPolicy for Outbound Access:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-egress
namespace: portkeyai
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
This allows the gateway to access LLMs hosted within your VPC and outside as well. This also enables connection for the sync service to the Portkey Control Plane.
Step 3: Configure Inbound Access for Portkey Control Plane
Ensure the Portkey control plane can access the service either over the internet or through VPC peering.
Over the Internet:
- Ensure the LoadBalancer security group allows inbound traffic on the specified port.
- Document the public IP/hostname and port for the control plane connection.
Through VPC Peering:
Set up VPC peering between your AWS account and the control plane’s AWS account. Requires manual setup by Portkey Team.
Required Permissions
To ensure the smooth operation of Portkey AI in your private cloud deployment on AWS, specific permissions are required based on the type of log store you are using. Below are the details for S3 or MongoDB compliant databases.
S3 Bucket
If using S3 as the log store, the following IAM policy permissions are required:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME",
"arn:aws:s3:::YOUR_BUCKET_NAME/*"
]
}
]
}
Please replace YOUR_BUCKET_NAME
with your actual bucket name.
MongoDB Compliant Database
If using a MongoDB compliant database, ensure the AI Gateway has access to the database. The database user should have following role:
{
"roles": [
{
"role": "readWrite",
"db": "YOUR_DATABASE_NAME"
}
]
}
The readWrite
role provides the necessary read and write access to the specified database. Please replace YOUR_DATABASE_NAME
with your actual database name.
Cache Store - Redis
The Portkey Gateway image ships with a redis installed. You can choose to use the inbuilt redis or connect to an outside Redis instance.
- Redis as Part of the Image: No additional permissions or networking configurations are required.
- Separate Redis Instance: The gateway requires permissions to perform read and write operations on the Redis instance. The redis connections can be configured with or without TLS.
Was this page helpful?