Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including your locally hosted models through Ollama.

Provider Slug. ollama

Portkey SDK Integration with Ollama Models

Portkey provides a consistent API to interact with models from various providers. To integrate Ollama with Portkey:

1. Install the Portkey SDK

Install the Portkey SDK in your application to interact with your Ollama API through Portkey.

npm install --save portkey-ai

2. Initialize Portkey with Ollama URL

Instantiate the Portkey client by adding your Ollama publicly-exposed URL to the customHost property.

import Portkey from 'portkey-ai'

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    provider: "ollama",
    customHost: "https://7cc4-3-235-157-146.ngrok-free.app" // Your Ollama ngrok URL
})

For the Ollama integration, you only need to pass the base URL to **customHost** without the version identifier (such as **/v1**) - Portkey takes care of it for Ollama.

Requests made to your localhost Ollama endpoints will fail. To integrate with Portkey and to observe your requests, you’ll need to expose your localhost URLs publicly via a service like ngrok.

3. Invoke Chat Completions with Ollama

Use the Portkey SDK to invoke chat completions from your Ollama model, just as you would with any other provider.

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'llama3',
});

console.log(chatCompletion.choices);

Next Steps

Explore the complete list of features supported in the SDK:

SDK


You’ll find more information in the relevant sections:

  1. Add metadata to your requests
  2. Add gateway configs to your Ollama requests
  3. Tracing Ollama requests
  4. Setup a fallback from OpenAI to Ollama APIs