Tracing

This feature is available for all plans:-

• Developer: 10k Logs / Month with 3 day Log Retention

• Production: 100k Logs / Month + $9 for additional 100k with 30 Days Log Retention

• Enterprise: Unlimited

The Tracing capabilities in Portkey empowers you to monitor the lifecycle of your LLM requests in a unified, chronological view.

This is perfect for agentic workflows, chatbots, or multi-step LLM calls, by helping you understand and optimize your AI application's performance.

How Tracing Works

Portkey implements OpenTelemetry-compliant tracing. When you include a trace ID with your requests, all related LLM calls are grouped together in the Traces View, appearing as "spans" within that trace.

"Span" is another word for subgrouping of LLM calls. Based on how you instrument, it can refer to another group within your trace or to a single LLM call.

Trace Tree Structure

Portkey uses a tree data structure for tracing, similar to OTel.

Each node in the tree is a span with a unique spanId and optional spanName. Child spans link to a parent via the parentSpanId. Parentless spans become root nodes.

traceId
├─ parentSpanId
│  ├─ spanId
│  ├─ spanName

Key - Node	Key - Python	Expected Value	Required?
traceId	trace_id	Unique string	YES
spanId	span_id	Unique string	NO
spanName	span_name	string	NO
parentSpanId	parent_span_id	Unique string	NO

---

Enabling Tracing

You can enable tracing by passing the trace tree values while making your request (or while instantiating your client).

Based on these values, Portkey will instrument your requests, and will show the exact trace with its spans on the "Traces" view in Logs page.

NodeJS

Add tracing details to a single request (recommended)

const requestOptions = {
    traceId: "1729",
    spanId: "11",
    spanName: "LLM Call"
}

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'gpt-4o',
}, requestOptions);

Or, add trace details while instantiating your client

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    virtualKey: "VIRTUAL_KEY",
    traceId: "1729",
    spanId: "11",
    spanName: "LLM Call"
})

Python

completion = portkey.with_options(
    trace_id="1729",
    span_id="11",
    span_name="LLM Call"        
).chat.completions.create(
    messages = [{ "role": 'user', "content": 'Say this is a test' }],
    model = 'gpt-3.5-turbo'
)

Pass Trace details while instantiating your client

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    virtual_key="VIRTUAL_KEY",
    trace_id="1729",
    span_id="11",
    span_name="LLM Call"
)

OpenAI NodeJS

import { createHeaders } from 'portkey-ai'

const requestOptions = {
    traceId: "1729",
    spanId: "11",
    spanName: "LLM Call"
}

const chatCompletion = await openai.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'gpt-3.5-turbo',
}, requestOptions);

OpenAI Python

from portkey_ai import createHeaders

req_headers = createHeaders(
    trace_id="1729",
    span_id="11",
    span_name="LLM Call
)

chat_complete = client.with_options(headers=req_headers).chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Say this is a test"}],
)

REST API

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-provider: openai" \
  -H "x-portkey-trace-id: 1729"\
  -H "x-portkey-span-id: 11"\
  -H "x-portkey-span-name: LLM_CALL"\
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user","content": "Hello!"}]
  }'

If you are only passing trace ID and not the span details, you can set the trace ID while making your request or while instantiating your client.

NodeJS

const requestOptions = {traceID: "YOUR_TRACE_ID"}

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'gpt-4o',
}, requestOptions);

console.log(chatCompletion.choices);

Pass Trace ID while instantiating your client

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    virtualKey: "VIRTUAL_KEY",
    traceID: "TRACE_ID"
})

Python

completion = portkey.with_options(
    trace_id = "TRACE_ID"
).chat.completions.create(
    messages = [{ "role": 'user', "content": 'Say this is a test' }],
    model = 'gpt-3.5-turbo'
)

Pass Trace ID while instantiating your client

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    virtual_key="VIRTUAL_KEY",
    trace_id="TRACE_ID"
)

OpenAI NodeJS

import { createHeaders } from 'portkey-ai'

const reqHeaders = {headers: createHeaders({"traceID": "TRACE_ID"})}

const chatCompletion = await openai.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'gpt-3.5-turbo',
}, reqHeaders);

OpenAI Python

from portkey_ai import createHeaders

req_headers = createHeaders(trace_id="TRACE_ID")

chat_complete = client.with_options(headers=req_headers).chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Say this is a test"}],
)

REST API

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-provider: openai" \ 
  -H "x-portkey-trace-id: TRACE_ID" \
  -d '{
    "model": "gpt-4-turbo",
    "messages": [{
        "role": "system",
        "content": "You are a helpful assistant."
      },{
        "role": "user",
        "content": "Hello!"
      }]
  }'

See Tracing in Action

Tracing in Langchain

Portkey has a dedicated handler that can instrument your Langchain chains and agents to trace them.

Trace Langchain Requests (Python)

1. First, install Portkey SDK, and Langchain's packages

$ pip install langchain_OpenAI portkey-ai langchain_community

2. Import the packages

from langchain_openai import ChatOpenAI
from langchain.chains import LLMChain
from portkey_ai.langchain import LangchainCallbackHandler
from portkey_ai import createHeaders

3. Instantiate Portkey's Langchain Callback Handler

portkey_handler = LangchainCallbackHandler(
      api_key="YOUR_PORTKEY_API_KEY", 
      metadata={
            "user_name": "User_Name",
            "traceId": "Langchain_sample_callback_handler"
      }
)

4. Add the callback to the ChatOpenAI instance

llm = ChatOpenAI(
    api_key="OPENAI_API_KEY",
    callbacks=[portkey_handler],
)

5. Also add the callback when you define or run your LLM chain

chain = LLMChain(
    llm=llm,
    prompt=prompt,
    callbacks=[portkey_handler]
)

handler_config = {'callbacks' : [portkey_handler]}

chain.invoke({"input": "what is langchain?"}, config=handler_config)

---

Tracing Llamaindex Requests

Portkey has a dedicated handler to instrument your Llamaindex requests on Portkey.

Trace Llamaindex Requests

1. First, install Portkey SDK, and LlamaIndex packages

$ pip install openai portkey-ai llama-index 

2. Import the packages

from llama_index.llms.openai import OpenAI
from portkey_ai.llamaindex import LlamaIndexCallbackHandler

3. Instantiate Portkey's LlamaIndex Callback Handler

portkey_handler = LlamaIndexCallbackHandler(
      api_key="PORTKEY_API_KEY",
      metadata={
            "user_name": "User_Name",
            "traceId": "Llamaindex_sample_callback_handler"
      }
)

4. Add it to OpenAI llm class

llm = OpenAI(
    model="gpt-4o",
    api_key="OPENAI_API_KEY",
    callback_manager=[portkey_handler],
)

5. In Llama Index, you can also set the callback at a global level

from llama_index.core import Settings
from llama_index.core.callbacks import CallbackManager

Settings.callback_manager = CallbackManager([portkey_handler])
Settings.llm = llm

---

Inserting Logs

If you are using the Insert Log API to add logs to Portkey, your traceId, spanId etc. will become part of the metadata object in your log, and Portkey will instrument your requests to take those values into account.

The logger endpoint supports inserting a single log as well as log array, and helps you build traces of any depth or complexity. For more, check here:

Insert a Log

---

Tracing for Gateway Features

Tracing also works very well to capture the Gateway behavior on retries, fallbacks, and other routing mechanisms on Portkey Gateway.

Portkey automatically groups all the requests that were part of a single fallback or retry config and shows the failed and succeeded requests chronologically as "spans" inside a "trace".

This is especially useful when you want to understand the total latency and behavior of your app when retry or fallbacks were triggered.

For more, check out the Fallback & Automatic Retries docs.

---

Why Use Tracing?

• Cost Insights: View aggregate LLM costs at the trace level.

• Debugging: Easily browse all requests in a single trace and identify failures.

• Performance Analysis: Understand your entire request lifecycle and total trace duration.

• User Feedback Integration: Link user feedback to specific traces for targeted improvements.

---

Capturing User Feedback

Trace IDs can also be used to link user feedback to specific generations. This can be used in a system where users provide feedback, like a thumbs up or thumbs down, or something more complex via our feedback APIs. This feedback can be linked to traces which can span over a single generation or multiple ones. Read more here:

Feedback