Multiple endpoints

Configure access to multiple OpenAI API endpoints like chat completions, embeddings, and models through one backend.

Configure access to multiple OpenAI API endpoints such as for chat completions, embeddings, and models through the same AgentgatewayBackend.

About

To set up multiple LLM endpoints, use the ai.routes field in the policies section of the AgentgatewayBackend resource. This field maps the API paths to supported route types. The keys are URL suffix matches, like /v1/models. The values are the route types, like Completions or Passthrough.

Alternatively, you can define the routes in a separate AgentgatewayPolicy resource attached to a gateway. This approach decouples routing configuration from the backend definition, which can simplify management when you have multiple routes pointing to the same backend.

Completions: Parses the request, translates it to the LLM provider format, and fully processes it as an LLM request. This route type unlocks the full set of agentgateway LLM features, such as tokenization and token-based rate limiting, prompt guards, prompt enrichment, model aliasing, transformations, cost tracking, and detailed observability.
Detect: Forwards the request as-is, but makes a best effort to extract the model and token counts so that a subset of policies still applies, specifically token-based rate limiting and telemetry. Guardrails and other request-shaping policies do not apply. Use this route type for endpoints with a format that cannot be automatically parsed by agentgateway, but where you still want metrics and rate limiting.
Passthrough: Forwards the request to the LLM provider as-is, with no parsing, processing, or policies. Use passthrough for endpoints that do not need any traffic policy or manipulation, such as health checks or custom endpoints. Otherwise, use the other route types.

Paths are matched in order, and the first match determines how the request is handled. The wildcard character * can be used to match anything. If no route is set, the route defaults to the Completions endpoint.

Before you begin

Set up an agentgateway proxy.
Set up API access to each LLM provider that you want to use. The example in this guide uses OpenAI.

Configure multiple endpoints

Configure access to multiple endpoints in your LLM provider, such as for chat completions, embeddings, and models through the same AgentgatewayBackend. The following steps use OpenAI as an example.

Update your AgentgatewayBackend resource to include a routes field that maps API paths to route types.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai: {}
        # Optional: specify default model
        #model: gpt-3.5-turbo
     # host: api.openai.com  # Optional: custom host if needed
     # port: 443  # Optional: custom port
  policies:
    auth:
      secretRef:
        name: openai-secret
    ai:
      routes:
        "/v1/chat/completions": "Completions"
        "/v1/embeddings": "Passthrough"
        "/v1/models": "Passthrough"
        "*": "Passthrough"
EOF

Setting	Description
`v1/chat/completions`	Routes to the chat completions endpoint with LLM-specific processing. This endpoint is used for chat-based interactions. For more information, see the OpenAI API docs for the endpoint.
`v1/embeddings`	Routes to the embeddings endpoint with `Passthrough` processing. This endpoint is used to get vector embeddings that machine learning models can use more easily than chat-based interactions. For more information, see the OpenAI API docs for the endpoint.
`v1/models`	Routes to the models endpoint with `Passthrough` processing. This endpoint is used to get basic information about the models that are available. For more information, see the OpenAI API docs for the endpoint.
`*`	Matches any path that doesn’t match the specific endpoints otherwise set. Typically, you set this value to `Passthrough` to pass through to the provider API without LLM-specific processing.

Create an HTTPRoute resource that routes traffic to the OpenAI AgentgatewayBackend along the /openai path matcher.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /openai
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /v1/chat/completions
    backendRefs:
    - name: openai
      namespace: agentgateway-system
      group: agentgateway.dev
      kind: AgentgatewayBackend
EOF

Alternative: define routes in an AgentgatewayPolicy

Instead of setting up the routes map on the AgentgatewayBackend, you can configure them in a separate AgentgatewayPolicy attached to the HTTPRoute. This decouples routing configuration from the backend definition.

Create the AgentgatewayBackend resource without the routes field:

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai:
        model: gpt-3.5-turbo  # Optional: specify default model
     # host: api.openai.com  # Optional: custom host if needed
     # port: 443  # Optional: custom port
  policies:
    auth:
      secretRef:
        name: openai-secret
EOF

Create an AgentgatewayPolicy resource that targets the HTTPRoute and defines the routes:

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: openai-routes
  namespace: agentgateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: openai
  backend:
    ai:
      routes:
        "/v1/chat/completions": "Completions"
        "/v1/embeddings": "Passthrough"
        "/v1/models": "Passthrough"
        "*": "Passthrough"
EOF

Send requests to different OpenAI endpoints. With the routes configured, you can access different OpenAI endpoints by including the full path in your requests:

Chat completions:

curl "$INGRESS_GW_ADDRESS/openai/v1/chat/completions" \
  -H content-type:application/json \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}]
  }' | jq

Embeddings:

curl "$INGRESS_GW_ADDRESS/openai/v1/embeddings" \
  -H content-type:application/json \
  -d '{
    "model": "text-embedding-ada-002",
    "input": "The food was delicious"
  }' | jq

Models list:

curl "$INGRESS_GW_ADDRESS/openai/v1/models" | jq