diff --git a/api-reference/inference-api/headers.mdx b/api-reference/inference-api/headers.mdx index 6ff982d1..04b5df34 100644 --- a/api-reference/inference-api/headers.mdx +++ b/api-reference/inference-api/headers.mdx @@ -403,6 +403,32 @@ const portkey = new Portkey({ +### Fetch Integrated Models + +Applies to `GET /v1/models` only. When `true`, forces the endpoint to return Portkey's integrated (Model Catalog) models even when a provider, virtual key, or config is passed on the request. Without this header, any provider signal causes the endpoint to proxy to the upstream provider's `/v1/models`. You can also set this as `fetch_integrated_models: true` inside a Portkey config. ([Docs](/api-reference/inference-api/models/models)) + + + + +```sh cURL {4} +curl https://api.portkey.ai/v1/models \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: @openai-prod" \ + -H "x-portkey-fetch-integrated-models: true" +``` + +```json Config +{ + "fetch_integrated_models": true, + "targets": [ + { "virtual_key": "openai-prod" }, + { "virtual_key": "anthropic-prod" } + ] +} +``` + + + ## Custom Headers You can pass any other headers your API expects by directly forwarding them without any processing by Portkey.
diff --git a/api-reference/inference-api/models/models.mdx b/api-reference/inference-api/models/models.mdx index 2e088b2e..ff68fd3e 100644 --- a/api-reference/inference-api/models/models.mdx +++ b/api-reference/inference-api/models/models.mdx @@ -2,3 +2,162 @@ title: Models openapi: get /models --- + +## Endpoint Behavior + +The `GET /v1/models` endpoint behaves differently depending on whether your request carries provider routing signals (headers or config). Understanding this is important when you're deciding which model list your application should consume. + +### Two response modes + + + + Returns the models configured in your Portkey **Model Catalog** — the ones your workspace can actually route to, including any custom models you've added. + + + Proxies the request to the upstream provider's own `/v1/models` endpoint (e.g. OpenAI, Anthropic, Bedrock) and returns their live model list. + + + +### How Portkey decides + +Portkey inspects your request for any of the following **provider signals**: + +- `x-portkey-provider` header +- `x-portkey-virtual-key` header +- `x-portkey-config` with `provider`, `virtual_key`, or `targets` set + +| Provider signal present? | `fetch_integrated_models` set? | Response | +| :-- | :-- | :-- | +| No | — | Portkey **integrated models** (model catalog) | +| Yes | No | **Provider-backed** — proxied to the provider's `/v1/models` | +| Yes | Yes | Portkey **integrated models**, filtered to the providers referenced in your config | + + +Because the default behavior with a provider signal is to hit the upstream, the response you see depends on which provider you're pointed at. Anthropic returns Claude models, OpenAI returns GPT models, and so on. If you want a single unified view, use the integrated mode described below. + + +--- + +## Provider-Backed Mode + +When you send a provider signal (e.g. `x-portkey-provider: @openai-prod`), Portkey calls that provider's `/v1/models` endpoint and returns the response. + +### Response format + +By default, Portkey normalizes the provider's response into the standard **OpenAI models list format** — the same shape as `openai.com/v1/models`: + +```json +{ + "object": "list", + "data": [ + { + "id": "claude-3-5-sonnet-20241022", + "object": "model", + "created": 1729555200, + "owned_by": "anthropic" + } + ], + "provider": "anthropic" +} +``` + +To receive the **raw provider response** without normalization, set `strict_open_ai_compliance` to `false`. See [Strict OpenAI Compliance](/product/ai-gateway/strict-open-ai-compliance) for details. + + + +```sh cURL +curl https://api.portkey.ai/v1/models \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: @anthropic-prod" \ + -H "x-portkey-strict-open-ai-compliance: false" +``` + +```py Python +from portkey_ai import Portkey + +portkey = Portkey( + api_key="PORTKEY_API_KEY", + provider="@anthropic-prod", + strict_open_ai_compliance=False +) + +models = portkey.models.list() +``` + +```js JavaScript +import Portkey from 'portkey-ai'; + +const portkey = new Portkey({ + apiKey: 'PORTKEY_API_KEY', + provider: '@anthropic-prod', + strictOpenAiCompliance: false +}); + +const models = await portkey.models.list(); +``` + + + +### Supported providers + +Native `listModels` support (with OpenAI-format normalization) is available for: + +- OpenAI +- Azure OpenAI +- Anthropic +- Google (Gemini) +- Google Vertex AI +- AWS Bedrock + +For providers without a native `listModels` transform, the raw provider response is returned as-is. + +--- + +## Integrated Mode Override + +Even when a provider signal is present, you can force Portkey to return your **integrated (model catalog) models** — the ones set up in your workspace — by using one of the following: + + + Header. Set to `true` to return integrated models even when a provider/virtual-key/config is passed on the request. + + + + Config field. Set to `true` on your Portkey config to achieve the same effect. + + +When active, Portkey filters the integrated model list to only the provider slugs referenced in your routing config (including virtual keys and `@provider/model` overrides). If any leaf in the config uses `passthrough: true`, all integrated models are returned unfiltered. + + + +```sh cURL {4} +curl https://api.portkey.ai/v1/models \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: @openai-prod" \ + -H "x-portkey-fetch-integrated-models: true" +``` + +```json Config +{ + "fetch_integrated_models": true, + "targets": [ + { "virtual_key": "openai-prod" }, + { "virtual_key": "anthropic-prod" } + ] +} +``` + + + + +Use this when your application logic depends on your workspace's curated model list — for example, to show end users only the models your team has approved and configured — even though the request still needs to carry provider auth for downstream routing. + + +--- + +## Choosing a mode + +| If you want to… | Use | +| :-- | :-- | +| Show the full list of models the provider currently offers | Provider-backed (default with a provider signal) | +| Show only the models available in your Portkey workspace | Send no provider signal, **or** set `x-portkey-fetch-integrated-models: true` | +| Get the untouched provider payload (extra fields, provider-specific metadata) | Provider-backed + `strict_open_ai_compliance: false` | diff --git a/changelog/enterprise.mdx b/changelog/enterprise.mdx index b3e9302a..dcefbfdb 100644 --- a/changelog/enterprise.mdx +++ b/changelog/enterprise.mdx @@ -37,6 +37,12 @@ Gen AI and guardrail OTel spans now carry Portkey tenant attributes — org ID, [OpenTelemetry Documentation](/product/observability/opentelemetry) +### Unified `/v1/models` Handler + +Provider-backed `GET /v1/models` requests now flow through a dedicated `listModels` handler that proxies to the upstream provider (OpenAI, Azure OpenAI, Anthropic, Google, Vertex AI, Bedrock) and normalizes the response to OpenAI's models list format. Setting `strict_open_ai_compliance: false` returns the raw provider payload. + +[Models API Documentation](/api-reference/inference-api/models/models) + ### Provider Updates - **Anthropic**: `thinking.display` parameter is now passed through to the upstream API, enabling clients to control whether extended thinking content is returned in responses