Gateway API — Chat Completions

Phase 2 AI Gateway. Base path: /api/v1/gateway. An OpenAI client points its baseURL at https://<host>/api/v1/gateway.

Related references: provider connections, virtual keys, budgets, usage analytics.

Auth for POST /chat/completions (gatewayAuth): a virtual key (Authorization: Bearer agh_sk_…) is the primary machine path; a session cookie or personal/team API key also works but requires role owner/admin/editor (viewers get 403). Response metadata is returned in x-gateway-* headers.

All examples below were verified with real curl output against a running server using live OpenAI and Gemini connections.

---

POST /api/v1/gateway/chat/completions (non-streaming)

OpenAI-compatible chat completion. The gateway resolves the team's connection for the model's provider, calls it, prices the call, and records a request-log row.

curl

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Content-Type: application/json" \
  --cookie "connect.sid=<session>" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi in one word."}],"max_tokens":10}'

TypeScript

const result = await hub.gateway.chat({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Say hi in one word.' }],
  maxTokens: 10,
});
console.log(result.content);

Python

result = await hub.gateway.chat(
    'gpt-4o-mini',
    [{'role': 'user', 'content': 'Say hi in one word.'}],
    max_tokens=10,
)
print(result.content)

Response headers (status 200):

x-gateway-request-id: 561e701d-d775-44b8-ab85-f9a7e75b3b3b
x-gateway-provider: openai
x-gateway-model: gpt-4o-mini-2024-07-18
x-gateway-cost-usd: 0.00000315
x-gateway-cache: miss

Response body (status 200):

{
  "id": "chatcmpl-DxoMTfx6c4OtbeOrvFXf9m6qKi2P6",
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion",
  "created": 1783147313,
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello!" },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 13, "completion_tokens": 2, "total_tokens": 15 }
}

---

POST /api/v1/gateway/chat/completions (Gemini — multi-provider routing)

The same endpoint routes to Gemini when the model belongs to a gemini connection. Note x-gateway-cost-usd is empty when the model has no configured price.

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Content-Type: application/json" \
  --cookie "connect.sid=<session>" \
  -d '{"model":"gemini-2.5-flash-lite","messages":[{"role":"user","content":"Say hi in one word."}],"max_tokens":10}'

Response headers (status 200):

x-gateway-request-id: ae680e52-1d75-4ceb-9447-b6d69312e00c
x-gateway-provider: gemini
x-gateway-model: gemini-2.5-flash-lite
x-gateway-cost-usd:
x-gateway-cache: miss

Response body (status 200):

{
  "id": "chatcmpl-eaa42e14-7bf5-4c5f-ad95-7ae5df61ccc8",
  "model": "gemini-2.5-flash-lite",
  "object": "chat.completion",
  "created": 1783147330,
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello" },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 7, "completion_tokens": 1, "total_tokens": 8 }
}

---

POST /api/v1/gateway/chat/completions (via virtual key)

Machine credential path — authenticate with a virtual key instead of a session. The virtual key was created via POST /gateway/keys; it is shown here redacted as agh_sk_...REDACTED (the real key was used and worked).

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer agh_sk_...REDACTED" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi in one word."}],"max_tokens":10}'

Response headers (status 200):

x-gateway-request-id: be005d49-e3b1-4611-82c2-b568ddc9a083
x-gateway-provider: openai
x-gateway-model: gpt-4o-mini-2024-07-18
x-gateway-cost-usd: 0.00000315
x-gateway-cache: miss

Response body (status 200):

{
  "id": "chatcmpl-DxoN4uwS5rPd9w7RewNVjrDSx7nsD",
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion",
  "created": 1783147350,
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello!" },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 13, "completion_tokens": 2, "total_tokens": 15 }
}

---

POST /api/v1/gateway/chat/completions (streaming)

Set "stream": true. The response is text/event-stream: one OpenAI-compatible chat.completion.chunk per data: frame, terminated by data: [DONE]. x-gateway-cost-usd is intentionally omitted from stream headers (cost is not known until the stream ends).

curl

curl --no-buffer -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Content-Type: application/json" \
  --cookie "connect.sid=<session>" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Count: one two three"}],"max_tokens":20,"stream":true}'

TypeScript

const stream = await hub.gateway.chat({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Count: one two three' }],
  maxTokens: 20,
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.delta.content ?? '');
}

Python

stream = await hub.gateway.chat(
    'gpt-4o-mini',
    [{'role': 'user', 'content': 'Count: one two three'}],
    max_tokens=20,
    stream=True,
)
async for chunk in stream:
    print(chunk.delta.get('content', ''), end='')

Response headers (status 200):

Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
x-gateway-request-id: 9777fa85-2ace-4d96-8d97-acf8ac84be04
x-gateway-provider: openai
x-gateway-model: gpt-4o-mini

Response body (SSE stream):

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":"One"},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":","},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":" two"},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":","},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":" three"},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":"."},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":null}]}

data: [DONE]

---

Naming a trace across several calls

A trace can span many completions. Two headers name it, and the difference between them is whether you are giving an instruction or a fallback.

Header	On a new trace	On a trace that already has a name
x-trace-name	names it	overwrites it
x-trace-name-if-unset	names it	ignored

x-trace-name is last-explicit-write-wins on purpose: an agent that only works out what a run is on its second call can still name the trace after it. Omitting the header never resets a name.

x-trace-name-if-unset is for a fallback name — the one a client library reaches for when the caller expressed no preference. It fills in the default timestamp name and is otherwise ignored, so a call that merely joins a trace cannot rename it. Both SDKs send their runToolLoop default on this channel.

Both are free text, so send them percent-encoded (encodeURIComponent) — a raw non-ASCII header value is rejected before the request leaves most HTTP clients. The server decodes, and falls back to the raw value if it was not encoded.

Call A names the trace:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-trace-name: content-supervisor-flow" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}' -i

The response's x-gateway-trace-id is the trace to join. Reading it back:

{ "trace": { "name": "content-supervisor-flow", "spanCount": 1 } }

Call B joins it, sending only a fallback name:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-trace-id: b91a9709-d91a-4df5-b3fd-447918c8a552" \
  -H "x-trace-name-if-unset: runToolLoop" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi again"}]}'

{ "trace": { "name": "content-supervisor-flow", "spanCount": 2 } }

The span landed; the name did not move. Call C joins with an explicit name instead:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-trace-id: b91a9709-d91a-4df5-b3fd-447918c8a552" \
  -H "x-trace-name: refund-request" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi once more"}]}'

{ "trace": { "name": "refund-request", "spanCount": 3 } }

Both are also accepted on the request body as trace.name and trace.nameIfUnset, for clients that would rather not set headers.

---

Tools and prompt lineage on a completion

Three request fields decide which tools a completion sees and which prompt version its trace is attributed to. All three work on both the streaming and non-streaming path.

prompt — send a stored-prompt reference instead of messages and the gateway renders it, fills the model from the version's bound model when model is omitted, and auto-attaches the tools bound to that prompt alias. Definitions only: the model's tool_calls come back for you to run.

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","prompt":{"name":"weather-brief","alias":"production","variables":{"city":"Lisbon"}}}'

Response (status 200) — the bound get_weather was attached without being named:

{
  "id": "chatcmpl-EEycj62PkeMuhzY0dJR3Q7yGnq6zC",
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion",
  "created": 1787238337,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "call_pZdUAL1IodyxSgups7E1Sgbd",
            "type": "function",
            "function": { "name": "get_weather", "arguments": "{\"city\":\"Lisbon\"}" }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ],
  "usage": { "prompt_tokens": 69, "completion_tokens": 15, "total_tokens": 84 }
}

tool_refs — name catalog tools by name when you send your own messages. Each ref follows an alias (default production) or pins one exact version; both on one ref is a 400. The gateway resolves them to schemas and merges them with any inline tools (a shared name is a 400). See tool resolve for the resolution rules.

prompt_version_id — for a caller that rendered the prompt itself and is sending the rendered messages. It stamps the request row and the llm span, so a client-side tool loop is as traceable back to its prompt as a server-rendered prompt reference is. It must be a prompt version in your own team.

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-4o-mini",
        "messages": [{"role":"user","content":"What'\''s the weather in Lisbon right now? Answer in one short sentence."}],
        "tool_refs": [{"name":"get_weather","alias":"production"}],
        "prompt_version_id": "d884bfc0-2f5f-4858-b119-682ff7caebbb"
      }'

Response (status 200):

{
  "id": "chatcmpl-EEyYkncDTfsaGVXWa5faoRUjKylu1",
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion",
  "created": 1787238090,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "call_mpQNFOQ4A0ZQ0O8ydYAq2ZWv",
            "type": "function",
            "function": { "name": "get_weather", "arguments": "{\"city\":\"Lisbon\"}" }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ],
  "usage": { "prompt_tokens": 69, "completion_tokens": 15, "total_tokens": 84 }
}

An id that is not one of your team's prompt versions is rejected rather than dropped, so a caller asking for lineage finds out when it is not happening:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "prompt_version_id '11111111-1111-1111-1111-111111111111' is not a prompt version in this team."
  }
}

Sending prompt_version_id alongside prompt is allowed and ignored: the reference renders here, so the gateway already knows the exact version it used. Neither field ever reaches the provider.

Send variables with it

Add the values you rendered with as a top-level variables object. They are not re-rendered — you already rendered, and a second pass could rewrite model or user text that happens to contain {{ — but they are stored on the llm span as the run's replay lineage. An evaluation dataset example is those variables, so a client-rendered run that omits them can never seed one.

This is the shape to use when the stored version has no placeholders at all — a fixed system message with the user turn composed by your app:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-4o-mini",
        "messages": [
          {"role": "system", "content": "You are a terse support agent. Answer in one sentence."},
          {"role": "user", "content": "How do I rotate an API key?"}
        ],
        "prompt_version_id": "792cf7a8-d061-4b2f-82a9-f9ad3200eb3b",
        "variables": {"question": "How do I rotate an API key?"}
      }'

Response (status 200):

{
  "id": "chatcmpl-ELkmu7PgavkoKukav8n0B6pGAXGVE",
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion",
  "created": 1788853448,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "To rotate an API key, generate a new key in your API management console and update your applications to use it while deactivating the old key after verification."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 31, "completion_tokens": 32, "total_tokens": 63 }
}

The span that call wrote carries both halves. From GET /traces/:id, the fields that matter here (the full span also carries timings, tokens and cost):

{
  "spanId": "90244880-496e-4021-90d6-8e4bd0aac682",
  "kind": "llm",
  "promptVersionId": "792cf7a8-d061-4b2f-82a9-f9ad3200eb3b",
  "payload": {
    "variables": { "question": "How do I rotate an API key?" }
  }
}

Feedback on that trace can then become a dataset example. Without variables the build skips the row with "no prompt variables were captured".

The SDKs fill all of this in for you — gateway.runPromptWithTools(rendered) / gateway.run_prompt_with_tools(rendered) derive the model, messages, refs and version id from one render result. See Call a prompt's tools from the SDK.

---

Error responses

Enforcement happens before any provider call. Observed shapes:

• 403 MODEL_NOT_ALLOWED — a virtual key with an allowedModels allow-list called a model outside it. See virtual keys.

  { "error": { "code": "MODEL_NOT_ALLOWED", "message": "Model 'gemini-2.5-flash-lite' is not allowed for this key." } }

• 402 BUDGET_EXCEEDED — the team-wide (or key-scoped) budget cap is reached. See budgets.

  { "error": { "code": "BUDGET_EXCEEDED", "message": "Team-wide budget exceeded." } }

When the provider is the one saying no

A rejection from the upstream provider comes back as 400 PROVIDER_BAD_REQUEST, carrying the provider's own message. That message is the part worth reading: it names what is actually wrong, which a status code cannot.

Two response_format requests that break different strict-mode rules, and therefore answer differently:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}],
       "response_format":{"type":"json_schema","json_schema":{"name":"x","strict":true,
       "schema":{"type":"object","properties":{"a":{"type":"string"},"b":{"type":"string"}},
       "required":["a"],"additionalProperties":false}}}}'

Response (status 400) — b is in properties but missing from required:

{"error":{"code":"PROVIDER_BAD_REQUEST","message":"Provider rejected the request (400): Invalid schema for response_format 'x': In context=(), 'required' is required to be supplied and to be an array including every key in properties. Missing 'b'."}}

Now with required complete but additionalProperties dropped:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}],
       "response_format":{"type":"json_schema","json_schema":{"name":"x","strict":true,
       "schema":{"type":"object","properties":{"a":{"type":"string"},"b":{"type":"string"}},
       "required":["a","b"]}}}}'

Response (status 400):

{"error":{"code":"PROVIDER_BAD_REQUEST","message":"Provider rejected the request (400): Invalid schema for response_format 'x': In context=(), 'additionalProperties' is required to be supplied and to be false."}}

Which statuses forward the provider's message, and which do not. A 400, 404, 413, 422 or 429 describes something you can act on — your request, or the upstream's rate limit — so the provider's own message is passed through. A 401, 403 or any 5xx describes the connection instead: its credential or the provider's own internals, neither of which the caller of this gateway can do anything about, and a provider's 401 body can echo a masked copy of the key it was sent. Those are summarised:

{"error":{"code":"PROVIDER_ERROR","message":"Provider error (401): OpenAI request failed with status 401"}}

That one answers 502 PROVIDER_ERROR, and a provider timeout answers 504 PROVIDER_TIMEOUT. If you see a 401 here, the gateway connection's stored key is the thing to check, not the request.

Upstream rate limits

When the provider rate-limits or refuses on quota, the gateway answers 429 PROVIDER_RATE_LIMITED and forwards the provider's own Retry-After when it sends one. Streaming requests answer the same way, as JSON, because nothing has been written to the stream yet.

curl -i -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"free-glm","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'

Response (status 429):

Retry-After: 5

{"error":{"code":"PROVIDER_RATE_LIMITED","message":"Provider rate limit (429): Provider returned error"}}

Do not confuse this with 429 RATE_LIMITED, which is your own virtual key's RPM/TPM window. The codes differ because the fixes differ: RATE_LIMITED is raised or lowered under Gateway → Virtual keys, while PROVIDER_RATE_LIMITED is the upstream account's limit and has to be resolved with that provider.

---

Response Cache (G6)

Per-team exact-match response cache. Opt-in per virtual key via cacheTtlSeconds; only requests with temperature: 0 are cached. A cache hit returns the stored body at zero cost with header x-gateway-cache: hit and does not increment any budget. Bypass per call with request header x-gateway-cache: no-store.

DELETE /api/v1/gateway/cache

Flush the calling team's response cache (owner/admin; editors/viewers 403). Returns the number of rows removed (0 when nothing was cached).

curl -X DELETE $ACRUXCORE_BASE_URL/gateway/cache \
  --cookie "connect.sid=<owner/admin session>"

Response (status 200):

{ "deleted": 0 }