Skip to main content

Build a RAG agent without the gateway

What you'll build: a Python script that indexes the AcruxCore docs, then answers a question about them two ways — retrieve-then-ask, and let-the-model-search — with every model call going straight to OpenRouter and every run showing up as a trace.

RAG means retrieval-augmented generation: before you ask the model a question, you look up the passages most likely to answer it and paste them into the prompt.

When to use BYO

The gateway path stores your provider key server-side and routes every call through AcruxCore — useful for budget controls, caching, and fallback routing. BYO skips that: you pass the SDK an OpenAI-compatible base_url and api_key, and it calls the endpoint directly. Your key never reaches our servers, you save a network hop, and you still get versioned prompts, a tool catalog, and full traces.

Prefer a notebook?

acrux_rag.ipynb is this whole page as one runnable notebook, written for a first-timer: a preflight cell that checks both keys — including whether your provider really serves embeddings — the index built step by step, a live read of the traces both answering styles produce, and four real ways to get BYO wrong. It renders on GitHub with its saved output, so you can read it through before running anything.

1. Store the two prompts​

Both answering styles need a prompt, and both belong in AcruxCore rather than hardcoded in the script — that way you can edit the wording and ship it without a redeploy. Create rag-chat for the linear path, with a system message that takes the retrieved passages as a context variable and a user message that takes the question.

The rag-chat editor with PRODUCTION → v1 and STAGING → v1, a "No default model" dropdown, a SYSTEM message ending in a highlighted context placeholder, and a USER message that is just a highlighted question placeholder

Leave Default model empty. That field binds a model from the gateway's registry, and this guide never touches the gateway — the model id belongs to OpenRouter, so the script passes it directly.

Then create rag-chat-agent the same way. Its system message tells the model to look things up with a tool instead of handing it context up front:

You answer questions about AcruxCore. Use the search_docs tool to look
things up before answering — search more than once if the first result is
thin, or if the question has several parts. Answer only from what the tool
returns, and say so plainly when it comes back without the answer.

2. Point the SDK at OpenRouter​

This is the whole BYO feature — one dictionary, passed to every model call:

import acruxcore as acrux

PROVIDER: acrux.ProviderConfig = {
    "base_url": "https://openrouter.ai/api/v1",
    "api_key": os.environ["OPENROUTER_API_KEY"],
}

CHAT_MODEL = "google/gemini-3.7-flash"
EMBED_MODEL = "openai/text-embedding-3-small"

Any OpenAI-compatible endpoint works here — Groq, Together, vLLM, or Ollama on your laptop. The api_key is sent as a bearer token to base_url and nowhere else.

3. Index the docs​

Fetch each guide, split it into overlapping chunks, and embed them. OpenRouter serves embeddings from the same host and the same key as the chat model, so the script needs only one provider credential:

def embed_texts(texts: list[str]) -> list[list[float]]:
    resp = requests.post(
        f"{PROVIDER['base_url']}/embeddings",
        headers={"Authorization": f"Bearer {PROVIDER['api_key']}"},
        json={"model": EMBED_MODEL, "input": texts},
        timeout=60,
    )
    resp.raise_for_status()
    data = resp.json()["data"]
    # OpenRouter does not promise input order, so sort by the echoed index.
    return [row["embedding"] for row in sorted(data, key=lambda r: r["index"])]

Store the vectors in Chroma, an embedded vector database that needs no server:

collection = chromadb.Client().get_or_create_collection(
    name="acrux_docs", metadata={"hnsw:space": "cosine"}
)
collection.add(ids=ids, embeddings=embeddings, documents=documents, metadatas=metadatas)

Running this over the five guides gives 31 chunks:

  fetched prompts.md: 4,805 chars → 6 chunks
  fetched gateway.md: 4,173 chars → 5 chunks
  fetched tracing.md: 3,903 chars → 5 chunks
  fetched tools.md: 7,441 chars → 9 chunks
  fetched evaluation.md: 4,421 chars → 6 chunks
  indexed 31 chunks from 5 documents

4. Answer it linearly​

Retrieve first, then make exactly one model call. Report the retrieval as its own span and hand its trace id to hub.gateway.chat(), so the passages and the answer land in the same trace:

reported = await hub.traces.ingest({
    "name": "rag-linear",
    "spans": [{
        "spanId": "retrieval", "name": "search_docs", "kind": "retrieval",
        "status": "ok", "startTime": started, "endTime": _now(),
        "input": {"query": question}, "output": {"context": context},
    }],
})

rendered = await hub.prompts.render("rag-chat", "production",
                                   {"context": context, "question": question})
result = await hub.gateway.chat(
    rendered.model or CHAT_MODEL,
    rendered.messages,
    provider=PROVIDER,
    prompt_version_id=rendered.version_id,
    trace={"trace_id": reported.trace_id},
)

Open Traces and the run is there — the retrieval that found the passages, then the model call that used them:

The rag-linear trace showing 2 spans and 1,771 tokens: a Retrieval span named search_docs at 869ms, then an LLM span for google/gemini-3.7-flash at 3.56s

Expand the LLM span and the BYO call is fully accounted for — tokens, latency, and openrouter.ai as the provider:

The expanded LLM span showing Model google/gemini-3.7-flash, Provider openrouter.ai, Tokens 1,771, Latency 3.56s, and a "View traces for this prompt version" link

Two things worth noticing. There is no Linked gateway request row, because no gateway was involved. And View traces for this prompt version works anyway: passing prompt_version_id from hub.prompts.render() links the trace back to the exact prompt version that produced it, so you can still ask "how did v3 do?" after you ship v4.

No dollar cost on BYO spans

Tokens, latency, model, and payloads are all recorded, but costUsd is empty — we only price calls that went through the gateway, where we know the rate card. Your provider's own dashboard has the bill.

5. Answer it agentically, with a tool​

The linear version always retrieves, once, whether or not the question needs it. The agentic version hands the retriever to the model as a tool and lets it decide. Decorate the function and pass it to run_tool_loop:

@acrux.tool
async def search_docs(query: str) -> str:
    """Search AcruxCore's documentation for relevant information.

    Args:
        query: The search query — a question or topic to look up.
    """
    return retrieve_context(_COLLECTION, query)


rendered = await hub.prompts.render("rag-chat-agent", "production", {"question": question})
result = await hub.gateway.run_tool_loop(
    rendered.model or CHAT_MODEL,
    rendered.messages,
    tools=[search_docs],
    provider=PROVIDER,
    prompt_version_id=rendered.version_id,
)

The loop calls the model, runs whatever tools it asks for, feeds the results back, and repeats until the model stops asking. The trace shows that rhythm directly — model, tool, model:

The rag-agentic trace showing 3 spans and 1,962 tokens: an LLM span at 2.05s, a nested Tool span named search_docs at 909ms, and a second LLM span at 3.92s

The tool span nests under the model call that requested it, so a loop that searched four times is four nested spans, not a flat list you have to reconstruct.

On the BYO path there is no gateway to look tool schemas up server-side, so the SDK inlines the full JSON Schema into the request for you. It also syncs the tool to your catalog on the first call, deriving the name, description, and parameters from the function itself:

The search_docs tool page marked "Defined in code", with v2 "Search AcruxCore's documentation for relevant information." tagged code, above v1 "Search the internal docs for a phrase." tagged dashboard

Tools are versioned like prompts. Here the docstring in the script became v2, and the earlier version someone wrote in the dashboard is still there to promote back to.

Doing this over the API​

You don't need the SDK. The BYO path is three independent HTTP calls: render the prompt from us, call OpenRouter yourself, report the trace back to us. This is the route to take from a language with no AcruxCore SDK.

Render the prompt. Note versionId — that's what links a trace to the version that produced it:

curl -X POST "$ACRUXCORE_BASE_URL/prompts/rag-chat/production/render" \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"variables":{"context":"[tools.md]\nOpen the prompt and go to the Tools tab.","question":"How do I attach a tool?"}}'
{
  "messages": [
    {
      "role": "system",
      "content": "You answer questions about AcruxCore using only the documentation excerpts below. If the excerpts do not contain the answer, say so plainly instead of guessing.\n\nDocumentation:\n[tools.md]\nOpen the prompt and go to the Tools tab."
    },
    { "role": "user", "content": "How do I attach a tool?" }
  ],
  "tools": [],
  "model": null,
  "versionId": "72bed36e-6d77-425e-878c-ec534ca43b8e",
  "versionNumber": 1
}

Call OpenRouter directly — this request never touches AcruxCore:

curl -X POST "https://openrouter.ai/api/v1/chat/completions" \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"google/gemini-3.7-flash","messages":[...],"max_tokens":80}'
{
  "id": "gen-1785427905-CA5ECKBA1Pg2RJAE7mfX",
  "model": "google/gemini-3.7-flash",
  "provider": "OpenAI",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "message": {
        "role": "assistant",
        "content": "[tools.md] Open the prompt and go to the Tools tab, then use Connect a tool from the catalog to pick the tool."
      }
    }
  ],
  "usage": { "prompt_tokens": 51, "completion_tokens": 29, "total_tokens": 80 }
}

Report the trace. Spans are a batch under traces, and keys are camelCase because they go to the API verbatim:

curl -X POST "$ACRUXCORE_BASE_URL/traces" \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "traces": [{
      "name": "rag-linear",
      "spans": [
        {"spanId":"s1","name":"search_docs","kind":"retrieval","status":"ok",
         "startTime":"2026-07-30T16:11:44.000Z","endTime":"2026-07-30T16:11:44.000Z",
         "input":{"query":"How do I attach a tool?"}},
        {"spanId":"s2","name":"google/gemini-3.7-flash","kind":"llm","status":"ok",
         "startTime":"2026-07-30T16:11:44.000Z","endTime":"2026-07-30T16:11:44.000Z",
         "model":"google/gemini-3.7-flash","provider":"openrouter.ai",
         "promptVersionId":"72bed36e-6d77-425e-878c-ec534ca43b8e",
         "usage":{"promptTokens":51,"completionTokens":29,"totalTokens":80}}
      ]
    }]
  }'
{ "accepted": 2, "traceIds": ["a8b9b021-ade7-4712-98d6-c9f9417f7e67"] }

The whole script​

Run python acrux_rag.py --setup once to create the two prompts, then python acrux_rag.py "your question" to answer with both styles. See acrux_rag.py on GitHub for the complete source.

What's next​