RAG APIs: Architecting Retrieval-Augmented Generation Endpoints

A chatbot that answers from its training data is a model call. A chatbot that answers from your policy documents or ticket history is something else. It is a data access system that happens to end in a model call. That second system is what a RAG API exposes, and it inherits the design problems of both an API and a search engine.

This guide covers the architecture decisions behind a retrieval-augmented generation endpoint. It walks through the pipeline stages, the request and response contract, vector database integration, hybrid retrieval, and evaluation. It closes with where governance belongs once your LLM starts reading your own data.

What a RAG API Actually Does (and Why Foundation Models Alone Fall Short)​

Foundation models learn from a fixed snapshot of data. That leaves three gaps for enterprise use. The first is the knowledge cutoff, since the model's weights hold nothing published after training ended. The second is missing context. Your internal runbooks, contracts, and product specs were never in the training set. The third is provenance. The model cannot tell you where an answer came from, so the user has no way to check it.

All three gaps lead to the same failure. When the model lacks the facts, it still produces fluent text, and that text can be wrong. The original RAG paper by Lewis et al. named provenance and updating world knowledge as open problems for models that keep knowledge only in their parameters. Their answer was to pair the model with a retrievable, non-parametric memory (Lewis et al., 2020).

A RAG API applies that idea as a service. Take an internal support assistant asked, "What is the refund window for enterprise contracts?" The API fetches the relevant passages from the billing policy in your knowledge base. It then asks the model to answer from those passages and cite them. The model's weights stay the same. What changes is the evidence placed in front of it on each request.

This is also why RAG and fine-tuning solve different problems. Fine-tuning adjusts how a model responds, such as its format, tone, or domain vocabulary. It is a poor way to keep facts current, because every document change would need another training run. RAG keeps facts outside the model, so updating an answer means updating an index.

The Four-Stage RAG Pipeline: Ingestion, Retrieval, Augmentation, Generation​

This guide uses a common vector-based RAG design. It supplements a model's parametric knowledge with retrieved evidence. Other systems can use lexical, structured, or hybrid retrieval. In the vector-based design, from a prototype to a production-grade service, the pipeline runs through four stages:

1. Ingestion: Split source documents into chunks, convert each chunk into a vector embedding, and load the results into an index.
2. Retrieval: Encode the user's query the same way and fetch the top-k most relevant chunks.
3. Augmentation: Combine the retrieved chunks and the user's question into one structured prompt.
4. Generation: Send that prompt to the LLM, which answers from the supplied context.

The list reads as a sequence, but the stages do not share a timeline. Ingestion runs offline, on a schedule or when source documents change. Retrieval, augmentation, and generation run inside each request. Placing ingestion in the request path is a common early mistake. It ties user latency to document processing and turns every re-index into a production event.

The diagram separates the two timelines. Ingestion writes to the indexes in the background. At request time, the RAG service reads from the indexes over the internal network. It then makes a separate outbound call to the model provider. The client never talks to the index or the model directly.

A minimal version of this pipeline, with one retriever and no evaluation loop, is often called naive RAG. It makes a useful baseline. The rest of this guide covers what production RAG systems add on top of it.

Designing the RAG API Endpoint​

The endpoint is where the pipeline becomes a contract. Consumers should not need to know how many chunks you retrieve or which embedding model you run. They need a question in, an answer out, and enough evidence to trust the answer.

The request and response contract​

A workable contract for a synchronous endpoint looks like this:

POST /v1/answers

{
  "query": "What is the refund window for enterprise contracts?",
  "filters": { "product_line": "enterprise" },
  "max_sources": 5
}

200 OK

{
  "answer": "<answer text grounded in the cited sources>",
  "sources": [
    {
      "document_id": "billing-policy-2026-03",
      "title": "Billing and Refund Policy",
      "chunk_id": "billing-policy-2026-03#4",
      "score": 0.82
    }
  ],
  "request_id": "req_7f3a"
}

Three fields carry most of the design weight:

• sources is the trust surface. Each citation should point to a document the caller can open. A chunk ID the caller cannot resolve only creates confusion.
• filters narrows scope with metadata. Treat it as a relevance hint, not an authorization mechanism. Access rules must come from the caller's verified identity, which the governance section covers.
• request_id links the answer to a log of what was retrieved. You will need that link for evaluation and for audits.

Synchronous and streaming responses​

Generation usually dominates latency. A synchronous response makes the user wait for the full answer. Streaming returns tokens as the model produces them, usually over server-sent events. For chat interfaces, streaming improves perceived responsiveness.

Streaming also changes the contract. The retrieved candidates are known before generation starts. The sources that will support the final answer are not. A common pattern sends a first event with the authorized retrieved candidates, labeled as candidates rather than citations. The answer tokens follow. A final event carries completion metadata and the citations, validated against the generated answer. Send identifiers and titles the caller is permitted to see, not full retrieved passages. Output guardrails behave differently on a stream, because a policy may need to buffer content before it can judge it. The WSO2 AI Gateway documentation covers how policies, analytics, and token counting behave when streaming a response through an LLM proxy.

How much retrieval to expose​

You have three practical options:

• Answer only. This is the simplest contract, but it is the hardest to debug.
• Answer plus citations. This is the sensible default for most enterprise use cases.
• A separate retrieval endpoint. A POST /v1/search route sits beside the answer route. It suits consumers that want to run their own generation, such as other services or agents.

The third option is also the shape agentic RAG takes. In that pattern, an agent decides when to retrieve and calls retrieval as a tool. Keep in mind that a raw retrieval endpoint widens your data surface. Every chunk it returns is data leaving your boundary, whether or not a model sits in between.

Keep ingestion off the serving API​

Ingestion belongs in its own pipeline, with its own deployment and credentials. The serving API needs read access to the index. It should not hold write access. That separation limits what a compromised or faulty serving path can change.

Vector Database Integration: What Actually Matters​

The vector database is where many retrieval problems start. It is also where some of the most expensive decisions get locked in.

Chunking strategy​

Chunk size is a trade-off between precision and context. Small chunks match queries closely but can lose the surrounding sentence that gives them meaning. Large chunks keep context but dilute the embedding. A chunk that covers five topics tends to match none of them well.

Structure-aware chunking often works better than fixed character counts for technical and policy documents. It splits on headings, sections, or table boundaries. Store the parent document ID and section path as metadata on every chunk in the vector database. You will need them for citations, filtering, and re-ingestion.

Retrieval depth has a cost on the generation side too. Liu et al. found that models use information best when it sits at the start or end of the context window. Performance dropped when the relevant passage sat in the middle of a long context. Passing more chunks is not a free way to raise recall (Liu et al., 2023). Order matters, and so does restraint.

Embedding model selection​

The embedding model defines what "similar" means for your whole system. Evaluate candidates on your own queries and documents, not only on public leaderboards.

The operational catch is lock-in. Vectors from different embedding models live in different spaces and cannot be compared. Switching models means re-embedding the entire corpus and rebuilding the index. Version the embedding model alongside the index. Plan a model change as a migration with a cutover, not as an in-place update.

Retrieval quality and latency at scale​

Exact nearest-neighbor search compares a query with every stored vector, which gets slow on large corpora. Most vector databases use approximate nearest neighbor (ANN) indexes instead. ANN gives up some recall in exchange for lower latency, and index parameters control that trade. Measure recall against exact search on a sample, so you know what the index is giving up.

Access control on the underlying data​

A vector index is a copy of your documents in a different format. It needs the same access rules as the source system. OWASP lists vector and embedding weaknesses as LLM08 in its 2025 Top 10 for LLM applications. It recommends permission-aware vector stores, logical partitioning between user groups, and immutable logs of retrieval activity.

In practice, store access metadata such as tenant, group, and classification on each chunk. Apply it as a filter inside the vector search query. Filtering after retrieval is weaker. It can leave the caller with too few results, and restricted text has already reached the service layer.

Hybrid Retrieval: Why Vector Search Alone Isn't Enough​

Dense retrieval, often called semantic search, embeds the query and the documents and matches them by meaning. It handles paraphrase well. A question about "cancelling a subscription" can find a page titled "Terminating your plan." That is the core strength of semantic search.

The weakness shows up with exact strings. Error codes, SKUs, acronyms, product names, and internal jargon often carry little semantic signal. An embedding of "ERR-4471" may land near other error codes rather than the one runbook that mentions it. Sparse retrieval handles this case. It usually means BM25 keyword search. BM25 ranks documents by exact keyword matching, weighted by term frequency and by how rare each term is across the corpus.

The BEIR benchmark is a reason to test both methods rather than assume one will win. It evaluated retrieval systems across 18 datasets. BM25 held up as a strong zero-shot baseline, and in-domain results did not predict out-of-domain performance. Enterprise corpora are full of unfamiliar domains. Your internal vocabulary was not in anyone's training data. Evaluate a hybrid candidate against dense-only and lexical-only baselines.

Hybrid retrieval runs both retrievers and merges their results. The two retrievers do not depend on each other, so run them in parallel rather than one after the other.

Fusing the results​

The two retrievers score on different scales. Raw, uncalibrated scores such as a cosine similarity of 0.82 and a BM25 score of 14 should not be added directly. Normalized or learned score fusion is one option. Reciprocal rank fusion (RRF) avoids the problem by ignoring raw scores and using only rank positions. Cormack, Clarke, and Büttcher introduced it in 2009 with a constant of k = 60. The method fits in a few lines:

def reciprocal_rank_fusion(result_lists, k=60):
    """Merge ranked lists of document IDs into a single ranking."""
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)


# dense_ids and bm25_ids are ranked lists of document IDs
fused_ids = reciprocal_rank_fusion([dense_ids, bm25_ids])

RRF rewards documents ranked highly by multiple retrievers, while allowing a strong result from one retriever to remain competitive. Treat k = 60 as a starting point and tune it against your evaluation set. Many teams also add a cross-encoder reranker after fusion to reorder the final candidates before augmentation.

Hybrid search has a cost. It adds a second index to keep in sync and a fusion step to tune. Whether it earns that cost depends on the queries your RAG systems receive, and only evaluation can answer that.

Evaluation: How Do You Know Your RAG System Is Actually Working?​

A RAG system can fail in two separate places. Retrieval can fetch the wrong chunks, or generation can misuse the right ones. A single "answer quality" score blends both failures, so it cannot tell you which component to fix.

Start with a ground-truth set​

Build an evaluation set of real queries with expected answers. Where possible, record the documents that should support each answer. Pull queries from logs or support tickets rather than inventing them. Include the hard cases, such as exact-identifier lookups, questions that span two documents, and questions with no answer in the corpus.

Run the set against your naive RAG baseline and again on every change to chunking, embeddings, retrieval, or prompts. Without it, you are comparing configurations by impression.

Measure retrieval and generation separately​

The retrieval metrics come from classic information retrieval and need labeled relevant documents for each query. Generation metrics can be partly automated. The RAGAS framework defines reference-free measures of faithfulness, answer relevance, and context relevance. It uses an LLM to check whether each claim in an answer is supported by the retrieved context (Es et al., RAGAS). LLM judges are useful for spotting trends. Calibrate them against a sample of human-graded answers before you trust their absolute numbers.

Read the two layers together​

Low recall with high faithfulness can indicate that the model is staying within incomplete retrieved evidence. Inspect missing relevant chunks and answer completeness before changing generation. High recall with low faithfulness means the evidence arrived but the model drifted from it. Look at the prompt template, the chunk order, or the model itself. A system that answers unanswerable questions with confidence has a grounding problem, whatever its average score says.

Evaluation also needs production signals. By default, log the request ID, the authorized chunk IDs, the model and version, and token usage for every request. Capture query text only when justified, with redaction or access controls and a defined retention period. That record turns a user complaint into a reproducible test case.

Governance: What Changes When Your LLM Calls Out to Your Own Data​

A plain LLM integration has one external dependency, which is the model provider. A RAG endpoint has two data paths with different risks. One reads from your knowledge base. The other sends a prompt, now filled with retrieved content, to a model.

WSO2's guidance on governance that travels with enterprise AI data states the requirement plainly. Every AI-driven data flow needs rate limiting, input validation, output filtering, audit trails, and policy enforcement. For RAG, those controls apply to both data paths, not only to the model call.

Map the traffic flows first​

Governance goes wrong when the architecture diagram is wrong. A RAG request involves three distinct flows:

1. Inbound: The client calls your RAG endpoint.
2. Internal retrieval: The RAG service queries the vector and keyword indexes.
3. Outbound: The RAG service calls the LLM provider with the augmented prompt.

The service receives the inbound request and initiates retrieval and model calls. These flows are not a chain of hops through one proxy, and each flow has its own enforcement point.

Decide who can retrieve what​

The central question is not only who can call the endpoint. It is which documents may be retrieved on a given caller's behalf. The inbound gateway authenticates the caller. The RAG service then receives a verified identity, typically as validated token claims, and applies it as a filter inside the retrieval query. The gateway cannot see individual chunks in the index, so document-level rules must live where the data lives.

Retrieved context needs an audit trail even when it never appears in the final answer. If a restricted chunk reached the prompt, it also reached the model provider. That is why OWASP's LLM08 guidance calls for immutable logs of retrieval activity.

Treat retrieved content as untrusted input​

Retrieved documents enter the prompt alongside your instructions. A document that contains injected instructions can steer the model. OWASP classifies this as indirect prompt injection and notes that RAG does not fully mitigate prompt injection. Screen documents at ingestion and delimit retrieved text clearly in the prompt template. Then inspect model output before it returns to the caller.

How WSO2 fits​

The WSO2 API Platform covers the inbound and outbound enforcement points under a single control plane. The API Gateway fronts the RAG endpoint itself and applies authentication and rate limiting to that inbound traffic. The WSO2 AI Gateway governs the outbound call to model providers. On that path it applies PII masking and redaction, plus JSON Schema and regex checks on output. It also integrates with Azure Content Safety and AWS Bedrock Guardrails. The gateway tracks token consumption and cost by provider and enforces token-level rate limits.

For agentic RAG, the same AI Gateway governs inbound MCP traffic. If you expose retrieval as an MCP tool, the gateway can require a specific scope for each tool. A call without that scope is rejected before it reaches the MCP server. This walkthrough on per-tool authorization at the gateway shows the pattern.

APIs, LLM traffic, and agent access share one governance model on the platform. Inbound, outbound, and tool-level policies are therefore configured and observed in one place. Gateway analytics can also be exported to an OpenTelemetry Collector. That lets you review gateway events alongside the telemetry from your RAG service.

When a gateway layer is worth adding​

A single-team prototype with one model and public documents rarely needs this layer. The case for centralized governance grows with specific operational triggers:

• Multi-model use: You route requests across more than one model or provider.
• Spend attribution: You need to attribute LLM costs to teams, products, or customers. Token-level controls for LLM cost control are the usual starting point.
• Regulated data: Your knowledge base holds personal, regulated, or access-restricted content.
• Production scale: The RAG endpoint serves several consumers with different access rights.

The regulated-data trigger carries real weight. DLA Piper's January 2026 survey puts aggregate GDPR fines at €7.1 billion between May 2018 and January 2026. A RAG pipeline that moves personal data into prompts sits inside that exposure.

Conclusion​

A RAG API succeeds or fails on decisions made away from the model. Chunk boundaries, the embedding model, and the retrieval mix determine what evidence reaches the prompt. The endpoint contract decides whether consumers can trust and trace an answer. An evaluation set tells you which of those choices is holding the system back.

Governance follows the same logic. Once a model reads your own data, access control has to cover retrieval as well as the model call. Inbound traffic, outbound LLM calls, and tool access each need their own enforcement point, and they are easier to run when one control plane manages all three.

Start with a naive baseline and an evaluation set. Add hybrid retrieval, reranking, and gateway policies as your queries, data, and consumers call for them. If your system has reached that point, see how the WSO2 API Platform provides a unified control plane for APIs, AI, and MCP traffic.

Frequently Asked Questions​

What is the difference between dense and sparse retrieval in RAG?​

Dense retrieval compares embeddings to match meaning. Sparse retrieval, such as BM25 keyword search, matches exact terms. Dense retrieval suits paraphrased questions. Sparse retrieval suits identifiers, acronyms, and rare terms.

When should you use hybrid search instead of semantic search alone?​

Use hybrid search when queries mix natural-language questions with exact identifiers, product names, or internal jargon, and confirm the gain on your own evaluation set.

How do you evaluate whether a RAG system is actually working?​

Build a ground-truth set of real queries and expected answers. Score retrieval quality and generation quality separately, so each failure points to the component that caused it.

Does hybrid retrieval always outperform vector-only search?​

Gains depend on the corpus and the query mix, so test it against a dense-only baseline on your own queries.

What governance controls does a RAG API need beyond LLM rate limiting?​

Start with caller authentication and document-level access filtering. Add audit logs of retrieved context, screening for injected instructions, and output guardrails such as PII masking.