RAG vs Long Context: Which Should You Use (or Both)?
Choosing between RAG vs long context is one of the more consequential architecture decisions you make when building LLM-powered applications. Both approaches solve the same problem — a model's built-in knowledge has a cutoff date and no awareness of your private data — but they solve it differently, and the right choice (or mix) depends on your corpus size, latency budget, and how much you want to pay per call. This guide works through both options honestly, with the cost math and a decision framework you can apply to your next feature.
The Core Problem Both Approaches Solve
A language model is a frozen artifact. Its weights encode knowledge up to a training cutoff, and it has no live access to your database, your documentation, or last week's customer tickets. Every production system that needs a model to reason about external data has to inject that data at inference time — and there are really only two ways to do it.
Retrieval-Augmented Generation (RAG) embeds your corpus into a vector store, runs a similarity search at query time, and injects the top-k most relevant chunks into the prompt. The model only sees the small slice of your data that's probably relevant.
Long-context stuffing takes the opposite stance: load as much of your corpus as will fit into a large context window and let the model find the relevant parts itself. With models now offering windows of 128K to 1M tokens, this is increasingly viable — but viability and affordability are different things.
RAG: What It Gets Right and Where It Breaks Down
RAG's big win is efficiency. If your corpus is 500 MB of documentation but the answer to any given query lives in a 2K-token passage, you only pay for those 2K input tokens — not the full corpus. At scale this is a dramatic cost difference. RAG also handles freshness naturally: update your vector store whenever your source data changes, and the model immediately has access to the new information without any retraining.
The catch is that retrieval quality is the hard ceiling on RAG quality. If the embedding model fails to surface the right chunk — because the query and the answer use different vocabulary, because the relevant information is spread across several documents, or because your chunking strategy split a key passage mid-sentence — the model gets bad context and produces a bad answer. Garbage in, garbage out, and the garbage is invisible to the caller.
Common RAG failure modes worth knowing:
- Vocabulary mismatch — the query says "cost" but the docs say "pricing"; embeddings don't bridge the gap perfectly.
- Multi-hop reasoning — the answer requires connecting facts from two separate documents that don't appear near each other in embedding space.
- Chunking artifacts — context that should be read together is split across chunks and never retrieved together.
- Stale index — the vector store lags the source of truth by hours or days if your ingestion pipeline has any friction.
Good RAG engineering — hybrid search, rerankers, query expansion, thoughtful chunk overlap — can push retrieval quality surprisingly high. But it adds engineering surface area and still doesn't eliminate the retrieval bottleneck.
Long Context: What It Gets Right and Where It Breaks Down
Long-context models are appealing because they eliminate the retrieval layer entirely. Dump your whole document set into the prompt, ask your question, and the model figures out what's relevant. This works remarkably well for corpora that fit — a 30-page contract, a codebase, a set of meeting transcripts — and it's far simpler to implement than a full RAG pipeline.
The well-documented failure mode is called "lost in the middle." Multiple studies have shown that transformer attention is biased toward the beginning and end of a long context; information buried in the middle receives less effective attention and is more likely to be ignored or misattributed. This effect worsens as context length grows, and it's model-dependent — some architectures handle long context better than others, but none are immune.
The other problem is cost. Unlike RAG, where you only pay for the retrieved slice, long-context stuffing bills you for every input token on every call. At 100K input tokens per request, even modest traffic generates massive token volumes — see the breakdown in AI tokenomics. Prompt caching mitigates this for static prefixes (the same system prompt or document set repeated across calls gets a cached-input discount from most providers), but caching requires that the prefix is identical and in position, which constrains your prompt structure.
There are also latency implications. Prefill — processing all those input tokens before the first output token is generated — scales with input length. A 200K-token prompt means a slower time-to-first-token even on fast hardware, which matters for interactive applications.
Side-by-Side Comparison
| Dimension | RAG | Long Context |
|---|---|---|
| Corpus size limit | Effectively unlimited — corpus lives in the vector store | Hard limit at the model's context window |
| Cost per call | Low — only retrieved chunks are billed as input | High — full stuffed context billed every call |
| Latency | Adds retrieval latency; short prompt = fast prefill | No retrieval step; long prefill can be slow |
| Freshness | Good if index is kept current | Instant — update the file, resubmit |
| Implementation complexity | High — embedding, vector DB, retrieval tuning | Low — read files, concatenate, call API |
| Quality ceiling | Capped by retrieval quality | Capped by "lost in the middle" degradation |
| Prompt caching benefit | Limited — retrieved chunks vary per query | High — static corpus prefix can be cached |
| Best corpus size | Large (millions of documents) | Small-to-medium (fits in window with room to spare) |
When to Use RAG
RAG is the right default when your corpus is large enough that it cannot fit in any context window, when freshness matters and your data changes frequently, or when you're running a high-QPS service where per-call input token costs compound painfully. It's also the correct approach when you have natural unit boundaries in your data (individual documents, support tickets, product pages) that a good chunking strategy can exploit.
Invest in the retrieval quality — a hybrid BM25 + dense retrieval setup with a cross-encoder reranker adds latency but significantly reduces the retrieval failure rate. The bottleneck in RAG is almost always retrieval, not generation.
When to Use Long Context
Long context wins when your corpus is small enough to fit with headroom, when the information needed to answer a query is genuinely scattered across the document (multi-hop, cross-section reasoning), or when speed of iteration matters more than cost — no embedding pipeline to maintain, no index to keep fresh, no chunking strategy to tune. Code review, contract analysis, and technical due diligence are natural fits.
It's also worth using when you have a good reason to believe prompt caching will cover most of your input tokens. If the corpus is static and sits at the start of the prompt, many providers will cache that prefix at a steep discount, making long-context calls much cheaper on the second and subsequent queries.
The Hybrid: Retrieve, Then Expand
For demanding workloads, the best results often come from combining both techniques. Run RAG to identify the most relevant chunks, then expand those chunks with their surrounding context — enough to eliminate chunking artifacts — and pass the expanded result to a model with a generous context window. This uses retrieval to narrow the search space and avoids the cost of stuffing the whole corpus, while giving the model enough surrounding context to reason coherently.
A related pattern is two-stage retrieval: a cheap, fast model ranks candidate chunks, and a more capable model does the final reasoning over the top candidates. This is structurally similar to how search engines work — cheap ranking at scale, expensive scoring only for the finalists.
Routing: Matching the Job to the Right Model
Whether you use RAG, long context, or a hybrid, you're almost certainly not sending every query to the same model. Long-context jobs require a model with a large window; RAG jobs can run on smaller, cheaper models because the context is already compact. That routing decision — which model handles which request — is where a lot of cost savings live in production.
An LLM gateway handles this routing automatically. You define rules (or let the gateway route based on token count thresholds), and each request goes to the appropriate model without changes to application code. You also get unified cost accounting across providers, so you can see exactly what each routing decision costs rather than reconciling invoices from five different providers at the end of the month.
flo2 is a developer-first LLM gateway that routes requests across providers with zero token markup — you pay provider prices directly using your own API keys. It's a practical way to run long-context jobs to big-window models and RAG jobs to cheaper ones within a single API surface, with per-request cost tracking built in. Free during beta.
- LLM context windows — how context windows work, why bigger isn't always better, and how to stay inside the limit without losing quality.
- AI tokenomics — the unit economics of tokens, and the six levers that reliably reduce LLM spend.