Do million-token context windows replace RAG?

Why RAG became the default

Context windows used to be small. Feeding a model a paragraph or two of company documents often hit the limit. For a knowledge base too large for those windows, RAG was a necessity.

RAG (retrieval augmented generation) became the standard way to give AI systems access to proprietary knowledge. You split the knowledge base into chunks, turn each chunk into a numerical embedding and store it in a vector database such as Qdrant, Pinecone or ChromaDB. At query time, you retrieve the most relevant chunks and add them to the prompt.

Tools like n8n.io simplify this. For example, the n8n workflow integrates Google Drive, Gemini and a Qdrant vector store to build a RAG chatbot for your documents.

Earlier models had context limits of 8k, 32k or 128k tokens. Those limits forced workarounds such as dropping messages, summarizing, or retrieving from a vector database.

Google's Gemini series now offers context windows of one million tokens or more. A million tokens can hold several novels or hundreds of podcast transcripts.

That raises a question for anyone building AI solutions. With so much room in the prompt, do we still need RAG?

The answer depends on size and complexity

The answer is not a simple yes or no. It depends on the size, nature and complexity of your knowledge base, and on the accuracy your application needs.

When putting everything in the prompt works

For a knowledge base of a few hundred pages, or roughly 200,000 tokens, a long-context model can take the whole thing in the prompt. Google's documentation for Gemini's long context capabilities says Gemini 1.5 Pro can process up to 1 million tokens, and up to 10 million in private preview. That is about 700,000 words, or many hours of video or audio. Google calls this a "more direct approach: providing all relevant information upfront."

This direct method lets the model use its in-context learning across the whole dataset at once. If the dataset fits and the queries are simple, such as finding facts or summarizing, you can skip a separate RAG pipeline and simplify the architecture. Prompt caching makes repeated queries over the same context faster and cheaper when you use the API.

When RAG and its more advanced forms still matter

RAG remains necessary when your knowledge base grows beyond what even a million-token window can hold. It is also needed when retrieval and synthesis get more complex.

A large window is not always the most effective or efficient option, even when it could hold all your data. Naive RAG has known shortcomings with complex, interconnected information:

• Fragmented context. RAG splits documents into chunks, and a chunk retrieved alone may lack the surrounding context. A sentence about revenue growth means nothing without the company and time period, which may sit in a chunk that was not retrieved.
• Structured and tabular data. Basic RAG often struggles with trends in spreadsheets, or with questions that need aggregation across rows or tables. It may retrieve only a small, unrepresentative slice of the data.
• Cross-document blindness. Connecting facts spread across several documents is a hard problem for simple RAG.
• Retrieval accuracy. Even getting the right chunks can fail. Benchmarks show non-trivial failure rates for basic RAG, which is often unacceptable in production.

A large context window does not fix a flawed retrieval. If RAG missed key information or pulled irrelevant snippets, the model still gets the wrong input. LLMs handle needle-in-a-haystack tasks, which find one fact in a long text, but their performance can drop when they search for several needles.

Beyond basic RAG

Building agents that use large, complex knowledge bases usually means moving past naive RAG. Several techniques go further:

1. Contextual retrieval. Teams such as Anthropic have benchmarked this approach, and it improves retrieval accuracy. The core idea is to enrich chunks before they are embedded and indexed. An LLM writes a short context for each chunk, based on the surrounding document or related documents. That context is added to the chunk, and queries search the enriched chunks. Anthropic's research shows that this technique alone reduces retrieval failures. Combined with lexical search (BM25) and reranking, the gains are larger. Prompt caching makes the upfront context generation cheaper.
2. Agentic RAG. In a standard setup, RAG is a passive tool that the LLM calls. In agentic RAG, the LLM reasons about the retrieval process itself. The agent can check the results of an initial lookup. If they are not enough, for example when it needs a whole spreadsheet rather than a few rows, it can use other tools. It can list documents by title, retrieve a full document, or query tabular data with SQL-like commands. That flexibility gets past many limits of fixed RAG pipelines.
3. Hybrid search. This combines semantic search, which matches meaning through embeddings, with lexical search such as BM25, which finds exact keywords. Specific technical terms, error codes and unique identifiers are then not missed by embedding models alone.
4. Reranking. An initial retrieval can pull hundreds of potentially relevant chunks. A reranker, often a smaller specialized LLM or a cross-encoder, scores them by relevance to the query. Only the top K chunks, such as the best 20, go to the main LLM. This filters noise, improves answers, and cuts the cost and latency of the final LLM call.
5. Knowledge graphs. For very large, interconnected knowledge bases, knowledge graphs can complement or replace pure vector search. Frameworks such as LightRAG build a graph of the relationships between topics, ideas and concepts in your documents, next to the vector store. A query can use vector retrieval and graph traversal together, which gives deeper context. This matters as knowledge bases grow into thousands or hundreds of thousands of documents, where naive RAG can start to degrade.

Your database and file processing matter too. Vector databases such as Qdrant or Chroma, or hybrid setups with Postgres, affect RAG quality, as do tools like Unstructured.io for complex file types. For real-time, constantly changing data, dynamic knowledge graph solutions are still being developed.

To RAG, evolve RAG, or go direct?

Million-token windows in models like Gemini change the picture. For smaller, self-contained knowledge bases, putting all the relevant information in the prompt is simpler. It may remove the need for a separate RAG pipeline.

For large, complex, growing datasets, some form of RAG is still needed for scalable access to external knowledge. Basic RAG is often just the starting point. Contextual retrieval, agentic RAG, hybrid search, reranking and knowledge graphs are becoming necessary parts of accurate agents that use vast knowledge stores.