back to blog

How to choose the right LLMs for production RAG systems

Read Time 9 mins | Written by: Cole

How to choose the right LLMs for production RAG systems

There's no single best LLM for retrieval augmented generation (RAG): it takes multiple models tuned for the right tasks in the RAG pipeline. RAG's job is to turn a person's or agent's request into a search, find the right documents, pull out the information that matters, and deliver an answer someone can use.

Each of those steps needs a different kind of model, and the right one is usually not the most powerful model from a frontier lab. Accurate answers at a cost that holds up in production come from matching the right model to each part of the pipeline.

Even on the last step, the most powerful model often loses. On Vectara's hallucination leaderboard, which measures how faithfully models answer from the documents they're given, small, cheap models like GPT-5.4 nano and Gemini 2.5 Flash-Lite made things up about 3% of the time. Bigger models like GPT-5.5, Claude Opus 5.5, and Gemini 3 Pro landed between 9% and 14%.

So pick a model for each job, then put a router in front so every query goes to the cheapest model that can handle it.

This short guide assumes you've shipped RAG and are deciding what runs inside it. If you want a refresher on how the pipeline fits together, start with RAG 101.

Why one LLM can't do every job in a RAG pipeline

Every RAG query touches several models before anyone sees an answer. A typical request goes through five steps:

  • Rewrite the question into something the index can search
  • Embed the query and retrieve candidate chunks
  • Rerank the chunks
  • Plan next steps and call tools, if the system is agentic
  • Write the final answer with citations

Rewriting needs speed. Retrieval needs a strong embedding model. Planning needs reasoning and reliable tool calls. The final answer needs faithfulness to the source. One model rarely wins at all four, and paying flagship prices for a rewrite step is how RAG budgets blow up.

Five criteria for choosing an LLM for RAG

  • Faithfulness to retrieved context. Does the model answer from your documents or from what it learned in training? General benchmarks don't measure this well, which is why Vectara's researchers built FaithJudge to score faithfulness across summarization, question answering, and data-to-text tasks.

  • Tool use and planning. Agentic RAG decides what to search, when to search again, and when to stop. That takes a model that calls tools reliably and recovers when a search comes back empty.

  • A context window you can actually use. Spec sheets advertise 1M tokens. Vectara found hallucination rates climb as documents get longer and more complex. Test at the lengths your queries really hit – our LLMs with the largest context windows roundup tracks current limits.

  • Cost and latency per query. Price per million tokens hides the number that matters. One question can trigger several model calls and tens of thousands of tokens of retrieved context. Track cost per answered question.

  • Where it runs. Vendor API, your cloud (Amazon Bedrock, Google Vertex AI, Azure), or your own hardware. Regulated teams usually decide this first, and it narrows everything else.


Mid-tier models for writing grounded RAG answers

For most final answers, each lab's middle tier is the right default – the Claude Sonnet line, OpenAI's mid-tier GPT model, or Gemini Flash. They're fast enough for interactive search, cheap enough to run on every query, and strong enough to synthesize a handful of retrieved chunks.

Two settings matter more than which one you pick. Require citations to the source document and reject answers that don't have them. And keep reasoning effort low on simple lookups. When Vectara rebuilt the leaderboard in November 2025, the reasoning models were the ones that drifted furthest from the source text.

A flagship earns its cost on questions that synthesize dozens of documents, compare two sources side by side, or carry real risk if the answer is wrong. Our most powerful LLMs roundup tracks current flagships and pricing.

Flagship models for agentic RAG planning and tool calls

Agentic RAG hands the model a loop: plan, search, read, and decide whether to search again. A wrong sentence in an answer is one bad output. A wrong plan sends five searches in the wrong direction and burns tokens on every one. This is where each lab's top model pays for itself.

Look for reliable tool calling, graceful recovery when a search returns nothing useful, and the judgment to stop. Coding performance is a decent proxy – coding agents run the same plan-act-check loop, so the models developers trust there tend to plan well in RAG too. Our best LLMs for coding roundup tracks them.

Most teams don't need an agent on every query. A router decides which ones get one.

Model routing sends each RAG query to the cheapest model that can answer it

Model routing ties the tiers together. In the simplest terms, a router sends easy work to cheap models and hard work to expensive ones. It scores each request and picks a model before any tokens get spent.

AT&T shows what this looks like at scale. Its AI usage grew from about 8 billion to more than 45 billion tokens a day while costs stayed roughly flat, according to Fierce Network's interview with Mark Austin, VP in AT&T's Data Office. Routing through LiteLLM cut costs on coding and other advanced AI tasks by as much as 56%, with a 2% drop in quality, PYMNTS reported, citing The Information. Part of that work now runs on open models like Google's Gemma and Nvidia's Nemotron.

"It can figure out when you need to escalate up to a premium model," Austin said. "Doing that smartly is key."

Uber and Ramp have seen similar results – our breakdown of how Uber, AT&T, and Ramp cut AI costs 30–56% with model routing has the numbers.

RAG needs routing more than most AI apps because every query is several model calls. Send the rewrite, the classification step, and the answer all to a flagship, and you pay flagship prices three times per question. Route on three things:

  • Question type. A lookup ("What's the warranty period on part X?") goes to a small model. Synthesis across several documents goes to mid-tier. Open-ended research goes to a flagship agent loop.
  • Whether it needs retrieval at all. Some questions don't need the index. Others work better with whole documents in a long context window than with chunks – Is RAG dead? runs that cost math.
  • Pipeline stage. Small models rewrite and classify, mid-tier models write answers, and flagships plan.

A new class of model fits here too. Decision models like TypeSafe's Jev, Liquid AI's d1, and Microsoft-Decision-1 return a typed answer with a probability – yes or no, one choice from a list, or a score – instead of text. That's the shape of a routing call: does this question need retrieval, which tier should answer it, should the agent search again. Jev answers in about 100 milliseconds and costs $0.042 per million input tokens, according to Future AGI's comparison.

These models are only a couple of months old, so check their confidence scores against labeled examples from your own traffic before you let one make a call on its own.

Cheap routing looks like a win on your billing dashboard within hours, but a drop in answer quality shows up weeks later in user complaints. AT&T measured its 2%. Measure yours with a fixed test set before and after every routing change.

Watch your prompt cache too. Austin pointed out that cached tokens cost about a tenth of fresh ones, so a router that bounces one conversation between models throws that discount away. Our guide to AI cost management for engineering teams covers the rest of the cost picture.

Embedding and reranking models for RAG retrieval 

The answer model can only work with what retrieval hands it. If the right chunk never makes the top 10, no LLM can fix that.

Embedding models turn documents and queries into vectors. Hosted options come from OpenAI, Google (Gemini Embedding 2 handles images, audio, and video too), Voyage, and Cohere. Open embedding models, including Qwen's, run on your own hardware for teams that can't send documents out.

Rerankers rescore the top candidates before they reach the LLM. Cohere Rerank, Voyage's rerankers, and open-source rerankers are the usual picks, and adding one is often the cheapest quality gain in the whole pipeline.

MTEB leaderboard scores are a starting point. Your contracts, support tickets, and scanned drawings aren't in MTEB, so test the top three candidates on a sample of your own documents. Our best RAG tools roundup has current picks.

Open-weight LLMs for self-hosted RAG in regulated industries 

Banks, healthcare systems, and government contractors often can't send documents to a vendor API. Open-weight models from DeepSeek, Qwen, Kimi, GLM, and Llama let them run the whole pipeline on their own infrastructure. They're also the cheap tier a router sends work to – part of AT&T's savings came from moving work to open models.

The trap is licensing, and it shifts between releases:

  • Licenses change between versions. Z.ai moved its GLM models from MIT to a custom license.
  • Sizes in one family can ship under different terms. Qwen 3.8 27B is Apache 2.0, while Qwen 3.8 Max carries a separate license for large commercial deployments.
  • Some licenses restrict by region. Meta's Llama 4 license doesn't grant rights to its multimodal models to companies based in the EU.

Have legal review the license for the exact checkpoint you deploy, not the family name. Our Chinese vs. US open-weight LLMs breakdown compares licenses family by family, and best open-weight LLMs for local AI tracks current models. For when self-hosting pays off, see Hybrid AI: routing work between frontier APIs and open-weight LLMs.

The RAG model stack most teams land on 

  • A small, fast model for query rewriting and classification
  • A mid-tier model for most final answers, with citations required
  • A flagship for agentic planning and the hardest synthesis
  • A router in front, deciding which model gets each call
  • A dedicated embedder and reranker, tested on your data
  • An open-weight fallback where data can't leave your infrastructure

The model names in each slot will change several times a year. The slots won't.

Every company building RAG ends up running more than one model, whether they plan for it or not. Teams that design for it – jobs defined, router in place, test set ready – can swap in a better or cheaper model in a day. Teams that hardcoded one flagship find out what that costs when the invoice arrives.

Codingscape builds production RAG and agentic AI systems with senior, AI-native engineering teams. If you're sorting out models or routing for your own pipeline, let's talk.

Don't Miss
Another Update

Subscribe to be notified when
new content is published
Cole

Cole is Codingscape's Content Marketing Strategist & Copywriter.