A normal chatbot answers from what the model already knows. A RAG chatbot answers from your documents.

That is the whole reason Retrieval-Augmented Generation, or RAG, became such a common LLM pattern. You take your company docs, product pages, PDFs, support articles, onboarding guides, policies, or knowledge base content, turn them into searchable chunks, retrieve the most relevant pieces, and pass those pieces into an LLM before it answers.

The result is a chatbot that can answer questions using your own data instead of guessing from general training.

In this guide, we’ll build a simple RAG chatbot architecture and compare how to do it with OpenAI, Cohere, Google Gemini, Pinecone, and other tools. We’ll also show where LLMAPI fits when you want one app that can route between several model providers.

Quick version

A RAG chatbot has five main parts:

Step	What happens
1. Load documents	Add PDFs, web pages, docs, FAQs, or database content
2. Chunk text	Split documents into small searchable sections
3. Create embeddings	Turn chunks into vectors
4. Retrieve context	Search for chunks related to the user question
5. Generate answer	Send the question + retrieved context to an LLM

For quick prototypes, OpenAI and Google now offer more managed file-search style options. For custom production systems, many teams use an embedding model, a vector database like Pinecone, and an LLM provider such as OpenAI, Cohere, Google, Anthropic, Mistral, or another model through LLMAPI.

Why we can write this guide

Our team spends a lot of time researching AI APIs, LLM workflows, retrieval systems, model routing, and developer tooling. For this guide, we reviewed official docs and current RAG research, including:

Resource	Why it matters
OpenAI embeddings docs	Shows how to turn text into vectors for search
OpenAI File Search docs	Shows OpenAI’s managed vector store and file retrieval path
Cohere RAG docs	Covers RAG with Cohere Chat, Embed, and Rerank
Cohere RAG complete example	Practical end-to-end RAG workflow
Google Gemini embeddings docs	Covers embeddings for semantic search and RAG
Google Gemini File Search docs	Native RAG with file import, chunking, indexing, and retrieval
Pinecone docs	Vector database setup for storing and searching chunks
Hybrid retrieval and reranking research	Shows why retrieval quality and reranking matter
ARAGOG RAG evaluation paper	Compares RAG methods and retrieval strategies
LiR³AG rerank reasoning paper	Looks at reranking and reasoning tradeoffs in RAG

We’ll keep this practical: how the pieces fit, which provider suits which use case, and how to avoid building a chatbot that sounds confident while using the wrong source.

What is a RAG chatbot?

A RAG chatbot combines search with generation.

First, it searches your knowledge base. Then it asks an LLM to answer using the retrieved text.

That makes RAG useful for:

Use case	Example
Customer support	“How do I reset my billing settings?”
Internal company assistant	“What is our PTO policy?”
Developer docs chatbot	“How do I authenticate API requests?”
Legal document search	“What does this contract say about renewal?”
HR onboarding	“Which benefits start after 30 days?”
E-commerce assistant	“Which product is best for dry skin?”
Healthcare admin	“Which intake form is needed for this appointment?”
Finance research	“Summarize risk factors from these filings.”

The main benefit is grounding. The chatbot can cite or reference the documents it used, which makes answers easier to verify.

RAG architecture

Here is the basic architecture:

Documents
↓
Text extraction
↓
Chunking
↓
Embeddings
↓
Vector database
↓
Retrieval
↓
Optional reranking
↓
LLM answer
↓
Citations and review

Each piece matters.

Component	What to choose
Document loader	PDF parser, web scraper, CMS export, database query
Chunking	Fixed-size chunks, heading-based chunks, semantic chunks
Embeddings	OpenAI, Cohere, Google, Voyage, BGE, E5
Vector database	Pinecone, Weaviate, Qdrant, Chroma, pgvector
Retriever	Similarity search, hybrid search, metadata filters
Reranker	Cohere Rerank, cross-encoder, LLM reranking
Generator	OpenAI, Cohere, Google Gemini, Anthropic, Mistral
Evaluation	RAGAS, custom QA set, citation checks, human review

A weak retrieval layer will hurt the whole chatbot. If the model gets the wrong chunks, even a strong LLM may produce a weak answer.

Provider comparison

Here is the practical difference between common RAG providers.

Provider	Best for	Main strength	Watch out for
OpenAI	Fast RAG apps and strong generation	Embeddings, file search, strong models	Costs can rise with large context/output
Cohere	Enterprise search and reranking	Embed + Rerank + Chat workflow	Best value appears when reranking is used well
Google Gemini	Gemini-native RAG and multimodal retrieval	Embeddings and File Search tool	API/product surface can change quickly
Pinecone	Production vector search	Managed vector database	You still need embeddings and LLMs
Qdrant	Open-source/self-hosted vector search	Flexible deployment	More infrastructure work
Weaviate	Hybrid search and knowledge apps	Vector + keyword search	Setup choices matter
Chroma	Local prototypes	Easy developer experience	Not always ideal for high-scale production
pgvector	Postgres-native search	Simple stack if you already use Postgres	Needs tuning for large-scale search
LLMAPI	Multi-provider model routing	One gateway for models, cost, fallback	Works around the RAG stack, not as the vector DB

For a small demo, use OpenAI or Google File Search.

For enterprise retrieval, look closely at Cohere Rerank and a vector database.

For a production SaaS app, use a dedicated vector database plus model routing through LLMAPI.

Option 1: Build RAG with OpenAI

OpenAI is a strong choice if you want a simple developer path and high-quality generation.

OpenAI’s embeddings docs explain how embeddings turn text into numbers for search, clustering, recommendations, and classification. OpenAI also has File Search, which uses vector stores to retrieve information from uploaded files.

OpenAI is a good fit when:

Need	Why OpenAI fits
Fast prototype	Simple docs and strong API ecosystem
High-quality answers	Strong generation models
Managed file search	Less custom vector DB work
Developer-friendly workflow	Good examples and SDKs
Multimodal expansion	Useful if the chatbot later handles images or files

A simple OpenAI-style RAG setup:

Upload docs → Create embeddings or vector store → Retrieve matching chunks → Send context to OpenAI model → Answer with sources

If you want more control, use OpenAI embeddings with Pinecone, Qdrant, Weaviate, Chroma, or pgvector.

Option 2: Build RAG with Cohere

Cohere is especially strong when retrieval quality matters.

Cohere’s RAG docs explain how to use Chat with retrieved documents, while the complete RAG example shows a workflow with Chat, Embed, and Rerank.

Cohere’s Rerank product is useful because the first retrieval pass often brings back noisy chunks. Reranking helps choose the best pieces before sending them to the LLM.

Cohere is a good fit when:

Need	Why Cohere fits
Enterprise search	Strong retrieval and reranking focus
Better source quality	Rerank filters noisy chunks
Lower context waste	Better chunks reduce prompt size
Multilingual retrieval	Cohere has strong multilingual retrieval options
Search-heavy chatbot	Retrieval quality matters more than model flashiness

A Cohere-style RAG flow:

User question → Embed query → Retrieve candidate chunks → Rerank chunks with Cohere Rerank → Send top chunks to Cohere Chat or another LLM → Return answer with citations

Reranking is one of the most useful upgrades for production RAG. A 2026 paper on hybrid retrieval and reranking for evidence-grounded RAG also points in this direction: retrieval quality, reranking, conservative prompting, and claim-level evaluation all affect grounded answers.

Option 3: Build RAG with Google Gemini

Google is a good choice if you already use Gemini, Google Cloud, or Google AI Studio.

Google’s Gemini embeddings docs describe embeddings for semantic search, classification, clustering, and RAG. Google’s File Search docs say the Gemini API can handle RAG through a File Search tool that imports, chunks, indexes, and retrieves files.

Google is a good fit when:

Need	Why Google fits
Gemini-based chatbot	Natural model fit
Managed file retrieval	File Search handles storage/chunking/indexing
Google Cloud stack	Easier infrastructure alignment
Multimodal RAG	Gemini ecosystem supports multimodal direction
Fast document chatbot	Less custom retrieval setup

A Gemini File Search setup can look like this:

Import files → Google chunks and indexes them → User asks question → Gemini retrieves relevant file context → Gemini answers with grounded information

This is useful for teams that want less custom RAG plumbing. For deeper control, use Gemini embeddings with an external vector database.

Option 4: Use Pinecone or Another Vector Database

A managed file-search API is convenient, but many production teams still prefer a dedicated vector database.

Pinecone supports vector indexes and integrated embedding workflows. Other common choices include Qdrant, Weaviate, Chroma, and pgvector.

Use a vector database when:

Need	Why it helps
Large document corpus	Better control over indexing and search
Metadata filtering	Filter by user, team, date, product, permission
Multi-provider setup	Use any embedding model and any LLM
Observability	Track retrieval quality
Hybrid search	Combine keyword and vector search
Fine-tuned retrieval	Tune chunking, filters, top-k, reranking

A production vector DB flow:

This setup takes more work, but it gives you more control.

Step-by-step RAG chatbot build

Here is a simple provider-neutral workflow.

Step 1: Install dependencies

For a basic Python prototype:

pip install openai python-dotenv numpy

If you want a local vector index:

pip install chromadb

For a production vector DB, use the SDK for Pinecone, Qdrant, Weaviate, or your database of choice.

Step 2: Prepare documents

Start small. Use 5 to 20 documents first.

documents = [
    {
        "id": "refund_policy",
        "text": "Customers can request a refund within 30 days of purchase..."
    },
    {
        "id": "shipping_policy",
        "text": "Standard shipping takes 3-5 business days..."
    },
    {
        "id": "account_security",
        "text": "Users can enable two-factor authentication from account settings..."
    }
]

A real app may load documents from PDFs, Notion, Google Docs, Zendesk, Confluence, Markdown files, or a CMS.

Step 3: Chunk the text

Chunking affects quality a lot. Chunks that are too small lose context. Chunks that are too large add noise.

def chunk_text(text, chunk_size=500, overlap=80):
    chunks = []
    start = 0

    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap

    return chunks

Create chunks:

all_chunks = []

for doc in documents:
    for i, chunk in enumerate(chunk_text(doc["text"])):
        all_chunks.append({
            "id": f"{doc['id']}_{i}",
            "source": doc["id"],
            "text": chunk
        })

For production, prefer heading-aware chunking. Keep section titles with chunks so the model has context.

Step 4: Create embeddings

With OpenAI embeddings:

from openai import OpenAI

client = OpenAI()

def embed_text(text):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

Embed chunks:

for chunk in all_chunks:
    chunk["embedding"] = embed_text(chunk["text"])

You can swap this step for Cohere embeddings, Google embeddings, Voyage embeddings, or open-source embedding models.

Step 5: Search similar chunks

For a tiny demo, use cosine similarity in memory.

import numpy as np

def cosine_similarity(a, b):
    a = np.array(a)
    b = np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def retrieve(query, chunks, top_k=3):
    query_embedding = embed_text(query)

    scored = []

    for chunk in chunks:
        score = cosine_similarity(query_embedding, chunk["embedding"])
        scored.append((score, chunk))

    scored.sort(reverse=True, key=lambda x: x[0])

    return [chunk for score, chunk in scored[:top_k]]

Test retrieval:

question = "How long do customers have to request a refund?"
matches = retrieve(question, all_chunks)

for match in matches:
    print(match["source"], match["text"])

For production, replace this with Pinecone, Qdrant, Weaviate, Chroma, or pgvector.

Step 6: Generate an answer

Now pass the retrieved chunks to an LLM.

def build_context(chunks):
    return "\n\n".join(
        f"Source: {chunk['source']}\n{chunk['text']}"
        for chunk in chunks
    )

def answer_question(question):
    chunks = retrieve(question, all_chunks)
    context = build_context(chunks)

    prompt = f"""
Answer the question using only the context below.
If the answer is not in the context, say you do not know.

Context:
{context}

Question:
{question}
"""

    response = client.responses.create(
        model="gpt-5.5",
        input=prompt
    )

    return response.output_text

Run it:

print(answer_question("How long do customers have to request a refund?"))

Expected answer:

Customers can request a refund within 30 days of purchase.

This is the basic RAG chatbot.

Add citations

Citations make RAG more trustworthy.

Modify the prompt:

prompt = f"""
Answer using only the context below.
Cite the source ID for each factual claim.
If the answer is missing, say you do not know.

Context:
{context}

Question:
{question}
"""

A good answer should look like:

Customers can request a refund within 30 days of purchase. Source: refund_policy.

Citations help users verify the answer. They also help your team debug retrieval issues.

Add a reranker

The first vector search pass may retrieve chunks that are close but not actually useful.

A reranker helps sort candidate chunks more carefully.

Cohere is popular here because its docs show RAG workflows with Embed, Rerank, and Chat. You can retrieve 20 chunks from your vector DB, rerank them, then send only the top 3 to the LLM.

Retrieve top 20 chunks → Rerank with Cohere → Keep top 3-5 chunks → Send to LLM

This helps with:

Problem	How reranking helps
Similar but wrong chunks	Pushes better evidence higher
Too much context	Sends fewer chunks
Higher token cost	Reduces prompt size
Weak citations	Improves source relevance
Multi-document confusion	Chooses better passages

A 2025 paper, LiR³AG, also focuses on reranking and reasoning strategies for RAG. Its core point is useful for builders: better evidence organization can reduce token overhead and latency while improving answer quality.

Add conversation memory

For a chatbot, users ask follow-up questions.

Example:

User: What is the refund window?

Bot: Customers can request refunds within 30 days.

User: Does that apply to subscriptions too?

The second question depends on the first. You can handle this by rewriting follow-up questions into standalone queries.

Original follow-up:

Does that apply to subscriptions too?

Rewritten query:

Does the 30-day refund window apply to subscriptions?

A simple approach:

def rewrite_query(chat_history, question):
    prompt = f"""
Rewrite the user's latest question as a standalone search query.

Chat history:
{chat_history}

Latest question:
{question}
"""

    response = client.responses.create(
        model="gpt-5.5-mini",
        input=prompt
    )

    return response.output_text

Then retrieve using the rewritten query.

Add guardrails

RAG bots should avoid making up answers.

Use a strict prompt:

Use only the provided context.

If the answer is not present, say: “I don’t have enough information in the provided documents.”

Do not guess.

Cite sources.

Also add checks:

Check	Why it matters
Minimum retrieval score	Avoid weak context
Source count	Require at least one strong source
Citation required	Forces grounding
Answer length limit	Reduces rambling
“I don’t know” path	Prevents guessing
Review flag	Helps for sensitive topics
Access control	Prevents data leaks across users

Access control is especially important. Your vector DB should filter by user, team, permission, or workspace before retrieving chunks.

Evaluate the RAG chatbot

Do not judge a RAG chatbot only by vibes. Build a small test set.

Test question	Expected source	Expected answer
What is the refund window?	refund_policy	30 days
How long does shipping take?	shipping_policy	3-5 business days
How do I enable 2FA?	account_security	Account settings

Track:

Metric	What it tells you
Retrieval accuracy	Did the right chunks show up?
Answer faithfulness	Is the answer supported by sources?
Citation quality	Are citations useful?
Refusal quality	Does it say “I don’t know” when needed?
Latency	Is the chatbot fast enough?
Cost per answer	Can the workflow scale?
User satisfaction	Did people get useful answers?

The ARAGOG RAG evaluation paper is useful because it compares RAG methods by retrieval precision and answer similarity. The practical takeaway: retrieval strategy matters, and the best method depends on your documents and questions.

Where LLMAPI fits

RAG apps often need more than one model.

You may want:

Task	Model type
Query rewrite	Cheap fast model
Answer generation	Stronger model
Citation checking	Reasoning model
Summarization	Long-context model
Fallback	Backup provider
Evaluation	Judge model

LLMAPI can help route these tasks across OpenAI, Cohere, Google, Anthropic, Mistral, and other providers through one workflow.

Example:

1. User question
2. LLMAPI routes query rewrite to a cheap model
3. Vector DB retrieves chunks
4. Cohere reranks chunks
5. LLMAPI routes answer generation to OpenAI or Gemini
6. LLMAPI routes fallback if provider fails
7. Track usage and cost

This is useful when you want model flexibility without rebuilding your app every time you test a new provider.

Which stack should you choose?

Here is the practical version.

Team need	Suggested stack
Fastest prototype	OpenAI File Search or Google Gemini File Search
Strong enterprise retrieval	Cohere Embed + Cohere Rerank + vector DB
Production SaaS chatbot	Pinecone/Qdrant + OpenAI/Cohere/Google + LLMAPI
Google Cloud team	Gemini embeddings + Gemini File Search or Vertex AI stack
AWS-heavy team	Bedrock Knowledge Bases + vector store + Cohere rerank
Open-source stack	BGE/E5 embeddings + Qdrant/Chroma + open model
Low-cost internal bot	Open-source embeddings + pgvector + cheaper LLM
Multimodal document RAG	Gemini File Search or multimodal-capable provider
High reliability	Vector DB + reranker + LLMAPI fallback routing

For most teams, start simple. Build one working RAG chatbot with a small document set. Then add reranking, metadata filters, citations, evaluation, and model routing.

Common mistakes

Mistake	Better approach
Chunking documents randomly	Keep headings and context
Sending too many chunks	Retrieve more, rerank, then send fewer
Skipping citations	Cite sources for trust and debugging
Ignoring access control	Filter by user/team permissions
Using only vector search	Add keyword or hybrid search for exact terms
No “I don’t know” behavior	Add strict refusal instructions
No test set	Build 20-50 real questions
One model for every task	Route cheap tasks and premium tasks separately
Evaluating only final answers	Evaluate retrieval and generation separately

Final thoughts

A RAG chatbot is one of the most useful LLM apps because it connects a model to real knowledge.

Use OpenAI if you want a fast, high-quality developer path. Use Cohere if retrieval and reranking matter a lot. Use Google Gemini if you want Gemini-native file search, embeddings, and multimodal direction. Use Pinecone, Qdrant, Weaviate, Chroma, or pgvector if you want more control over vector storage and retrieval.

For a production-grade system, focus on retrieval quality first. Good chunks, useful metadata, reranking, citations, and evaluation will do more for answer quality than swapping models every day.

Once the chatbot uses several models or providers, add LLMAPI as the routing layer. It can help route tasks, manage fallback, compare costs, and keep the RAG workflow flexible as models change.