mcpRAG

mcpRAG - Retrieval-Augmented Generation System

A production-ready RAG (Retrieval-Augmented Generation) implementation combining open-source embeddings, FAISS vector database, and Gemini LLM for accurate document-based question answering.

Overview

This project implements a complete RAG pipeline for querying text documents using:

• Embeddings: Nomic embeddings via Ollama (local, open-source)
• Vector Database: FAISS (Facebook AI Similarity Search)
• LLM: Gemini 2.0 Flash (fast, cost-effective)
• Document Processing: Chunking with overlap for context preservation
• Incremental Updates: Add new documents without rebuilding entire index

Architecture

Documents (TXT) → Chunking (60 words, 10-word overlap)
                ↓
         Nomic Embeddings (Ollama, 768 dimensions)
                ↓
         FAISS Vector Index (L2 distance similarity)
                ↓
User Query → Query Embedding → FAISS Search → Top-K Chunks
                                              ↓
                                     Context + Query → Gemini LLM
                                              ↓
                                           Answer

Key Features

• Local Embeddings: Uses Ollama with Nomic model - no external API calls for embeddings
• Efficient Search: FAISS IndexFlatL2 for fast similarity search
• Chunking Strategy: 60-word chunks with 10-word overlap to preserve context
• Incremental Updates: appendEmbeds.py adds new documents without rebuilding
• Metadata Tracking: Stores document name, chunk ID, and chunk text for traceability
• Persistent Storage: Index and metadata saved locally for reuse

Installation

Prerequisites

• Python 3.11+
• Ollama installed and running locally
• Gemini API key
• Nomic embedding model downloaded in Ollama

Setup

1. Install Ollama (if not already installed):

# macOS/Linux
curl -fsSL https://ollama.ai/install.sh | sh

# Windows
# Download from https://ollama.ai/download

2. Pull the Nomic embedding model:

ollama pull nomic-embed-text

Verify Ollama is running:

curl http://localhost:11434/api/embeddings -d '{"model":"nomic-embed-text","prompt":"test"}'

3. Clone and navigate to the project:

cd mcpRAG

4. Create virtual environment and install dependencies:

python -m venv .venv

# Windows
.venv\Scripts\activate

# macOS/Linux
source .venv/bin/activate

pip install -r requirements.txt

5. Configure API credentials:

Create a .env file:

GEMINI_API_KEY=your_gemini_api_key_here

Get your API key from Google AI Studio

6. Prepare your documents:

Place your .txt files in the docfiles/ or sapfiles/ directory.

Usage

1. Create Initial Embeddings and Index

Run ollamaEmbed.py to process your first document and create the FAISS index:

# Edit ollamaEmbed.py to set your document path
file_path = Path("C:\\path\\to\\your\\document.txt")

# Then run
python ollamaEmbed.py

What it does:

• Reads the specified text file
• Chunks text into 60-word segments with 10-word overlap
• Generates embeddings using Ollama's Nomic model
• Creates FAISS index with L2 distance metric
• Saves index to my_faiss_index.faiss
• Saves metadata to my_metadata.json

Output:

Processing document.txt...
✅ Indexed 142 chunks
✅ Saved FAISS index to my_faiss_index.faiss
✅ Saved metadata to my_metadata.json

2. Query the RAG System

Run ragModel.py to search the index and generate answers:

# Edit ragModel.py to set your query
query = "how does SAP integrate with other systems?"
top_k = 5  # Number of relevant chunks to retrieve

# Then run
python ragModel.py

What it does:

1. Loads the FAISS index and metadata
2. Converts your query to embeddings using Nomic
3. Searches FAISS for the top-K most similar chunks
4. Passes retrieved context + query to Gemini 2.0 Flash
5. Returns a contextual answer

Example Output:

Query: how does SAP integrate with other systems?

Result 1:
  Document: SAP_Joule.txt
  Chunk: SAP integrates with various external systems through APIs...
  Chunk ID: SAP_Joule_42
--------------------

Gemini Response: SAP integrates with other systems primarily through REST APIs
and OData services. The integration layer supports real-time data synchronization...

3. Add New Documents to Existing Index

Run appendEmbeds.py to incrementally add documents without rebuilding:

# Edit appendEmbeds.py to set your new document path
new_file_path = "C:\\path\\to\\new_document.txt"

# Then run
python appendEmbeds.py

What it does:

• Loads existing FAISS index and metadata
• Processes the new document (chunking + embeddings)
• Appends new embeddings to existing FAISS index
• Updates metadata with new chunks
• Saves updated index and metadata

Output:

Processing SAP_GENAI.txt...
✅ Added 87 new chunks to the index
✅ Saved updated FAISS index to my_faiss_index.faiss
✅ Saved updated metadata to my_metadata.json

Project Files

File	Purpose
ollamaEmbed.py	Initial document processing, embedding generation, FAISS index creation
ragModel.py	Query interface - search index and generate answers with Gemini
appendEmbeds.py	Incrementally add new documents to existing index
requirements.txt	Python dependencies
.env	API credentials (create this file)
my_faiss_index.faiss	Saved FAISS vector index (auto-generated)
my_metadata.json	Chunk metadata with document names and text (auto-generated)
docfiles/	Directory for source documents
sapfiles/	Directory for SAP-related documents

Configuration

Chunking Parameters

Adjust in any of the Python files:

CHUNK_SIZE = 60        # Number of words per chunk
CHUNK_OVERLAP = 10     # Overlapping words between chunks

Recommendations:

• Smaller chunks (30-50 words): Better precision, more chunks, slower indexing
• Larger chunks (80-100 words): More context, fewer chunks, faster indexing
• Overlap (10-20 words): Prevents context loss at chunk boundaries

Retrieval Settings

In ragModel.py, adjust top_k:

top_k = 5  # Number of chunks to retrieve

Recommendations:

• top_k=3: Fast, focused answers
• top_k=5: Balanced (default)
• top_k=10: More comprehensive context, slower

LLM Model

Change Gemini model in ragModel.py:

model="gemini-2.0-flash"      # Fast, cost-effective (default)
model="gemini-1.5-pro"         # More capable, higher cost
model="gemini-1.5-flash"       # Balanced

Embedding Model

Change Ollama embedding model (requires model pull):

"model": "nomic-embed-text"     # 768 dimensions (default)
"model": "mxbai-embed-large"    # 1024 dimensions

If changing models, delete existing .faiss and .json files and rebuild.

How It Works

1. Document Chunking

Text is split into overlapping chunks:

Original: "The quick brown fox jumps over the lazy dog..."

Chunk 1: "The quick brown fox jumps..." (words 0-60)
Chunk 2: "...fox jumps over the lazy..." (words 50-110, 10-word overlap)
Chunk 3: "...lazy dog runs through..." (words 100-160, 10-word overlap)

2. Embedding Generation

Each chunk is converted to a 768-dimensional vector via Ollama:

response = requests.post(
    "http://localhost:11434/api/embeddings",
    json={"model": "nomic-embed-text", "prompt": chunk}
)
embedding = np.array(response.json()["embedding"])  # Shape: (768,)

3. FAISS Indexing

Embeddings are stored in a FAISS IndexFlatL2 (L2 distance):

dimension = 768
index = faiss.IndexFlatL2(dimension)
index.add(np.stack(all_chunks))  # Add all embeddings

FAISS IndexFlatL2 uses Euclidean distance:

• Closer vectors (smaller L2 distance) = more similar
• Exhaustive search guarantees best matches
• Fast for up to ~100K vectors

4. Semantic Search

Query is embedded and searched against index:

query_embedding = get_embedding(query)  # Shape: (768,)
D, I = index.search(query_embedding.reshape(1, -1), top_k=5)
# D: distances, I: indices of most similar chunks

5. Answer Generation

Retrieved chunks are passed to Gemini as context:

context = "\n".join([metadata[i]["chunk"] for i in I[0]])
prompt = f"Query: {query}\n\nContext: {context}\n\nAnswer:"
response = client.models.generate_content(model="gemini-2.0-flash", contents=prompt)

Example Workflow

Scenario: Building a SAP Documentation RAG System

1. Initial setup with SAP Joule documentation:

# Edit ollamaEmbed.py
file_path = Path("sapfiles/SAP_Joule.txt")

python ollamaEmbed.py
# Output: ✅ Indexed 142 chunks

2. Query the system:

# Edit ragModel.py
query = "What is SAP Joule?"

python ragModel.py
# Output: SAP Joule is an AI assistant that helps users...

3. Add more documentation:

# Edit appendEmbeds.py
new_file_path = "sapfiles/SAP_GENAI.txt"

python appendEmbeds.py
# Output: ✅ Added 87 new chunks to the index

4. Query with expanded knowledge base:

# Query now searches across both documents
python ragModel.py

Advanced Usage

Batch Processing Multiple Documents

Modify ollamaEmbed.py to process a directory:

from pathlib import Path

all_chunks = []
metadata = []

for file_path in Path("docfiles").glob("*.txt"):
    chunks, meta = process_document(file_path)
    all_chunks.extend(chunks)
    metadata.extend(meta)

# Then create index as usual

Custom Similarity Metrics

Replace IndexFlatL2 with other FAISS indices:

# Inner product (for normalized embeddings)
index = faiss.IndexFlatIP(dimension)

# Approximate search (faster for large datasets)
index = faiss.IndexIVFFlat(quantizer, dimension, nlist=100)

Multi-Query RAG

Query multiple times with refined prompts:

queries = [
    "What is SAP Joule?",
    "How does SAP Joule integrate with SAP systems?",
    "What are the key features of SAP Joule?"
]

for query in queries:
    results = search_index(index, metadata, query, top_k=5)
    # Process results...

Performance Optimization

For Large Document Collections (>10K chunks)

1. Use IVFPQ index (Inverted File with Product Quantization):

nlist = 100  # Number of clusters
m = 8        # Number of subquantizers
quantizer = faiss.IndexFlatL2(dimension)
index = faiss.IndexIVFPQ(quantizer, dimension, nlist, m, 8)
index.train(embeddings)  # Training required
index.add(embeddings)

2. Batch embedding generation:

# Process chunks in batches
batch_size = 50
for i in range(0, len(chunks), batch_size):
    batch = chunks[i:i+batch_size]
    # Generate embeddings for batch...

For Faster Queries

1. Reduce top_k: Retrieve fewer chunks (3 instead of 5)
2. Use Gemini Flash: Faster than Pro models
3. Cache embeddings: Store query embeddings for repeated queries

Troubleshooting

Ollama Connection Error

Error: requests.exceptions.ConnectionError: Failed to establish a new connection

Solution:

# Check if Ollama is running
curl http://localhost:11434/api/tags

# If not, start Ollama
ollama serve

Model Not Found

Error: model 'nomic-embed-text' not found

Solution:

ollama pull nomic-embed-text
ollama list  # Verify model is downloaded

FAISS Dimension Mismatch

Error: RuntimeError: Error in void faiss::IndexFlat::add_with_ids()

Solution: Delete existing .faiss and .json files and rebuild:

rm my_faiss_index.faiss my_metadata.json
python ollamaEmbed.py

Gemini API Error

Error: google.api_core.exceptions.PermissionDenied: 403 API key not valid

Solution: Verify your .env file:

cat .env  # Should show: GEMINI_API_KEY=AIza...

Get a new key from Google AI Studio

Memory Issues (Large Documents)

Error: MemoryError: Unable to allocate array

Solution: Process documents in smaller batches or use dimensionality reduction:

# Use PCA to reduce dimensions
import faiss
pca_matrix = faiss.PCAMatrix(768, 256)  # Reduce to 256 dimensions
index = faiss.IndexFlatL2(256)

Technical Details

FAISS IndexFlatL2

• Algorithm: Exhaustive search (brute force)
• Metric: L2 (Euclidean) distance
• Accuracy: 100% (finds exact nearest neighbors)
• Speed: O(N*D) where N=number of vectors, D=dimension
• Best for: Up to 1M vectors with 768 dimensions

Nomic Embeddings

• Model: nomic-embed-text
• Dimensions: 768
• Context Length: 8192 tokens
• Trained on: Billions of text pairs for semantic similarity
• License: Open-source (Apache 2.0)

Chunking Strategy

Why 60 words with 10-word overlap?

• 60 words: ~300-400 characters, fits typical semantic units (paragraphs)
• 10-word overlap: Prevents context loss at boundaries
• Result: Each chunk has some context from neighbors

Trade-offs:

• More overlap → Better context, more storage
• Less overlap → Faster processing, potential context gaps

Comparison with Alternatives

Component	This Project	Alternative
Embeddings	Ollama (local, free)	OpenAI ($0.0001/1K tokens)
Vector DB	FAISS (local, fast)	Pinecone/Weaviate (cloud, $$)
LLM	Gemini Flash (cheap)	GPT-4 (expensive)
Storage	Local files	Cloud databases
Privacy	Embeddings local	Data sent to APIs

Advantages:

• Lower cost (only LLM API calls)
• Privacy (embeddings stay local)
• Fast (local FAISS search)
• No vendor lock-in

Disadvantages:

• Requires Ollama setup
• Local storage needed
• No built-in scaling

Extending the Project

Add Support for PDF Documents

pip install PyPDF2

import PyPDF2

def extract_text_from_pdf(pdf_path):
    with open(pdf_path, 'rb') as file:
        reader = PyPDF2.PdfReader(file)
        text = ""
        for page in reader.pages:
            text += page.extract_text()
    return text

Add Conversational Memory

Store conversation history and include in context:

conversation_history = []
conversation_history.append({"role": "user", "content": query})
conversation_history.append({"role": "assistant", "content": response.text})

Add Web UI with Streamlit

pip install streamlit

import streamlit as st

st.title("RAG Document Q&A")
query = st.text_input("Ask a question:")
if st.button("Search"):
    results = search_index(index, metadata, query, top_k=5)
    answer = generate_answer(results, query)
    st.write(answer)

Deploy as MCP Server

Integrate with the MCP (Model Context Protocol) pattern:

• Expose RAG search as an MCP tool
• Add resources for document management
• See ../agenticMCP_rag/ for reference implementation

References

• FAISS Documentation
• Ollama Documentation
• Nomic Embeddings
• Gemini API
• RAG Paper (Lewis et al., 2020)

License

Part of the experimental AI/ML monorepo. See root repository for license details.

Related Projects

For more advanced RAG implementations, see:

• ../agenticMCP_rag/ - RAG with MCP integration and agent architecture
• ../R_RAG/ - Alternative RAG implementation
• ../perceptionAgent/ - Agent architecture with perception-decision-action pattern