How I Built a Production RAG System with Multi-Query Expansion and Re-Ranking

Building a RAG system for a demo is straightforward. Building one that handles real queries from real users — that's a different problem entirely.

At Emulxion.ai, I was tasked with building a production RAG system that could handle complex, multi-part queries in real time. Here's what I learned.

The problem with naive RAG

A basic RAG pipeline — embed the query, retrieve top-k chunks, pass to LLM — works surprisingly well in demos. But in production, users don't ask clean single-intent questions. They ask things like: "What are the side effects of X and how does it compare to Y when combined with Z?"

A single embedding can't capture all of that.

Multi-query expansion

The first fix: don't just embed the original query. Use an LLM to generate 3-5 alternative phrasings of the same question, retrieve for each, then merge the results.

def expand_query(query: str, llm) -> list[str]:
    prompt = f"""Generate 4 alternative phrasings of this query for document retrieval.
Query: {query}
Return as a JSON array of strings."""
    result = llm.invoke(prompt)
    return json.loads(result.content)

RAG Fusion with Reciprocal Rank Fusion

Once you have multiple result sets, you need to merge them intelligently. Reciprocal Rank Fusion (RRF) works well — it rewards documents that rank highly across multiple queries.

def reciprocal_rank_fusion(results: list[list], k=60):
    scores = {}
    for result_list in results:
        for rank, doc in enumerate(result_list):
            doc_id = doc.metadata["id"]
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (rank + k)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Re-ranking

After fusion, you still have 20-30 candidate chunks. A cross-encoder re-ranker (Cohere Rerank or a local model) scores each chunk against the original query and picks the top 5.

This was the single biggest quality improvement in the system.

Results

After implementing all three layers, retrieval accuracy improved significantly on our internal benchmark. The system now handles ambiguous and multi-part queries that completely stumped the naive version.

The key insight: retrieval is not a solved problem. The more effort you put into the retrieval layer, the better your LLM responses — regardless of which LLM you use.