<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Timothé Pearce on Medium]]></title>
        <description><![CDATA[Stories by Timothé Pearce on Medium]]></description>
        <link>https://medium.com/@tim_pearce?source=rss-3381bad13477------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2nTOPaeVLEEsiANhXiYr4g.jpeg</url>
            <title>Stories by Timothé Pearce on Medium</title>
            <link>https://medium.com/@tim_pearce?source=rss-3381bad13477------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:53:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tim_pearce/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a RAG Router in 2025]]></title>
            <link>https://medium.com/@tim_pearce/building-a-rag-router-in-2025-e0e9d99efe44?source=rss-3381bad13477------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0e9d99efe44</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Timothé Pearce]]></dc:creator>
            <pubDate>Fri, 10 Jan 2025 13:51:59 GMT</pubDate>
            <atom:updated>2025-01-10T14:55:39.993Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Through the towering maze of production deployment, RAG routing weaves paths of order and clarity, turning chaos into orchestrated ascension.</em></p><figure><img alt="The Tower of Babel — Pieter Bruegel the Elder" src="https://cdn-images-1.medium.com/max/1024/1*vI6y4yaQyXLniMmu5UeV-w.png" /><figcaption>The Tower of Babel — Pieter Bruegel the Elder</figcaption></figure><p>Retrieval-Augmented Generation (RAG) has become a cornerstone of modern AI applications, enabling companies to build powerful question-answering systems on top of their data.</p><p>However, as these systems move to production, a major challenge emerges: chatbots need to handle questions across a wide range of topics, this broad scope makes it challenging to maintain high-quality responses.</p><p>Traditional RAG approaches, with their one-size-fits-all strategies, often struggle to perform consistently across such diverse queries.</p><p>This is where RAG routing comes into play. A RAG router (aka. « semantic router ») acts as an intelligent traffic controller, directing queries to specialized processing pipelines, enabling you to optimize every steps of your RAG system.</p><p>In this article, we’ll explore how to design and train an effective RAG router that can:</p><ul><li><strong>Optimize retrieval strategies</strong> using automatic query analysis to select the best document sources and parameters</li><li><strong>Enhance answer quality</strong> with specialized prompts and pipelines for different query types</li><li><strong>Reduce operational costs and inference time</strong> by matching query complexity with appropriate model size</li><li><strong>Implement safe deployment strategies</strong> with existing system fallback for safe transition</li></ul><p>Let’s dive in.</p><h3>Understanding RAG Routing</h3><p>A RAG router applies the fundamental software engineering principle of “divide and conquer”.</p><p>By analyzing the semantic field of user queries, it classifies user intent into specific domains, each handled by specialized components designed for optimal performance.</p><p>For « pre-sales » queries, it might focus on product specifications and competitive advantages, while « post-sales » queries could trigger a different retrieval strategy focusing on troubleshooting guides and support documentation.</p><p>This classification allows you to adapt the entire pipeline — from retrieval parameters to prompt engineering and model selection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iwqTzQErq6ASau-8MkVQcw.png" /><figcaption>A simple RAG router handling two topics</figcaption></figure><p>The value of a router becomes even more apparent when scaling to production environments, where RAG systems typically need to handle dozens of topics.</p><p>As your knowledge base grows to address multiple domains, the router’s ability to direct queries to specialized processing pipelines becomes crucial for maintaining system performance.</p><p>More importantly, when certain topics require specialized handling — whether it’s entity extraction through dedicated models or graceful fallback to human agents for sensitive matters — a router ensures each query receives the most appropriate processing strategy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uc3QI-Dge9xlETpH7mQbow.png" /><figcaption>A more complex router with custom RAG components by topic</figcaption></figure><h3>Designing Your RAG Router</h3><h4>Identifying Router Labels</h4><p>The first step in designing your router is identifying the topics your RAG system needs to address. Unlike traditional chatbots that rely on numerous specific intents, a RAG router should focus on broader, more generalist topics to maintain a manageable number of processing pipelines.</p><p>From production experience, successful RAG routers typically handle between 5 to 20 topics at most.</p><p>Finding the right balance is crucial. While more topics might seem to offer better specialization, they also increase the risk of overlap between topics.</p><p>Your goal should be to find the sweet spot between specificity and maintainability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0lUjdpp9U5FvIs9Ikk6o8A.png" /><figcaption>Topics with overlapping queries</figcaption></figure><p>For example, in an e-commerce context, instead of creating specific categories like product_price_query, product_availability_query, and product_specifications_query, you might want to group these under a broader product_information macro topic.</p><p>This consolidation ensures that your RAG system has access to a comprehensive set of product-related documents during retrieval, allowing it to answer various product queries accurately while maintaining a single, well-optimized pipeline.</p><p>This approach is particularly effective because product-related questions often require overlapping context — a question about price might also benefit from availability information, while a specification query might need pricing details for different variants.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oUDrq7kzyTVm-xFmxDIfnw.png" /><figcaption>Macro topics with clear boundaries</figcaption></figure><p>Identifying macro topics is straightforward when you have historical chatbot data to analyze. If not, start with a combination of intuition and business domain knowledge then refine your topics once your system collects real user interactions.</p><h4>Choose the Right architecture</h4><p>When developing your router, several architectural approaches exist, each with its own trade-offs. Let’s examine each approach to help you choose the most practical solution for your needs.</p><p>❌ <strong>Rule-based Approach</strong></p><p>While keyword matching is the simplest approach to implement, it performs poorly in production. Users employ varied language, make typos, use acronyms of synonyms, making rule maintenance challenging and intent detection unreliable. This approach should only be considered for proof-of-concept projects and is not recommended for production systems.</p><p>If you still want to experiment with this approach, consider using <a href="https://spacy.io/">SpaCy</a>, which provides an easy-to-use rule engine for text pattern matching along with many other NLP features.</p><p>❌ <strong>LLM-based Approach</strong></p><p>Using a Large Language Model for topic classification can be effective with limited topics but comes with significant drawbacks in production environments:</p><ul><li>Slow inference time (seconds)</li><li>High operational costs at scale</li><li>Poor accuracy on domain-specific topics</li><li>Complex prompt engineering required</li></ul><p>For those still interested in exploring this approach, the <a href="https://github.com/aurelio-labs/semantic-router">Semantic Router</a> package provides a good starting point, though keep its limitations in mind for production use.</p><p>✅ <strong>Model fine-tuning Approach</strong></p><p>The most efficient approach is fine-tuning your own specialized classification model. Pre-trained embedding models, which represent text in high-dimensional space, can be adapted to your needs by simply adding a logistic regression layer on top of the embedding output. This fine-tuning leverages the model’s pre-learned language understanding while optimizing it for your specific domain.<br>The benefits are clear:</p><ul><li>Fast inference times (milliseconds rather than seconds)</li><li>Lower operational costs</li><li>Better accuracy on domain-specific topics</li></ul><p>The most effective architecture is based on a pre-trained transformer encoder processing the input text token by token, with a special [CLS] token that captures the overall meaning of the sentence. The model’s hidden state corresponding to this [CLS] token is then fed into a trained logistic regression layer that classifies the query into predefined topics (like “return_refund”).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*24_1ezyLJrMHrZmWvZ3Nfg.png" /><figcaption>A text-classification model based on an encoder and a logistic regression layer</figcaption></figure><p>To implement this architecture, several proven solutions exist. I recommend starting with these two NLP libraries:</p><ul><li><a href="https://huggingface.co/docs/setfit/index"><strong>SetFit</strong></a>: Perfect for quick starts with limited training data</li><li><a href="https://huggingface.co/docs/transformers/index"><strong>Transformers</strong></a>: The most popular library for fine-tuning pre-trained models leveraging the Hugging Face ecosystem</li></ul><p>While building a dataset and training a model might seem daunting, transfer learning and modern NLP frameworks has made this process accessible with minimal data needed. Let’s explore how to implement this with Transformers, with code you can easily adapt to your needs.</p><h3>Training Your RAG Router</h3><p>In this section, we’ll walk through the process of training a RAG router using the Transformers library. While the code can run on CPU, we recommend using a GPU for faster training.</p><h4>Data collection and preparation</h4><p>For this step, we’ll use the <a href="https://huggingface.co/datasets/timpearce/banking10">Banking10 dataset</a>, a simplified version of <a href="https://huggingface.co/datasets/PolyAI/banking77">Banking77</a> I built, it contains customer service queries classified into 10 macro topics.</p><p>Let’s start by installing the required packages:</p><pre>pip install transformers datasets evaluate</pre><p>First, we’ll import the necessary libraries and load our dataset, using the widely adopted BERT base uncased model for its lightweight yet effective performance:</p><pre>import evaluate<br>import numpy as np<br>from datasets import load_dataset<br>from transformers import (<br>    AutoModelForSequenceClassification, <br>    AutoTokenizer, <br>    DataCollatorWithPadding, <br>    Trainer, <br>    TrainingArguments<br>)<br><br>pre_trained_model_name = &quot;google-bert/bert-base-uncased&quot;<br>dataset = load_dataset(&quot;timpearce/banking10&quot;)</pre><p>Next, we need to prepare our labels. Instead of manually mapping each label to an ID, we can automatically create our label mappings from the dataset:</p><pre>unique_labels = sorted(set(dataset[&quot;train&quot;][&quot;label&quot;]))<br><br>id2label = {id: label for id, label in enumerate(unique_labels)}<br>label2id = {label: id for id, label in id2label.items()}</pre><p>Now, we’ll tokenize our data. The tokenizer converts text inputs into a format that our model can understand:</p><pre>tokenizer = AutoTokenizer.from_pretrained(pre_trained_model_name)<br><br>def preprocess_dataset(examples):<br>    tokenized = tokenizer(examples[&quot;text&quot;], truncation=True)<br>    tokenized[&quot;label&quot;] = [label2id[label] for label in examples[&quot;label&quot;]]<br>    return tokenized<br><br>tokenized_dataset = dataset.map(preprocess_dataset, batched=True)<br>data_collator = DataCollatorWithPadding(tokenizer=tokenizer)</pre><p>This preprocessing pipeline:</p><ol><li>Loads a pre-trained BERT tokenizer</li><li>Converts text to token IDs</li><li>Maps text labels to numeric IDs</li><li>Prepares the data for batch processing</li></ol><h4>Fine-tuning the pre-trained model</h4><p>Now that our data is properly prepared and tokenized, we’re ready to move on to the model training phase. First, let’s define our evaluation metric. We’ll use accuracy to measure how well our model classifies queries:</p><pre>accuracy = evaluate.load(&quot;accuracy&quot;)<br><br>def compute_metrics(eval_pred):<br>    predictions, labels = eval_pred<br>    predictions = np.argmax(predictions, axis=1)<br>    return accuracy.compute(predictions=predictions, references=labels)</pre><p>Next, we’ll initialize our classification model. We’ll use our pre-trained BERT model and add a classification head on top:</p><pre>model = AutoModelForSequenceClassification.from_pretrained(<br>    pre_trained_model_name,<br>    num_labels=len(unique_labels),<br>    id2label=id2label,<br>    label2id=label2id<br>)</pre><p>Now, let’s configure our training parameters. While these hyperparameters should be tuned for your specific task, they represent a solid starting point:</p><pre>training_args = TrainingArguments(<br>    output_dir=&quot;semantic_router&quot;,<br>    learning_rate=2e-5,              # Small learning rate for fine-tuning<br>    per_device_train_batch_size=16,  # Adjust based on your GPU memory<br>    per_device_eval_batch_size=16,<br>    num_train_epochs=2,              # Usually 2-3 epochs are sufficient<br>    weight_decay=0.01,               # Help prevent overfitting<br>    eval_strategy=&quot;epoch&quot;,           # Evaluate after each epoch<br>    save_strategy=&quot;epoch&quot;,           # Save model after each epoch<br>    load_best_model_at_end=True,     # Keep the best performing model<br>)</pre><p>Finally, we’ll create our trainer and start the fine-tuning process:</p><pre>trainer = Trainer(<br>    model=model,<br>    args=training_args,<br>    train_dataset=tokenized_dataset[&quot;train&quot;],<br>    eval_dataset=tokenized_dataset[&quot;test&quot;],<br>    tokenizer=tokenizer,<br>    data_collator=data_collator,<br>    compute_metrics=compute_metrics,<br>)<br><br>trainer.train()</pre><p>The training process will automatically:</p><ul><li>Fine-tune the pre-trained model on our dataset</li><li>Evaluate performance on the test set after each epoch</li><li>Save the best performing model</li><li>Track and report training metrics</li></ul><p>After training for approximately 2 minutes, we obtain the following results:</p><pre>Epoch    Training Loss    Validation Loss    Accuracy<br>1        No log           0.309061           0.924026<br>2        0.715300         0.219540           0.939610</pre><p>With just two epochs of training, our model achieves an impressive accuracy of approximately 94% on the test set.</p><p>This demonstrates how transfer learning allows us to quickly build an effective topic classifier with minimal training time.</p><h4>Analyzing Model Performance with Confusion Matrix</h4><p>To better understand our model’s performance and identify potential areas of improvement, especially where topics might overlap, we can visualize its predictions using a confusion matrix. Here’s how to create one:</p><pre>import numpy as np<br>from sklearn.metrics import confusion_matrix<br>import seaborn as sns<br>import matplotlib.pyplot as plt<br><br>def plot_confusion_matrix(trainer, dataset):    <br>    predictions = trainer.predict(dataset)<br>    predicted_classes = np.argmax(predictions.predictions, axis=1)<br>    true_classes = predictions.label_ids<br>    <br>    plt.figure(figsize=(10, 8))<br>    <br>    sns.heatmap(<br>        confusion_matrix(true_classes, predicted_classes),<br>        annot=True,<br>        fmt=&#39;d&#39;,<br>        cmap=&#39;Blues&#39;,<br>        xticklabels=list(trainer.model.config.id2label.values()),<br>        yticklabels=list(trainer.model.config.id2label.values()),<br>        square=True<br>    )<br>    <br>    plt.xticks(rotation=45, ha=&#39;right&#39;, rotation_mode=&#39;anchor&#39;)<br>    plt.yticks(rotation=45, va=&#39;top&#39;)<br>    <br>    plt.title(&#39;Confusion Matrix&#39;, pad=15)<br>    plt.xlabel(&#39;Predictions&#39;, labelpad=15)<br>    plt.ylabel(&#39;Ground Truth&#39;, labelpad=15)<br>    <br>    plt.tight_layout(pad=1.2)<br>    <br>    return plt<br><br>confusion_plot = plot_confusion_matrix(trainer, tokenized_dataset[&quot;test&quot;])<br>confusion_plot.show()</pre><p>Let’s analyze the resulting confusion matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*QeerkliGKG1YCaDJ5ZEKAQ.png" /><figcaption>Confusion Matrix of the fine-tuned model</figcaption></figure><p>Looking at the confusion matrix, our model demonstrates strong overall performance with most predictions correctly aligned on the diagonal.</p><h4>Using Your Model</h4><p>Here’s how to use your trained model to classify new queries:</p><pre>def classify_query(text, model, tokenizer):<br>    inputs = tokenizer(text, return_tensors=&quot;pt&quot;, truncation=True)<br>    inputs = {k: v.to(model.device) for k, v in inputs.items()}<br>    <br>    outputs = model(**inputs)<br>    predictions = outputs.logits.softmax(dim=-1)<br>    predicted_label_id = predictions.argmax().item()<br>    <br>    return {<br>        &quot;label&quot;: id2label[predicted_label_id],<br>        &quot;confidence&quot;: predictions[0][predicted_label_id].item()<br>    }<br><br>queries = [<br>    &quot;How do I reset my PIN number?&quot;,<br>    &quot;I want to transfer money to my friend&quot;,<br>    &quot;My card was declined&quot;<br>]<br><br>for query in queries:<br>    result = classify_query(query, model, tokenizer)<br>    print(f&quot;Query: {query}&quot;)<br>    print(f&quot;Predicted topic: {result[&#39;label&#39;]}&quot;)<br>    print(f&quot;Confidence: {result[&#39;confidence&#39;]:.2%}&quot;)<br>    print(&quot;---&quot;)</pre><p>Running this code produces:</p><pre>Query: How do I reset my PIN number?<br>Predicted topic: ACCOUNT_VERIFICATION<br>Confidence: 94.26%<br>---<br>Query: I want to transfer money to my friend<br>Predicted topic: TRANSFERS_AND_PAYMENTS<br>Confidence: 94.66%<br>---<br>Query: My card was declined<br>Predicted topic: CARD_TRANSACTIONS<br>Confidence: 76.61%<br>---</pre><p>The model successfully identifies query topics with high confidence. Note the lower confidence (76.61%) for the card decline query, which might indicate some overlap between card-related topics.</p><h3>Conclusion</h3><p>In this guide, we’ve explored how to build an effective RAG router from design to implementation. The demonstrated approach using fine-tuned models offers a robust solution for production environments.</p><p>Looking ahead, your RAG system can evolve beyond basic routing. Consider customizing prompts based on detected topics, filtering retrieved documents, or even fine-tuning embedding models for your specific domain.</p><p>Remember that building a successful RAG router is an iterative process — start with broader topics, monitor real usage patterns, and continuously refine your approach based on actual user interactions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0e9d99efe44" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>