Beyond RAG: Building an Intelligent Shipment Query System with Tool Calling

The Problem That Started It All

Picture this: You’re building a customer service chatbot for shipment queries. Your first instinct? Implement RAG (Retrieval-Augmented Generation). After all, it’s the go-to solution for grounding LLMs in specific data, right?

I thought the same thing when I started building a shipment query system using Llama 3.1 via Ollama. But I quickly hit a wall that made me rethink everything.

Where RAG Falls Short

RAG excels at retrieving and presenting information. Need the status of shipment #12345? RAG handles that beautifully. But here’s where things get tricky:

“What’s the average invoice amount across all my shipments?”

“Show me the total value of shipments to New York in the last quarter.”

“How many packages are currently in transit to international destinations?”

These aren’t retrieval problems, they’re computation problems. You can’t just pull a document chunk and expect the answer. You need to:

• Query databases with specific filters
• Perform aggregations and calculations
• Execute business logic
• Return structured, accurate results

RAG alone can’t do this reliably. The model might hallucinate numbers, misinterpret date ranges, or provide outdated information.

The Solution: Tool Calling

Instead of feeding raw data chunks to the LLM, I gave it something more powerful specialized tools that it could intelligently call based on user queries.

Think of it like this: Rather than giving someone a massive spreadsheet and asking them to find answers, you give them a set of purpose-built calculators and query functions.

The Architecture

Here’s what the system looks like:

1. LLM Layer — Llama 3.1 (via Ollama) handles intent analysis and orchestration
2. Tool Selection — Model determines which tools to invoke based on the query
3. Specialized Tools:

• GetShipmentAnalysisTool- Trend analysis and pattern detection
• GetShipmentsByMonthTool - Monthly aggregation and reporting
• GetShipmentsByDestination - Location-based queries
• GetShipmentByTrackingNum - Individual package tracking

4. Shipment Database — Single source of truth for all operations

5. Recursive Calling — Handles complex queries requiring multiple tools

The beauty of this approach? The model decides which tools to call, in what order, and how to combine their outputs into a coherent response.

Implementation with OllamaSharp

I chose to build this in .NET using OllamaSharp, which made the implementation surprisingly straightforward. Here’s why:

1. Local Execution

Everything runs locally. No API costs, no data privacy concerns, and complete control over the model behavior.

2. Type-Safe Tool Definitions

With C# and OllamaSharp, defining tools is clean and type-safe:

public static class LlamaTools
{
    /// <summary>
    /// Calculates the average weight of a list of shipments.
    /// </summary>
    /// <param name="shipments">The list of shipments to analyze.</param>
    /// <returns>The average weight of all shipments in kilograms.</returns>
    [OllamaTool]
    [Description("Calculates the average weight of a list of shipments.")]
    public static double AverageWeight(List<Shipment> shipments)
    {
        if (shipments == null || !shipments.Any())
        {
            return 0;
        }
      
        return shipments.Average(s => s.Shipment_Weight ?? 0);
    }
}

3. Automatic Tool Selection

The model analyzes the user’s intent and calls the appropriate tool:

public class ChatService
{
    private readonly IConfiguration _configuration;
    private readonly OllamaApiClient _ollama;

    private readonly string _model;
    private readonly string _baseUrl;

    public ChatService(IConfiguration configuration)
    {
        _configuration = configuration;
        _baseUrl = _configuration.GetValue<string>("Ollama:BaseUrl") ?? "http://localhost:11434";
        _model = _configuration.GetValue<string>("Ollama:Model") ?? "llama3:8b";

        _ollama = new OllamaApiClient(_baseUrl, _model);
    }

    public static string prompt = "";

    public async Task<IAsyncEnumerable<string>> ChatAsync(CustomerRequest customerRequest)
    {
        Chat? chat = null;

        try
        {
            List<Shipment> shipments = await _context.Shipments
                .Where(x => x.CustomerId == customerRequest.CustomerId)
                .OrderByDescending(x => x.InvoiceDate)
                .ToListAsync();

            chat = new Chat(_ollama)
            {
                Messages = [
                    new (ChatRole.System, prompt)
                ],
                AllowRecursiveToolCalls = true,
                Model = _model
            };

            var llamaTools = new List<object>
            {
                new GetShipmentAnalysisTool(),
                new AverageInvoiceAmountTool(),
                new AverageWeightTool()
            };


            chat.OnToolCall += (sender, toolCall) =>
            {
                toolCall.Function!.Arguments!["shipments"] = shipments;
            };

            return chat.SendAsync(customerRequest.Query, llamaTools);
        }
        catch (Exception ex)
        {
            return null!;
        }
    }
}

4. Recursive Tool Calling

Complex queries often need multiple operations:

Get Syed Shahriyar Ali’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

User: “Compare my shipment volumes between New York and Los Angeles in Q3, and tell me which destination had higher average invoice amounts.”

The model:

1. Calls GetShipmentsByDestination for New York
2. Calls GetShipmentsByDestination for Los Angeles
3. Calls GetShipmentAnalysisTool to compute averages
4. Synthesizes the results into a natural language response

All of this happens automatically.

Real-World Benefits

After implementing this approach, the improvements were dramatic:

1. Zero Hallucinations

Because the model isn’t generating numbers it’s calling functions that return actual data hallucinations are eliminated. The math is always correct.

2. Consistent Accuracy

Whether a user asks “show me last month’s shipments” or “what did I ship in November 2024,” the tool translates the intent into precise database queries.

3. Complex Query Handling

Users can ask multi-part questions, and the system breaks them down into logical tool calls without explicit programming for each scenario.

4. Natural Conversation Flow

The responses feel conversational because the LLM handles the language generation, while tools handle the facts.

A Lightweight MCP Alternative

If you’re familiar with Model Context Protocol (MCP), you might recognize similarities. This is essentially a private, internal MCP implementation:

• Tools are registered with the model
• The model calls tools based on context
• Results are structured and reliable
• The system is extensible

The key difference? This isn’t exposed externally it’s a self-contained system designed for a specific domain.

When Should You Use Tool Calling Over RAG?

Tool calling shines when you need:

• Computations: Averages, sums, counts, trends
• Structured queries: Database operations with filters and joins
• Real-time data: Current inventory, tracking status, availability
• Business logic: Rules, validations, calculations
• Multi-step operations: Workflows requiring multiple data sources

RAG is still excellent for:

• Document retrieval: Policy documents, FAQs, guides
• Semantic search: Finding relevant information across large text corpora
• Context injection: Providing background information to the model

The best systems often use both RAG for knowledge retrieval and tool calling for operations.

Key Takeaways

1. RAG isn’t always enough — When you need computations, not just retrieval, tool calling is your answer
2. Local LLMs are production-ready — Llama 3.1 via Ollama handles complex intent detection reliably
3. OllamaSharp simplifies implementation — Clean APIs make tool integration straightforward in .NET
4. Let the model orchestrate — Trust the LLM to call tools intelligently rather than hardcoding logic
5. Type safety matters — Strongly-typed languages like C# prevent many runtime errors in tool implementations

If you’re building LLM applications that need to interact with structured data or perform operations beyond simple retrieval, tool calling might be the missing piece in your architecture.