All three of us are engineers, so we did what engineers do: we documented everything, tried a few CRMs, and eventually decided to build an agent to handle our commercial pipeline.

The job we were automating

Selling craft beer to bars and restaurants is a grind of small, sequential steps:

1. Find candidate venues in strategic parts of the city
2. Verify each one is a real prospect a place that sells beer, not a brewery that makes it (that’s a competitor)
3. Check it isn’t already in our leads database, or worse, already an active client
4. Research the venue enough to write something that isn’t a template
5. Reach out via Instagram DM, WhatsApp, or email
6. Log everything and, eventually, book a meeting

Attempt 1: the cheapest model that benchmarked well

We’re a young company. Most of our capital goes to production and point-of-sale material, not to an AI budget. So we started with the cheapest model that looked good on benchmarks.

It was fine at reasoning about a lead and drafting a message. It was unreliable at using tools. The failure mode wasn’t refusal, it was worse. It would announce the action and never take it:

“Perfect. I’m going to write the handle in the search bar now”

And then nothing. No tool call at all. Four times in a row, on the same lead. In a browser-driven task with a dozen fragile steps, “almost” is indistinguishable from “no.”

Attempt 2: frontier models

We moved to stronger models Kimi K3, then Grok 4.5. The difference was immediate. They completed the full loop at a natural pace, produced genuinely personalized messages, and handled several leads a day.

Then we looked at the bill.

Two real days from our logs, same agent, same job:

Eleven times the cost, for roughly half the calls.

The unit economics explain it. Compared to what we run today, the frontier model charged ~5× more per input token, ~8× more per output token, and the one that actually hur t ~14× more per cached input token.

That last number is the trap. Our agent’s system prompt is large: operating manual, tool documentation, business rules, memory. It gets re-read on every single turn. On the expensive model, that cache re-read alone was ~$1.60/day. Same tokens, on our current setup: about twelve cents.

We also found we were paying for nothing. Of those 177 daily calls, 48 were idle heartbeats the agent waking up every 30 minutes to check whether anything needed attention, thinking at full depth to conclude “no.” That was 27% of the day’s spend on doing literally nothing.

The idea: use the expensive model as a teacher, not as the operator

We initially used an expensive frontier model because it was much better at completing our browser-based sales workflow. But it cost roughly 10× more than a cheaper model.

Instead of treating the expensive model as the final solution, we realized it could be used as an exploration and data-collection tool.

What are traces?

A trace is the complete record of an agent’s execution: what it tried to do, which tools it called, what happened, where it failed, and how it recovered. By reviewing these traces, we could see the actual procedure required to complete the task not just the final answer.

The frontier model tried to complete the job, made mistakes, recovered from failures, and discovered what actually worked in our environment. All of this was captured in its execution traces.

We reviewed those traces and turned what we learned into three things:

1. Deterministic tools: Successful browser workflows became reusable tools instead of something the model had to recreate every time.
2. Playbooks: Important lessons and constraints became explicit instructions and documentation.
3. Verification gates : Actions only count as successful when there is explicit proof that they actually happened.

This changed the cheap model’s job. It no longer needs to figure out how to perform the task. It only needs to follow a known procedure and verify each step.

We didn’t fine-tune or retrain the cheap model. The knowledge was moved from the model into tools, documentation, and verification.

The result: we could run the same workflow with a much cheaper model while maintaining reliable execution.

The key takeaway

Don’t think of an expensive model as your permanent operator. Use it to discover the procedure, then turn what it learned into tools, rules, and verification. Once the process is deterministic and checkable, a much cheaper model can execute it.

I’ve been living in that gap for years. As a data scientist working with startups and growing companies, I kept finding myself acting as the bridge between business questions and data reality. That experience is what pushed me to start building an analytics agent: something that could help anyone in an organization access information without needing to become a SQL expert, a database archaeologist, or a BI power user.

At first, the goal was simple. Help people turn questions into SQL. Over time, the goal evolved into something much bigger.

Getting to “Good Enough” at SQL

Through multiple iterations : moving from RAG to agentic workflows, introducing memory, adding human-in-the-loop clarification, and building proper evaluation frameworks. The agent reached an important milestone. Thanks to feedback from real users and systematic evals, we arrived at the same conclusion from different angles: the agent was finally good enough at understanding business questions and reliably generating SQL that answered them.

That moment was both satisfying and uncomfortable. Satisfying because it validated a lot of engineering decisions. Uncomfortable because it immediately exposed the next limitation.

Once users trusted the SQL, they started asking a different class of questions.

-- Agents SQL
sql
SELECT count(*) AS total_orders,
       count(DISTINCT date_trunc('week', orderdate)) AS total_weeks,
       ROUND( COUNT(*)::numeric / NULLIF(count(DISTINCT date_trunc('week', orderdate)),0),2) AS avg_orders_per_week
FROM public.orders
WHERE orderdate IS NOT NULL;

They didn’t just want queries. They wanted to see patterns. They wanted visualizations. They wanted reports they could share with their teams. And they wanted all of this to happen in the tools they already used every day.

The Shift from Answers to Insights

Two problems became obvious very quickly.

First, a correct SQL query is rarely the final artifact. Most decisions are made by looking at charts, trends, comparisons, and cohorts. A table of numbers is useful, but a well-designed visualization is what actually drives understanding.

Second, an agent that lives in isolation does not scale inside a company. If insights stay trapped inside a one-on-one chat interface, they never become part of the organization’s shared knowledge. Real collaboration happens in places like Slack, not inside a private agent session.

This is where the problem stopped being about text-to-SQL and became about how insights are created, refined, and shared.

When a Single Agent Was No Longer Enough

For a long time, a single agent with good tools and careful context management worked surprisingly well. It could explore schemas, test assumptions, fix SQL errors, and generate accurate queries. For the task of translating business questions into SQL, this approach scaled further than I initially expected.

But once visualization and reporting entered the picture, cracks started to show.

Designing visualizations is not just an extension of query generation. It introduces a different kind of reasoning. The agent now has to think in terms of aggregations, trends, comparisons, and cohorts. It has to validate whether a chart actually makes sense, not just whether the SQL runs. And on top of that, it must explain results clearly and distribute them inside collaboration tools like Slack.

Trying to make a single agent handle all of this quickly became problematic. Not because the model was incapable, but because the responsibilities were fundamentally different. The agent was constantly juggling database reasoning, visualization logic, and communication concerns, all while fighting an ever-growing context window. The more capable it became, the more overloaded its context grew.

This is where context engineering stopped being an abstract concept and became a practical necessity.

While researching long-horizon agent systems, I came across work from Anthropic on effective context engineering. One idea immediately resonated with what I was seeing in practice: complex agent systems tend to fail not because models lack intelligence, but because context becomes overloaded, misaligned, or poorly shared across tasks.

That insight reframed the problem. The issue wasn’t that the agent needed to be smarter. It needed to be more focused.

From One Overloaded Agent to Focused Collaboration

This realization led directly to a multi-agent design, but specialization introduced a new challenge: collaboration. The query agent needs to explain why a query was written a certain way so the visualization agent can adapt it safely. The visualization agent must surface failures or inconsistencies so assumptions can be corrected. At the same time, a communication agent needs to extract business intent from Slack conversations and present queries and visual results in a way humans can trust and act on.

To make this work, we designed a shared memory layer where context lives in one place but is selectively accessed by each agent. Instead of passing raw conversation history around, agents exchange distilled knowledge — queries, field summaries, visualization specs, and reasoning steps. This allows information to flow across agents without overwhelming any single one of them, turning collaboration into a structured process and context into a shared asset rather than a bottleneck.

Solution

The end result is a platform where Metagent connects directly to Metabase and Slack to turn business questions into shared insights, end to end. The system now generates queries, builds visualizations using Metabase’s native tools for filtering, aggregation, and chart definition, and publishes those results directly into Slack channels and group conversations. So insights show up where teams already work and discuss decisions.

If you’d like to try it, you can join for free at https://metagent.app/ , and if you’re interested in following the journey or diving deeper into the architecture, feel free to connect with me on X or LinkedIn.

Here’s a snapshot of where we started, what we’ve learned, and how the Metagent is evolving.

Why Metabase?

I selected Metabase for several pragmatic reasons:

• It offers a comprehensive metadata API (tables, columns, relationships, data catalogue) which helps in building an assistant that “knows” the schema.
• Because it supports self-hosting, users retain data security and control rather than shipping the data to any SaaS.
• The UI is already familiar for many analysts/BI teams, so we can layer intelligence rather than re-invent the entire UI experience.

In short: we’re not replacing Metabase, we’re augmenting it.

The First Version: RAG (Retrieval-Augmented-Generation)

The earliest iteration focused on a simple architecture:

• Metadata (schema, table descriptions) is extracted from Metabase.
• A vector store (e.g., Pinecone) stores indexed documentation for retrieval.
• A large-language model takes the user’s natural-language question + retrieved context and generates SQL.

That got us off the ground: users could type “Show total sales by region” (for example) and get an SQL query which could then be executed in Metabase to render a chart. But we quickly hit limitations:

• The generated SQL was often correct technically, but missed business nuance (e.g., “approved” status vs “confirmed” status) or real-world data quirks.
• There was little exploration — the agent didn’t run test queries or inspect distinct values to validate its assumptions.
• Performance and context-window issues surfaced quickly as schemas grew.

From RAG to Agentic: Tool-Based Workflow

To address those limitations I shifted to an agentic architecture where the assistant has tools it can call (just like an analyst):

• Database Overview Tool: fetch schema, tables, relationships.
• Relationship Explorer: dig into table relationships and join paths.
• Table Inspector: show field types, domain values, sample rows.
• Query Tester: run exploratory SQL (read-only) to inspect actual data (e.g., SELECT DISTINCT status FROM invoices) before committing to a final query.

The agent’s thought process became more like:

1. Understand the business question
2. Explore relevant tables & relationships
3. Run small queries to probe the data
4. Construct and refine the final SQL

This led to better results: more accurate SQL, more aligned with real business logic, and fewer surprises when the query ran on actual data.

But as we onboarded larger clients (hundreds of tables, many schemas) new challenges surfaced.

Scaling Up: Memory, Human-in-the-Loop, and Large Databases

As the complexity and size of client databases increased, the agentic workflow by itself started to struggle:

• Massive schemas meant enormous context windows; the agent could lose focus on what the user really asked.
• Business logic ambiguity (what “revenue” means, what “active user” means) became a blocker. The agent could guess but incorrect guesses erode trust.

To overcome this, I implemented two major innovations:

Memory Management

• Short-term memory: holds the current context for one interaction (recent findings, test query results) so the agent doesn’t re explore everything from scratch.
• Long-term memory: stores persistent insights across queries (e.g., “the status field uses values ‘active’, ‘inactive’, ‘pending’”; “table X join table Y on key Z”) so future queries can reuse knowledge.

This dramatically improved performance in large schema environments: fewer redundant explorations, faster query generation, better accuracy.

Human-in-the-Loop Clarification

Rather than forcing the agent to guess ambiguous business logic, we built a conversational interface where the agent can ask clarifying questions:

“When you say ‘active users’, do you mean users with status = ‘active’ in table A, or those with last_login > 30d in table B?”

This ensures the final SQL aligns with how the business defines key metrics not just what the database happens to call them.

Performance & Metrics

In the latest version (as of October 2025), some results have been promising:

• Accuracy improvement: for one benchmark we increased the success rate (first-try correct SQL) by ~50 %. Read more.
• Response time: reduced from ~65 seconds to ~58 seconds on average.
• Token consumption: with better context management, we reduced token usage by ~75 % in trials.

Those numbers still don’t hit elite human-expert performance, but they’re moving in the right direction. For reference, a standard text-to-SQL benchmark reports ~82 % accuracy for AI models and ~92 % for humans.

Lessons Learned

Some takeaways from this journey:

• Having a large LLM + schema context is necessary, but not sufficient. The agent must behave like an analyst: exploratory, adaptive, and iterative.
• Providing tools (inspection, query testing) significantly improves results letting the agent test its assumptions instead of trusting documentation alone.
• Memory matters. Real-world databases are big, changes accumulate, patterns repeat. Without remembering past findings, you waste time rediscovering.
• Transparent questioning (human-in-the-loop) builds trust. Users must feel in control, not like they’re handing over keys to a black box.
• Metrics are important. Without tracking accuracy, time, token cost, tool usage and user feedback, you don’t know whether your agent is improving.

Evolution Through User Feedback

Initial user feedback guided us to transition from a basic Retrieval-Augmented Generation (RAG) approach to a more sophisticated agentic flow with improved tools and human-in-the-loop methodologies. This shift resulted in significant improvements in both accuracy and efficiency, reducing token consumption and response times.

After several iterations, MetaAgent achieved a 50% success rate for correct first-try answers across queries. While this was a solid foundation, it fell short of current text-to-SQL benchmarks — approximately 82% accuracy for AI models and 92% for human experts (data engineers and database students) according to the BIRD Benchmark.

This gap prompted critical questions: Should we switch models? Refine prompt engineering? Or redesign the agent flow and tool definitions? To answer these systematically, I leveraged my data science expertise to implement a comprehensive agent evaluation framework.

Our Evaluation Process

Framework Design

Agent evaluation involves tracking three core dimensions: Accuracy, Efficiency, and Security. Achieving this requires robust observability to monitor:

• Tool usage patterns.
• Response times.
• Token consumption.
• Customer feedback.

We initially adopted LangSmith, which integrates seamlessly with LangGraph, for observability. Open-source alternatives like Arize Phoenix are also excellent options for LLM tracing and monitoring.

Metrics Definition

We established clear metrics for evaluation:

• Correctness: The SQL query must accurately answer the user’s question based on the database.
• Efficiency (Cost): Measured by total tokens used (input + output).
• Efficiency (Time): Shorter response times indicate optimized query generation and tool usage.
• Security: Leveraging Metabase API keys and encryption ensures only authorized queries execute.

Test Dataset Selection

To rigorously evaluate MetaAgent, we needed diverse databases and complex queries requiring database expertise. We adopted the BIRD Dataset

(Big Bench for Large-Scale Database Grounded Text-to-SQLs), which provides:

• 12,751+ unique question-SQL pairs.
• 95 large databases (33.4 GB total).
• 37+ professional domains (blockchain, healthcare, education, etc.).

Example

Question: Mention the first name of employee who took care the order id 10250

Human SQL

SELECT T1.FirstName, T1.LastName 
FROM Employees AS T1 
INNER JOIN Orders AS T2 ON T1.EmployeeID = T2.EmployeeID 
WHERE T2.OrderID = 10250

Optimization Environment

With observability, metrics, and test data in place, we began systematic optimization to reduce tokens and time while improving accuracy. We experimented with:

• AI model selection.
• System prompt engineering.
• Tool definitions.
• Agentic flow design.

After hundreds of experiments, three key insights emerged:

1. Simplicity is Best: We simplified our complex graph-based agent architecture to a streamlined experimentation flow.
2. Context Management Matters: We optimized context by including only relevant facts, re-contextualizing when needed, or asking clarifying questions when context grew too large.
3. Model Selection Impact: Models with stronger coding capabilities improved accuracy, while smaller, faster models handled simpler steps efficiently.

Results

These improvements yielded remarkable outcomes:

• 50% accuracy improvement, bringing MetaAgent close to current benchmarks.
• Response time optimization: From ~65 seconds to ~58 seconds.
• Token reduction: 75% decrease through improved context management.

Get Involved

We’d love to hear about your experience with AI agents and SQL generation. Connect with us:

• Create and test your account at MetaAgent.app
• Reach out via X (Twitter) or LinkedIn
• If you don’t have Metabase but want to try MetaAgent, we now offer setup services for your analytics environment.

The Journey from Fork to Platform

MetAgent began in February 2025 as a personal mission. As a data scientist who had helped found several startups, I’d spent countless hours fielding “quick” SQL requests from business teams. The pattern was always the same: talented people with great questions but limited SQL expertise, trying to extract insights from increasingly complex databases.

My initial solution was straightforward: fork Metabase and integrate AI directly into it. While the concept proved valuable, deployment was a nightmare. Users struggled to install my fork alongside their existing Metabase instances, and many worried about missing critical upstream updates. The RAG-based approach, though functional, was too rigid for real-world complexity.

That’s when I decided to pivot from a fork to a standalone platform: MetAgent.

The Agentic Evolution

My first MetAgent release introduced an agentic workflow using LangGraph, giving the AI tools to think like a data analyst rather than a translation service. The agent followed a structured approach:

1. Understanding business needs: Parsing the intent behind questions
2. Exploring database schemas and relationships: Mapping the data landscape
3. Building and running exploratory queries : Testing assumptions with real data

This approach delivered significantly better results than my RAG implementation. The agent made intelligent decisions, managed context effectively, and produced accurate queries for moderately complex databases.

But then we hit a wall.

The Scale Challenge

When enterprise customers started testing MetAgent on their production databases with hundreds of tables across multiple schemas , the system began to struggle. The context window became massive, causing the agent to lose focus on critical details. It was like asking someone to memorize an entire encyclopedia while answering a specific question.

I needed a more sophisticated approach, which led to two major innovations:

1. Memory Management: Teaching AI to Remember Like Humans

I implemented a dual-memory system inspired by human cognition:

Short-term Memory: Holds current context for single interactions, enabling quick recall of recent discoveries. After each tool use, the agent summarizes key findings, keeping only what’s relevant for the immediate task.

Long-term Memory: Stores persistent information across multiple queries. When the agent discovers important patterns through exploratory queries like specific status values or common join patterns it saves these insights for future use.

This approach dramatically improved performance on large databases. The agent could now handle complex schemas while maintaining focus on the specific question at hand.

2. Human-in-the-Loop: Embracing Ambiguity

Even with improved memory management, we encountered a fundamental challenge: ambiguous business logic. In the messy reality of startup databases, concepts like “revenue,” “GMV,” or “profit” might have multiple interpretations depending on the table or context.

Rather than guess, we built a conversational interface where the agent can ask for clarification:

This collaborative approach transforms MetAgent from a black box into a transparent partner that learns your business logic through interaction.

Try MetAgent Today

MetAgent.app is now available as a SaaS platform that connects securely to your Metabase instance. Your data never leaves your infrastructure , we only access schema information and run read-only test queries that you explicitly approve.

Key features:

• Secure API Integration: Works with any Metabase instance (self-hosted or cloud)
• Intelligent Memory: Handles databases of any size efficiently
• Conversational Interface: Clarifies ambiguities through natural dialogue
• Full Transparency: Shows exactly what queries will run before execution

What’s Next?

Building MetAgent has taught me that the future of data accessibility isn’t about replacing analysts , it’s about augmenting everyone’s analytical capabilities. By combining agentic workflows, intelligent memory management, and human collaboration, we’re making data truly accessible to entire organizations.

I’d love to hear about your data accessibility challenges. Are you struggling with complex queries? Is your team bottlenecked on SQL expertise?

Reach out on X/Twitter or LinkedIn, or try MetAgent yourself at metagent.app.

For more information go to our website https://churnless.ai/

The Evolution of My SQL Assistant

In my previous article, I showed how I built an AI-powered SQL assistant for Metabase using RAG architecture. After testing it with several customers, I learned something crucial: my agent was thinking like a machine, not like a data analyst.

Real analysts don’t just translate questions to SQL. They explore, validate, and iterate. They run quick queries to understand the data structure. They check what values actually exist in a status column before filtering on them.

So I rebuilt my assistant to work the way analysts actually work.

The Problem with My First Approach

My original RAG based agent had access to database schemas and documentation, but it was limited. It would often generate SQL that was technically correct but missed business nuances:

• It didn’t know that “approved” invoices might be marked as “success” in one database but “confirmed” in another
• It couldn’t explore the data to understand edge cases
• It had no way to validate its assumptions before presenting results

Building a Tool-Based Agent

Instead of just feeding my agent static documentation, I gave it tools to explore the database like a real analyst would:

1. Database Overview Tool

Provides high level schema information tables.

2. Relationship Explorer

Maps table relationships, even those nested 3+ levels deep. Essential for complex joins.

3. Table Inspector

Deep dives into specific tables, showing field types, constraints, and what aggregations are possible.

4. Query Tester

Lets the agent run exploratory queries to understand the data.

@tool
def test_sql_query_tool(user_id: str, connection_id: str, database_id: str, query: str) -> str:
    """Execute a SQL query and return formatted results"""
    try:
        response = call_agentapp_lambda('execute_query', {
            'userId': user_id,
            'connectionId': connection_id,
            'databaseId': database_id,
            'sqlQuery': query
        })
        
        if not response.get('success', False):
            error_msg = response.get('error', 'Unknown error')
            return f"❌ Query Failed Error: {error_msg}"

The Agent’s Thought Process

Using LangGraph, I structured the agent to think like an analyst:

1. Understands the business question
2. Explores relevant tables and relationships
3. Runs test queries (like SELECT DISTINCT status FROM invoices) to understand data patterns
4. Iteratively refines its approach based on actual data
5. Handles errors intelligently, retrying with new strategies
6. Metabase’s REST API lets it run read-only SQL from a Lambda.
7. Results flow back as a markdown table the LLM can inspect.

Try It Yourself

We’ve just launched www.metagent.app where Metabase users can try the new agent with 10 free credits. It works with any Metabase instance (self-hosted or cloud) through secure API keys.

What’s Next?

This journey taught me that the key to better AI agents isn’t just better prompts or more data. It’s giving them the right tools and teaching them to think like the humans they’re meant to help.

Have you tried building agents that mimic human problem-solving? I’d love to hear about your approach. Reach out on X/Twitter and let’s discuss!

Update August 2025: MetAgent has evolved again! I’ve solved the context window challenges for large databases through intelligent memory management and added human-in-the-loop collaboration for handling ambiguous queries. Learn about these improvements in my new article: https://medium.com/@sebastiancajamarca/beyond-agentic-building-a-human-in-the-loop-sql-assistant-that-scales-a276dfa20f60

As a data scientist, I’ve worked across multiple companies, leveraging databases to extract insights for machine learning models. Over time, I found myself frequently assisting BI and BA teams in solving complex SQL queries, especially when launching new projects or defining new metrics. These teams often struggled with limited knowledge of database structures and SQL syntax, leading me to wonder: Is there a better way to make data accessible to everyone, regardless of their technical background?

Inspired by Pinterest’s text-to-SQL approach and Uber’s QueryGPT, I decided to build an AI-powered SQL assistant — an accessible tool that enables users to query databases using natural language.

Why Metabase?

To build this tool, I chose Metabase, an open-source data visualization platform I had years of experience working with. Metabase offers several advantages:

• Comprehensive API: Extracts metadata from databases, including tables, columns, and relationships.
• Self-hosting: Ensures data security and control without requiring external access.

By leveraging Metabase, I could focus on designing an AI-driven query assistant without worrying about security, access control, or user interface development.

Implementation Overview

To enable AI-driven SQL generation, I designed a two-step agentic process within Metabase:

1. QueryDesigner Agent: Processes the user’s natural language question, leveraging a complete database schema to determine the necessary tables and query structure.
2. QueryArchitect Agent: Retrieves detailed table documentation from a vector database, generating a refined SQL query that users can execute within Metabase.

Architecture Breakdown

• Vector Database (Pinecone): Stores indexed metadata and table descriptions for retrieval-augmented generation (RAG).
• LLM Service (OpenAI): Translates natural language queries into SQL, using database metadata as context.
• Metabase UI Integration: Seamlessly integrates into Metabase’s query builder, allowing users to enter plain-language queries and receive executable SQL statements.

Designing the AI Query Agent Flow

To build an AI agent that delivers meaningful insights from data regardless of a user’s SQL expertise or familiarity with database structures I needed to start with a foundational approach. My goal was to create the simplest possible architecture that could function within Metabase while maintaining efficiency and ease of use. The following illustration provides an overview of this process.

This AI-powered querying system operates in two steps:

1. QueryDesigner Agent: When a user submits a question, this agent processes the full database documentation, including schema details and an overview of the tables. Based on this information, it determines the relevant tables, extracts the necessary data, and outlines the SQL query structure.
2. QueryArchitect Agent: Using the information from the QueryDesigner Agent, this second agent retrieves detailed table documentation from the vector database. The vector database contains structured documentation for every table and field, including relationships and user-provided descriptions. With this enriched context, the QueryArchitect Agent generates an optimized SQL query. The final query is then presented in the Metabase interface, allowing users to execute it within their database and select their preferred visualization format.

Setting Up the AI Configuration

Before processing queries, the system requires initial configuration:

1. Database Documentation Storage

• Users select tables to index and provide a high-level database description.
• Metabase’s metadata service extracts schema details, including field types and relationships.

2. AI Service Integration

• Admins securely add API keys for Pinecone (vector database) and OpenAI (LLM processing).
• A new AI Configuration section in Metabase’s settings allows easy management.

3. Indexing Process

• Once API keys are added, users click a Create Index button.
• The system generates structured documentation, providing a comprehensive database overview.
• This documentation enhances the AI’s ability to infer relationships (e.g., linking orders to addresses and cities for location-based queries).

User Experience: Querying with AI

To keep the workflow simple, I integrated the AI assistant directly into Metabase’s query builder:

1. User inputs a natural language question (e.g., “Show total sales per city”).
2. QueryDesigner Agent determines required tables and relationships.
3. QueryArchitect Agent fetches metadata from the vector database and constructs an SQL query.
4. SQL query is displayed in Metabase, allowing users to review, modify, and run it.
5. Users select visualizations based on the retrieved data.

This approach minimizes disruptions while making data queries accessible to non-technical users.

Future Enhancements

This is just the first iteration of the AI-powered SQL assistant. Future improvements include:

Leveraging Pre-stored Queries & User Logs

• Identifying commonly used queries as authoritative examples.
• Enhancing AI accuracy by learning from past queries.

Adding Feedback Loops

• Automatically validating generated SQL queries before presenting them to users.
• Iteratively refining queries based on detected errors.

Expanding Use Cases

• Exploring additional AI-powered features based on user feedback.

Get Involved

I’m actively looking for companies interested in testing this platform. If you’d like to try it for free, feel free to DM me on X (https://x.com/sebas88caja)!

Git: https://github.com/sazboxai/metabase_scg

Conclusion

By integrating AI with Metabase, I aim to democratize data access, allowing anyone, regardless of technical expertise, to extract insights effortlessly. This is just the beginning, and I’m excited to refine and expand this tool with real-world feedback. Let’s make data truly accessible!

Update August 2025: The journey continues! After evolving from RAG to agentic architecture, I’ve taken MetAgent even further. No more fork installations, it’s now a full SaaS platform with memory management for massive databases and human-in-the-loop collaboration. Read how MetAgent transformed from a Metabase fork to a scalable platform in my latest article: https://medium.com/@sebastiancajamarca/beyond-agentic-building-a-human-in-the-loop-sql-assistant-that-scales-a276dfa20f60

After much research and anticipation, last year we discovered a fantastic investment opportunity in Laureles, a neighborhood recently named one of the coolest in the world (source: Time Out).

Leveraging my fiancée’s expertise in civil engineering and home décor, we transformed the property into an extraordinary space. It’s now beautifully designed to comfortably accommodate up to five guests, ready to welcome any type of traveler.

Before renting out our property, we needed to make sure our investment would be profitable, especially with the current high interest rates. We wanted to see if this was a smart move given the opportunity cost.

To do this, I tapped into my data science skills to collect Airbnb listing data. I focused on key details like location, number of guests, price per night, ratings, and reviews. Using this data, I aimed to estimate a good nightly price that would keep occupancy high and ensure profitability.

This guide explains my process. I hope it helps other data-driven entrepreneurs considering similar investments. You can also experiment with the data using this tool: [Air_knn].

Gathering the data

Scraping data from Airbnb is no simple task, especially when you’re aiming to gather precise geolocated information. I discovered that by supplying two coordinate points to the Airbnb search URL, you can retrieve listings within that specific rectangular area. By providing the northeast and southwest points like this :

ne_lat={ne_lat}&ne_lng={ne_lon}&sw_lat={sw_lat}&sw_lng={sw_lon}

Airbnb generates a map view where each listing is accompanied by its exact coordinates. This method ensures you know the precise location of each listing and can accurately capture their coordinates.

Thanks to H3 library from Uber I get to divide Medellin main Neighborhoods (Laureles and Poblado ).

By using this approach, I was able to efficiently optimize the search for listings and extract their coordinates and features. As a result, I successfully gathered data from 2,380 listings, complete with all relevant characteristics. Now, it’s time to move on to creating the estimation model and suggesting optimal prices.

Pricing model

With all this data in hand, my next step was to develop a generalized model to explore the relationship between occupancy and price per night. The aim was to estimate the optimal price that would maximize revenue by balancing price and occupancy.

Airbnb listings come with a wide variety of amenities, but for this initial model, I chose to focus on a few key features to clearly understand the relationship between price and occupancy. Therefore, the model considered factors like the number of beds, guests, baths, location, and price.

As I ran the first model, I noticed collinearity between the price per night and the number of guests. To address this, I opted to use the price per guest per night instead.

Beta regression

The initial model I tested was a beta regression model, which is similar to linear regression but restricted to producing outcomes (Y) between 0 and 1. By evaluating the Mean Absolute Percentage Error (MAPE) through cross-validation, I found that this straightforward model achieved an error rate below 23%. While not perfect, this level of accuracy is sufficient to begin testing some hypotheses.

# Step 3: Set up K-Fold cross-validation
kf = KFold(n_splits=5, shuffle=True, random_state=42)
mae_scores = []
test_df = test_df[['occupancy' , 'price_per_guest' , 'number_of_beds' , 'number_of_bathrooms' ]]
# Step 4: Perform manual cross-validation
for train_index, test_index in kf.split(test_df):
    # Split data
    train_data, test_data = test_df.iloc[train_index], test_df.iloc[test_index]

    # Fit a Beta regression model (using GLM with logit link for Beta distribution)
    model = glm(formula='occupancy ~ price_per_guest + number_of_beds + number_of_bathrooms  ', 
                data=test_df, 
                family=Binomial()).fit()
    
    # Predict on test data
    predictions = model.predict(test_data)

    # Clip predictions to avoid values outside (0, 1)
    predictions = np.clip(predictions, 1e-5, 1 - 1e-5)

    # Step 5: Calculate MAE for this fold
    mae = mean_absolute_error(test_data['occupancy'], predictions)
    mae_scores.append(mae)


print(f"Mean MAE across 5 folds: {np.mean(mae_scores)}")
print(f"Standard Deviation of MAE: {np.std(mae_scores)}")

I aimed to use the estimated coefficient for price per guest to determine the demand elasticity. However, since the model assumes linearity, it didn’t accurately capture the diminishing returns associated with marginal price changes. As a result, the elasticity estimation wasn’t reliable.

Random forest regression

The next step involved using Random Forest Regression to explore how different pricing scenarios affect occupancy. By adjusting the price input and observing the changes in occupancy, I could calculate the percentage change in occupancy relative to price changes to estimate price elasticity. This approach captures complex, non-linear interactions, helping me understand how occupancy varies with different prices to create effective pricing strategies. However, it didn’t work as well as expected because I initially overlooked important features that influence price, such as amenities and image quality (with models like Vision language models or captioning models ), which reflect the apartment’s luxury and decor. I’ll address these factors in the next model, but for now, I needed something simple and effective to deploy quickly.

K nearest Neighbors

Ultimately, the best approach for analyzing price versus occupancy for similar listings was to use the K-Nearest Neighbors (KNN) method. By identifying the most similar listings in the area, I could estimate their revenue based on their price per guest and occupancy rates. This allowed me to pinpoint the optimal balance where price and occupancy levels maximize revenue. Consequently, this method provided a suggested price per night, a revenue estimate, and a list of comparable nearby listings.

Conclusions

While this model isn’t perfect, it offers a starting point for Airbnb entrepreneurs to estimate occupancy and pricing. Although it’s missing a lot of information, it helped us gauge how competitive our listing is and set expectations for our investment.

For future models, I plan to incorporate several improvements:

• Add amenities to the features, which will require more data to enhance prediction accuracy.
• Schedule the scraper to continuously update occupancy metrics, providing more accurate insights over time.
• Use vision and captioning models to analyze images and extract features related to home decor, luxury, and other amenities.
• Expand the model to include more cities, allowing more users to make well-informed decisions.

At Mercadoni we are changing how customers buy groceries in Latin America. Currently, we have partnered with more than 80% of the biggest retailers in Colombia, Mexico, and Argentina, giving us the possibility of delivering groceries from +200 different locations in Latam. In a weekly basis, our +3k shoppers are buying and delivering +100k different products from our retailers all over Latam. Failing on doing this process on time implies not only missing a customer expectation but also delaying the following scheduled orders and messing up our operations.

As simple as our delivery process may seem (see Fig. 1), we were having a huge problem at the moment we used to transfer money to the debit cards of our shoppers when they were about to pay at the supermarket cashier: In about 40% of our daily orders, we were sending less money than the shopper needed. This turned out in huge delays of our process since our shoppers had to wait for someone to manually transfer them the missing money (with thousands of orders per day, this process could take up to hours) and, on top of that, retailers were really pissed off since our shoppers were blocking their cashiers and their customers were unhappy about it.

To give you guys some context, the nature of this problem derives from data inconsistencies between the time our customers make an order and the moment our shoppers pick the products: (1) Prices vary daily in supermarkets, thus a price that you saw in our app may change with a given probability. (2) Prices of products that are charged by weight such as fruits and vegetables are very difficult to predict since shoppers may choose approximate amounts of the products when picking them. (3) Customers may ask for special products that are not shown in the app or change products when the shopper is picking. The total at the cashier will be for sure different in these cases.

The first solution that we tried was driven by our mythical entrepreneur common sense: A “quick win” fixed percentage increase to every money transfer. A simple excel analysis showed that a linear 25% increase would ensure 70% of the orders could be paid in the cashier… However, as our provider charges us per any amount transfer to debit cards, this cost increased by more than 25% the month we did it. In addition, giving more money to our shoppers increased fraud. Some of them used to accumulate money in their debit cards, then they bought things for them at supermarkets, and then they disappeared. At the end, the solution was generating more problems than the original problem :(

Frustrated by our common sense, we decided to use data science to solve this situation. To be honest, at the beginning it was difficult to convince the team. In startups like ours, we are obsessed with speed. And an exploratory data analysis with no clear deadlines is like Voldemort in this environment. This article is the first of many that we want to publish to change this wrong belief. Data Science and Machine Learning should not be confused with complexity. On the contrary, it is a synonym of scalability and optimization. At the end, we solved the problem in less time, and with better results than expected.

1. A mathematical problem

The first thing we did was to define our problem in mathematical terms. Particularly, we can see it as an optimization problem where the objective value is the money we transfer. In other words, we must find a function in terms of several dimensions such as order price, number of products, retailer, type of products, country, etc., such that we can optimize the money we transfer considering 2 major costs: transaction fees and time (See Fig. 2).

2. Our baseline

As Peter Drucker once wrote “what gets measured, gets managed”, in data we like to say “what gets measured, gets improved”. The second thing we did was to measure the baseline of our major costs (transaction fees and time delay per underestimating). The objective of this is to keep track of given metrics in order to discern whether our decisions incrementally optimize our function or not. In the case of time delay, we chose to measure the number of manual transfers as it is directly correlated with time, and it was easier to track. In fact, one order with manual transfer increases its delivery time in +25 minutes.

When we started this project, manual transfers were around 35–40% (See Fig. 3). We set our goal aggressively in 5%. In the case of transactional fees, we set a goal of a 40% decrease.

3. Exploratory data analysis

With our goals set and a clearly defined problem scope, now it is the time for exploring data. The idea behind this is to find the features of our model. That means, the inputs that generate the most impact to the function we are optimizing. Choosing these features well improves the prediction accuracy of the function model we are going to implement. Throughout this process, we found more than 20 features.

Just to mention a couple of examples in this step, we started by exploring the differences between the real price for the orders and our predictions depending on the supermarket.

• We found that retailers that are specialized in fruits and vegetables registered more differences between our prediction and the real price of our orders. So, we defined the retailer type as one of the features.
• We also discerned that retailers that update stock frequently in Mercadoni registered the least amount of differences between our prediction and the real price of our orders. Thus, an important feature for our model is the stock update frequency.

We also explored correlations between the orders in which our shoppers do not find some of the products customer ordered and the price difference of our predictions (See Fig. 5). The more they report products as not found or replace, the more variation we have. Consequently, another feature was the number of not found products in a given retailer.

Model definition and testing

The last part of this process is to create our data model. This is to implement a function that, depending on the features we defined in step 2, is going to predict an optimized value for the money we transfer. We also have to consider that this function is going to be used many times per day, even per minute. Thus, we chose to clean some of our features and implement a not so complex function using regressions. In particular, we minimized the mean square error (MSE) with an extra constraint in mind: ensure that 90% of our estimation was above the real value.

But a simple linear regression wasn’t enough. It will tend to centralize the error and we need to ensure 90% of the estimation to be above the value avoiding increasing fees in transaction. For this case in particular, quantile regression was the best solution because it ensures the estimation for a quintile.

As we tested, we saw that it slightly increase the MSE but it ensures that almost all of our shoppers will have their money on time. For those transactions we have doubts they will be assigned to manual support.

Conclusions

The main things we learnt with this project were :

1. With a good problem definition you will have key metrics to validate and a baseline
2. Data visualization allows you to test your hypothesis fast and build better features for your model
3. You are not seeking the best MSE or accuracy. You are seeking business metrics instead. Cost reduction, automation and process on time
4. Simple and fast data science service let you scale and validate at a good pace

The services had been up for almost 2 months. We reduced almost 70% of our expenditure on fees and reduced in about 80% the amount of manual transfers.

Many other data science processes have to be leveraged with manual processes such as data labeling, and for that reason we are now launching a game that will help us labeling products and prices. We are just scratching the surface of Data Science. Come, join us and help us shape the future of groceries in Latam.

Special thanks to David Camargo and Camilo Jiménez

Optimizing our delivery process with Data Science was originally published in Mercadoni Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.