I Built a Competitor Intelligence Agent with Claude and SerpApi

Build an AI competitor analysis agent with Claude and the SerpApi MCP server. Python code, multi-engine search, and a ready-to-use competitor report.

Most competitor research isn’t research.

It’s 20 open tabs, a half-finished spreadsheet and a gut feeling you present as a conclusion.

I’ve done it that way for years. Every time I pitch a client, I need to know who else is ranking for their keywords, who just raised money, who launched a new feature last week. So I built a competitor analysis AI agent that does the boring part for me.

If you’re:

• building AI agents that need fresh data,
• doing market research for clients,
• or tracking competitors for your own SaaS,

This is for you.

Why manual competitor analysis breaks

Here’s what a typical competitor check looks like when you do it by hand:

• Google the competitor’s brand and main keywords
• Open Google News, filter by last month
• Copy titles, links and dates into a sheet
• Repeat for every competitor
• Try to remember what changed since last time

It works once. It doesn’t scale.

The usual “fix” is scraping Google. Most developers discover too late that this means fighting CAPTCHAs, rotating proxies and HTML that changes without warning. You end up maintaining a scraper instead of analyzing competitors.

The second fix is asking an LLM. That one’s worse. Ask a model about a competitor’s latest launch and it will answer with confidence, using data that might be a year old.

The real problem: your agent can’t see the web

An LLM is good at reading, comparing and summarizing.

It’s bad at knowing what happened yesterday.

So competitor intelligence with AI is mostly a data access problem. Give the model live search results, force it to cite them, and the summarizing part takes care of itself.

The SerpApi MCP server: live search as a tool

SerpApi turns search engine results into structured JSON. Google, Google News, Bing, YouTube, Google Maps and many more, through one API.

The part that matters for this project is the official SerpApi MCP server. It’s hosted at mcp.serpapi.com, and it exposes a single search tool that covers every SerpApi engine. You pick the engine with a parameter, so one tool gives your agent organic results, news, shopping or local data.

Three things made me pick it for this agent:

1. No MCP client to build. It’s a hosted endpoint. You point Claude at a URL and it works.
2. One tool, many engines. The agent decides whether it needs google or google_news. I don't hardcode the flow.
3. A compact response mode. SerpApi responses are rich. For an agent, that richness costs tokens. Passing "mode": "compact" returns a lighter payload, which keeps long multi-search runs cheaper.

Try SerpApi for free
The free plan is enough to run this tutorial end to end.
→ Get your SerpApi API key

The 30-second version: Claude Desktop

If you just want to try the Claude MCP web search integration without code, add this to your Claude Desktop config:

{
  "mcpServers": {
    "serpapi": {
      "url": "https://mcp.serpapi.com/YOUR_SERPAPI_API_KEY/mcp"
    }
  }
}

Restart Claude, ask “What did Postmark announce in the last 30 days?” and watch it call the search tool.

That’s nice for a quick check. But a competitor monitoring workflow should run without you typing prompts. So let’s move it to Python.

Building the competitor analysis AI agent in Python

The plan is simple:

1. Claude gets access to the SerpApi MCP server through the Messages API.
2. A system prompt defines the research method and the report format.
3. A loop runs the agent for each competitor and saves a Markdown report.

No MCP client code. Anthropic’s MCP connector lets the Messages API talk to a remote MCP server directly. You declare the server in mcp_servers and reference it from tools with an mcp_toolset.

Step 1: Install and configure

pip install anthropic python-dotenv

Create a .env file:

ANTHROPIC_API_KEY=your_anthropic_key
SERPAPI_API_KEY=your_serpapi_key

Step 2: Define the research method

This is where most agents fail. “Research this competitor” is too vague. The model searches once, gets bored and writes a generic summary.

Get Kevin Meneses González’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

So the system prompt tells it exactly which engines to use and what to extract:

SYSTEM_PROMPT = """
You are a competitor intelligence analyst. You have a `search` tool
backed by SerpApi. Always pass "mode": "compact".

For the competitor you receive, run these searches:
1. engine "google": the brand name -> official site, positioning, top pages
2. engine "google": "<brand> pricing" -> pricing page and plan names
3. engine "google_news": the brand name -> news from the last 30 days
4. engine "google": "<brand> vs" -> who they get compared with
Rules:
- Only state facts you found in search results.
- Every claim needs a source URL.
- If you find nothing for a section, write "No public data found".
Return Markdown with these sections:
## <Competitor name>
### Positioning (1-2 sentences)
### Pricing signals
### Recent news (date, headline, URL)
### Compared against
### What changed / what to watch
"""

The “only state facts you found” rule is the important one. It turns the model from a storyteller into an analyst.

Step 3: Connect Claude to the SerpApi MCP server

import os
from datetime import date
from dotenv import load_dotenv
import anthropic

load_dotenv()
client = anthropic.Anthropic()
COMPETITORS = ["Resend", "Postmark", "Mailgun"]

def research(competitor: str) -> str:
    response = client.beta.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=4000,
        betas=["mcp-client-2025-11-20"],
        system=SYSTEM_PROMPT,
        mcp_servers=[
            {
                "type": "url",
                "url": "https://mcp.serpapi.com/mcp",
                "name": "serpapi",
                "authorization_token": os.environ["SERPAPI_API_KEY"],
            }
        ],
        tools=[{"type": "mcp_toolset", "mcp_server_name": "serpapi"}],
        messages=[
            {"role": "user", "content": f"Research this competitor: {competitor}"}
        ],
    )
    searches = [b for b in response.content if b.type == "mcp_tool_use"]
    print(f"{competitor}: {len(searches)} searches")
    # Keep only the text after the last tool result: the final report
    last_tool = max(
        (i for i, b in enumerate(response.content) if b.type == "mcp_tool_result"),
        default=-1,
    )
    final = response.content[last_tool + 1:]
    return "\n".join(b.text for b in final if b.type == "text")

Two details worth pointing out.

The API key goes in authorization_token, which SerpApi accepts as a Bearer header. That keeps it out of the URL, so it won't end up in logs.

And the response mixes block types. mcp_tool_use blocks show which searches Claude ran. text blocks hold the analysis, but also the small notes Claude writes between searches ("Let me check recent news..."). That's why the function keeps only the text after the last tool result. That's the actual report.

Step 4: Run it and save the report

if __name__ == "__main__":
    sections = [research(c) for c in COMPETITORS]

    report = f"# Competitor report: {date.today()}\n\n" + "\n\n---\n\n".join(sections)
    filename = f"competitor_report_{date.today()}.md"
    with open(filename, "w", encoding="utf-8") as f:
        f.write(report)
    print(f"Saved {filename}")

Run it:

python competitor_agent.py

Resend: 4 searches
Postmark: 4 searches
Mailgun: 5 searches
Saved competitor_report_2026-10-18.md

Notice the last line. Mailgun got five searches, not four. The agent decided one news query wasn’t enough and refined it. That’s the difference between an agent and a script.

A real use case: tracking email API competitors

I tested this with three email APIs I covered in a previous comparison article: Resend, Postmark and Mailgun.

Each section of the report looks like this:

## Postmark

### Positioning
[One or two sentences taken from their homepage and top results]
### Pricing signals
[Plan names and prices, with the pricing page URL]
### Recent news
- [date] [headline] ([source URL])
### Compared against
[Brands that appear in "Postmark vs" results]
### What to watch
[Claude's read on what changed, grounded in the items above]

What I liked:

• Every line has a source. I can verify any claim in one click.
• “Compared against” is gold. The "<brand> vs" query shows who the market sees as the real alternatives. Sometimes it's not who the company thinks.
• Empty sections are honest. When there’s no recent news, the report says so instead of inventing a launch.

What it doesn’t do:

• It doesn’t know traffic, revenue or market share. Search results are public signals, not private data.
• It costs money per run. Four or five SerpApi searches per competitor plus the Claude tokens. Cheap for a weekly report, not something to run every minute.
• Search results are a snapshot. Rankings change by country and by day. If location matters, pass location or gl in the search parameters.

From here you can build

Once the agent works for three competitors, the extensions are small:

• Weekly monitoring: run it with cron or an n8n schedule and diff this week’s report against last week’s.
• Ad intelligence: add SerpApi’s Google Ads Transparency Center engine to see which ads competitors are running.
• Local competitors: swap in the google_maps engine for businesses that compete by location, like clinics or restaurants.
• Alerts: send only the “What changed” section to Slack or email.

Key takeaways

• Competitor analysis with AI is a data access problem first. The model is only as current as its search results.
• The SerpApi MCP server gives Claude one search tool across many engines, so the agent decides which source it needs.
• A strict system prompt (“only facts you found, always cite”) is what makes the report trustworthy.

FAQs

❓ Does SerpApi have an MCP server?
✅ Yes. SerpApi maintains an official, open-source MCP server, also hosted at mcp.serpapi.com. It works with Claude Desktop, Cursor, VS Code and any MCP-compatible client.

❓ Can Claude search the web through MCP?
✅ Yes. Connect an MCP server that exposes a search tool, like SerpApi’s, and Claude can call it during a conversation or through the Messages API with the MCP connector.

❓ How do I get Google News results in Python?
✅ Use SerpApi with the google_news engine. You get titles, sources, dates and links as JSON, without scraping Google News yourself.

❓ Is it legal to monitor competitors with search data?
✅ This agent only reads public search results, the same ones anyone sees on Google. Still, check the terms of the tools you use and avoid collecting personal data.

Final thoughts

Most teams don’t lack competitor data.

They lack a system that collects it before they need it.

A competitor intelligence agent with Claude and SerpApi takes an afternoon to build. After that, the 20 tabs are gone.

Need technical content for your API or developer tool?
I write tutorials like this one: real code, real use cases, no marketing fluff.
→ See my work at kevinmeneses.com

Looking for technical content for your company? I can help — LinkedIn · kevinmenesesgonzalez@gmail.com