The Agentocene
Imperva’s 2025 Bad Bot Report put the number at 51% — for the first time in a decade, automated traffic had passed human traffic across the web. Akamai documented a 300% year-over-year surge in AI-driven bots. Cloudflare’s crawler data told the same story from a different angle. I’d been reading these reports for weeks when I got to lead a fireside chat at AWS EdgeCon 2026 — and two words were on everyone’s mind, PMs to architects: Agentic Experience. In passing, someone noted our own numbers tracked the same direction. Nobody even blinked. Three independent data sources, three independent vantage points, all converging on the same conclusion. The web’s fastest-growing traffic source is no longer human. Not as a trend line. As a present-tense measurement.
The Numbers
The shift didn’t announce itself. It showed up in the metrics.
Akamai’s 2025 Digital Fraud and Abuse Report, published in November, documented a 300% year-over-year surge in AI-driven bot traffic. That number captures what their threat detection systems can classify — the known crawlers with identifiable user-agent strings. The actual number is higher, because the most sophisticated agents don’t announce themselves.
Cloudflare’s data is more granular and, frankly, more unsettling. Their August 2025 analysis of crawler-to-referral ratios revealed the gap between what AI systems consume and what they send back:
- Akamai — AI bot traffic grew +300% YoY (Nov 2025)
- Cloudflare — Anthropic’s crawl-to-refer ratio: 38,065 : 1 (Aug 2025)
- Cloudflare — PerplexityBot request growth: +157,490% (Jul 2025)
- Cloudflare — GPTBot request growth: +305% (Jul 2025)
- Originality.AI — 35.7% of top 1,000 sites blocking GPTBot (Aug 2024)
38,065 pages crawled for every single human visitor referred back — and that was actually down from 286,930:1 in January 2025, an 86% improvement. Still not a search engine building an index to send you traffic. That’s a knowledge extraction pipeline with a rounding-error referral rate.
And these numbers understate reality. They only measure crawler traffic — bots with identifiable user-agent strings hitting web pages. They don’t capture API-to-API agent traffic, which bypasses web servers entirely. They don’t capture agents operating through browser automation frameworks. They don’t capture the internal orchestration calls between agent components. The visible crawler traffic is the tip of an iceberg, and the iceberg is growing faster than the tip.
The Multiplication Effect
Here’s the mechanism. When a human browses the web, they generate a predictable, modest number of requests. Open a page, click a link, scroll, maybe search. A typical browsing session might produce 50–100 HTTP requests across 10–15 minutes.
When an AI agent acts on a human’s behalf, the math changes completely.
A single human intent triggers agent decomposition, which fans out into parallel API calls, retries on failures, synthesis of results, and a final response. The MCPMark benchmark, published in September 2025, quantified this multiplication effect by stress-testing agent behavior across 127 demanding tasks. A single task averaged 16.2 execution turns and 17.4 tool calls. Each tool call can itself trigger multiple HTTP requests — an API query, authentication, pagination, retries. One unit of human intent becomes hundreds of machine requests.
The contrast is stark:
- Requests per intent — Human: 50–100. Agent: 500–5,000+.
- Duration — Human: 10–15 min session. Agent: seconds to hours, goal-driven.
- Parallel connections — Human: 6–10 per domain. Agent: dozens across many domains.
- Retry behavior — Human: manual refresh, maybe once. Agent: automatic, exponential backoff, multi-strategy.
- Session model — Human: login → browse → logout. Agent: stateless or delegated, no session concept.
One human thought becomes a cascade of machine actions, each generating its own traffic, its own load, its own entry in your access logs.
The Blocking Response
The web’s response to this shift has been predictable: block everything.
Originality.AI’s August 2024 study found that 35.7% of the world’s top 1,000 websites actively block GPTBot via `robots.txt`. The major news publishers led the charge — The New York Times blocks GPTBot, CCBot, and Google-Extended over training data concerns. CNN and Reuters block GPTBot to prevent content summarization that bypasses ad impressions. Bloomberg blocks both GPTBot and Anthropic’s crawler over paywall circumvention.
But `robots.txt` was designed for an era when crawlers were polite and identifiable. It’s a gentleman’s agreement, not a technical control. The evidence suggests that many AI crawlers ignore `robots.txt` directives entirely, or use user-agent strings that don’t match their block rules. Cloudflare’s data shows crawler traffic from AI systems continuing to grow even as blocking adoption increases.
This creates an arms race with no clear winner. Publishers add more bot signatures to their block lists. Crawlers rotate user-agent strings. Publishers deploy JavaScript challenges. Crawlers use headless browsers. The pattern is familiar to anyone who’s worked in bot mitigation — and it’s a pattern where the defense always runs behind.
Get Alex Moening’s stories in your inbox
Join Medium for free to get updates from this writer.
The web’s immune response — robots.txt, rate limits, CAPTCHAs — was designed for a slower pathogen. Agents evolve faster than the antibodies.
What This Means If You Build Things
If you’re an API designer, a platform architect, or an edge engineer, the assumptions underlying your systems are changing faster than most organizations realize.
- Old: Rate limiting protects fairness. New: Rate limiting is containment — one “user” generates 100x the requests.
- Old: Identity = a session cookie. New: Identity = delegated authority from a human to an agent to a sub-agent.
- Old: Caching optimizes delivery. New: Caching is part of a decision loop — stale data breaks agent reasoning.
- Old: Routing is geography-based. New: Routing is intent-based — agents don’t have geography.
- Old: Auth gates protect resources. New: Auth gates must understand delegation chains, not just credentials.
For API designers: Your rate limits need to understand that a single API key might back an orchestration layer generating hundreds of calls per task. Your error responses need to be machine-actionable, not human-readable. A 429 with a `Retry-After` header is useful. A 429 with “Please try again later” in an HTML page is useless to an agent.
For platform architects: Your authentication model needs to handle delegation. A human authorizes an agent. The agent spawns sub-agents. Each sub-agent calls your API. Whose credentials are those? Whose quota do they count against? OAuth scopes were designed for apps, not for autonomous agents that compose workflows across services.
For edge engineers: This is personal for me — I work on CloudFront at AWS, so I watch this play out in the traffic data every day. Edge logic is no longer just about delivering content closer to users. It’s about making decisions at the point of request: Is this an agent? What’s its delegation chain? Should I serve a cached response or force a fresh one because the agent needs current data?
The infrastructure layer isn’t just plumbing anymore. It’s part of the agent’s reasoning environment.
Naming the Epoch
I’m using this term deliberately. The parallel is precise.
The Anthropocene — the geological epoch defined by human activity as the dominant influence on the environment — gave us a framework for understanding how our direct actions reshape the planet. We built cities. We burned fossil fuels. We altered ecosystems. The defining characteristic was human agency acting directly on the physical world.
The Agentocene is the next layer. We’re no longer the primary actors on the systems we’ve built. We create agents. Those agents act on our behalf, at scale, continuously, across every digital system they can reach. The human doesn’t navigate the web anymore. The web is navigated — and increasingly, *executed* — on their behalf.
Tim Berners-Lee saw this coming. In Weaving the Web (1999), he wrote: “The intelligent agents people have touted for ages will finally materialize.” He was 27 years early on the timeline, but his architectural intuition was exact — the web was always designed to be machine-readable. We just didn’t have machines capable of reading it with intent until now.
The phase shift isn’t that agents exist — or even that bots outnumber humans on the web, which Imperva’s reports have documented since the mid-2010s. It’s that the type of bot changed. Legacy bots scraped and indexed. These agents reason, decide, and act. That’s not incremental. That’s epochal.
We didn’t just add AI to the web. We changed the primary actor. The Anthropocene was humans reshaping the world. The Agentocene is humans building systems that reshape it for them — faster, wider, and without stopping to sleep.
What I’m Watching
The shift is real, the data confirms it, and the legal system is scrambling to draw boundaries. Here’s what I think comes next:
Agent-to-agent protocols. The current model — agents pretending to be browsers, hitting web pages meant for humans — doesn’t scale. Purpose-built protocols for agent-to-agent communication are already forming. Anthropic’s Model Context Protocol (MCP) and Google’s Agent-to-Agent (A2A) framework are the early drafts.
Delegated identity standards. OAuth was built for “this app can access your data.” The Agentocene needs “this agent, acting on behalf of this human, with these constraints, can perform these actions, and can delegate sub-tasks to these other agents.” I expect RFC-track work on this within 12–18 months.
Intent-aware CDNs. Edge networks will evolve from “deliver content fast” to “understand what’s asking and why.” The same URL requested by a human browser, an AI crawler, and an autonomous agent executing a purchase workflow should potentially return different responses — not to deceive, but to serve each requestor appropriately.
The end of the session model. Sessions assume a human sitting at a screen. Agents don’t have sessions. They have tasks. Your analytics, your A/B tests, your conversion funnels — all built on the session abstraction — need rethinking when a growing share of your “users” don’t maintain state between requests.
I’ve been moving bits across the internet since 1999. I watched the web go from static HTML to dynamic applications to mobile-first to API-first. Each transition changed what “a user” meant. This transition is the biggest one yet, because it changes whether “a user” is human at all.
That’s what I see from the edge.
Also documented over on my website https://alexmoening.com/dev-thoughts/the-agentocene.html
