How Google Trends’ hidden API works, and why your scraper keeps getting 429s

AI-assisted: I drafted this with Claude and checked it against the code of a Google Trends scraper I run in production.

Google Trends has no official public API. Every chart on trends.google.com is filled by a small set of internal JSON endpoints, though, and they’re stable enough to build on. This post walks through the request sequence, the one gotcha in parsing the JSON, and the part that actually breaks most scrapers: rate limiting.

The two-step dance: explore, then widgetdata

Every Trends chart needs two requests.

Step 1: /trends/api/explore. You describe the comparison you want, and Google answers with a list of widgets: one per chart on the page, each with its own request object and a short-lived token.

const req = {
  comparisonItem: [{ keyword: 'python', geo: 'US', time: 'today 12-m' }],
  category: 0,
  property: '', // '' = web search; also 'images', 'news', 'froogle', 'youtube'
};
const url = 'https://trends.google.com/trends/api/explore?' +
  new URLSearchParams({ hl: 'en-US', tz: '0', req: JSON.stringify(req) });

To compare terms, put up to five items in comparisonItem. That's Google's own limit, same as on the website.

Get Darian Danciu’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Step 2: /trends/api/widgetdata/<endpoint>. For each widget, send its request and token back:

• TIMESERIES → endpoint multiline: interest over time (0–100)
• GEO_MAP → endpoint comparedgeo: interest by region
• RELATED_QUERIES → endpoint relatedsearches: top and rising queries
• RELATED_TOPICS → endpoint relatedsearches: top and rising topics

const data = await getJson(`/trends/api/widgetdata/multiline`, {
  req: JSON.stringify(widget.request),
  token: widget.token,
});

Tokens expire quickly. If a widget call fails, don’t retry it with the old token: re-run the whole explore → widgetdata sequence.

The )]}' prefix

Every response starts with )]}' (sometimes )]}',), an old defence against JSON hijacking. JSON.parse chokes on it, so strip everything before the first { or [:

function stripPrefix(text) {
  const i = text.search(/[\[{]/);
  return JSON.parse(text.slice(i));
}

That prefix is also a handy health check. When Google is throttling you, it sometimes answers HTTP 200 with an HTML “unusual traffic” page instead of JSON. Treat any response that doesn’t start with )]}' as a block, not as data:

if (!text.trimStart().startsWith(")]}'")) throw new RetryableError('Blocked (non-JSON response)');

Why you get 429s, and what actually helps

Trends rate-limits hard, and per IP. Here’s what made the difference for me, in order of impact:

1. Get a cookie first. Load https://trends.google.com/trends/?geo=US once and keep the NID cookie it sets. API calls without it are throttled almost at once.
2. Treat a block as “this session is burned”. On a 429, a 403 or the HTML page, throw away the cookie and the IP together and start a new session. Retrying on the same IP mostly earns more 429s.
3. Back off exponentially, with jitter. Something like min(30s, 1s × 2^attempt) × random(0.5–1.5). Without the jitter, parallel workers retry in lockstep.
4. Body-read timeouts are blocks too. A throttled connection sometimes sends headers and then stalls. If your retry logic only catches connection errors, this one will kill the run.
5. Datacenter IPs are shared and often already burned. On a cloud platform, two blocks in a row is a good signal to switch the rest of the run to residential IPs. Keep bulky requests off them, since residential traffic costs more.

Things that will surprise you

• Values are relative, not search counts. 100 is the peak of that chart. Two separate requests for “python” and “rust” are not comparable. Compare them in the same request.
• The last point is often isPartial: true. That's the current, unfinished week or day. It will still change.
• Related topics are often empty. Google currently returns an empty list for many searches, even when the website shows topics. Plan for it.
• Daily trending searches don’t need any of this. https://trends.google.com/trending/rss?geo=US is a plain RSS feed with approximate traffic and news links.

If you’d rather not maintain this

I packaged all of the above (session rotation, the residential fallback, flat rows ready for a spreadsheet, and a link back to the matching chart on every row) as a Google Trends Scraper on Apify. It charges per result, and failed terms are free. The approach in this post works fine on its own if you’d rather run it yourself.