curl https://www.npmjs.com/ returns 403

[trentm]

Select Topic Area

Question

Body

Hi. I've noticed that something within the last 24 hours, HTTP requests to any path on https://www.npmjs.com returns an HTTP 403 when run outside of the browser. For example:

% curl -I https://www.npmjs.com/package/@opentelemetry/exporter-metrics-otlp-grpc
HTTP/2 403
date: Fri, 19 Sep 2025 18:35:03 GMT
content-type: text/html; charset=UTF-8
content-length: 4512
x-frame-options: SAMEORIGIN
referrer-policy: same-origin
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
set-cookie: __cf_bm=l4t9x.U4Qpd2t6G3txISABC6hFsfx.PdgcgWlC8eEt8-1758306903-1.0.1.1-DYZhqP1rjt7HYjfWUo0A4UmZKBuOC6b4G4beHj9GmOyQOr9.TdKyk6GludU3HaFYGySh5lqU1eI7eNkNpM1bV9wbiuJBH7o_5TnYutLVBc0; path=/; expires=Fri, 19-Sep-25 19:05:03 GMT; domain=.npmjs.com; HttpOnly; Secure; SameSite=None
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
server: cloudflare
cf-ray: 981b2e42de03322f-SEA

The same request works in the browser -- i.e. the same request from Firefox (and observing in the Web Dev Tools "Network" panel) receives an HTTP 200 response. I've attempt to reproduce the same request Firefox was making by using the same headers in the curl ... command, but I still get a 403 response, so obviously I'm missing some detail.

% curl -sv https://www.npmjs.com/package/@opentelemetry/exporter-metrics-otlp-grpc -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:143.0) Gecko/20100101 Firefox/143.0' -H 'Accept-Encoding: gzip, deflate, br, zstd' -H 'DNT: 1' -H 'Accept-Language: en-CA,en-US;q=0.7,en;q=0.3' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -o /dev/null
...
< HTTP/2 403
...

FWIW, this is breaking linking checking in docs for the https://github.com/open-telemetry/opentelemetry-js repo.
If this is an intentional change for whatever reason (limiting against scraping, or whatever), then fair enough.
Thanks.

[V-Kisielius]

TL;DR: Within the last ~24 hours, the npm website (https://www.npmjs.com/*) began enforcing stricter Cloudflare Bot Management. Non‑browser clients (e.g., curl, link checkers, CI) now receive HTTP 403 with a __cf_bm cookie, while real browsers pass the challenge and get 200. This does not affect the registry APIs. For automation, either (A) use the registry/NPMS APIs, (B) treat 403 from www.npmjs.com as acceptable/skip those URLs, or (C) run a real browser (Playwright) for link checks. Avoid header hacks; they’re brittle and against the spirit of site policy.

---

Step‑by‑step: what’s happening and how to fix it

1) What changed

• Your curl -I to https://www.npmjs.com/package/@opentelemetry/exporter-metrics-otlp-grpc returns HTTP/2 403 and sets __cf_bm=....
• That cookie identifies a Cloudflare Managed Challenge. Browsers solve it (via JS + token); plain curl cannot.
• The npm website is the only thing impacted. The registry endpoints and NPMS API are designed for programmatic access and remain accessible without JS challenges.

2) Why copying Firefox headers doesn’t work

Cloudflare’s bot defenses check far more than headers:

• TLS & HTTP/2 fingerprint (JA3/ALPN, cipher suite ordering, pseudo‑header shapes)
• Timing/behavioral signals
• JavaScript proof‑of‑work / token binding via the __cf_bm cookie
  curl (or most link checkers) cannot produce a valid challenge solution, so the request is blocked with 403.

3) Options for CI and docs link checking

A) Prefer API endpoints (recommended for reliability)

Use these for existence/metadata checks without touching the website:

# Package existence via the npm registry (JSON)
curl -sS -I https://registry.npmjs.org/@opentelemetry/exporter-metrics-otlp-grpc

# Full package metadata (percent-encode the slash)
curl -sS https://registry.npmjs.org/%40opentelemetry%2Fexporter-metrics-otlp-grpc | head -n 20

# NPMS metadata (useful for docs/link verification that a package exists)
curl -sS https://api.npms.io/v2/package/%40opentelemetry%2Fexporter-metrics-otlp-grpc | head -n 20

If your docs currently link to https://www.npmjs.com/package/<name>, keep the link (for humans), but verify with the registry/NPMS URL in CI.

B) Tell your link checker to accept/skip www.npmjs.com

lychee (add .lychee.toml):

# Treat Cloudflare 403 as acceptable for human-facing sites
accept = [200, 203, 206, 301, 302, 307, 308, 403]

# Optionally skip the npm website entirely (keeps other links strict)
exclude = ["^https://www\\.npmjs\\.com/"]

markdown-link-check (add .markdown-link-check.json):

{
  "ignorePatterns": [
    { "pattern": "^https://www\\.npmjs\\.com/" }
  ]
}

This is a common, accepted approach for sites that intentionally block bot traffic.

C) Use a real browser for link checks

If you must verify the actual website URL, run a headless browser in CI (Playwright). It legitimately passes the challenge.

Minimal Playwright harness (Node.js):

# Install once in CI image
npx playwright install --with-deps

// scripts/check-link.js
import { chromium } from "playwright";

const urls = [
  "https://www.npmjs.com/package/@opentelemetry/exporter-metrics-otlp-grpc",
  // add more URLs as needed
];

let hadError = false;

const timeoutMs = 25000;

const check = async (url) => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  try {
    const resp = await page.goto(url, { waitUntil: "domcontentloaded", timeout: timeoutMs });
    // Some CF challenges may transiently 503/403, but resolve after redirects.
    // After DOM is ready, re-check the final response if available.
    const status = resp ? resp.status() : 200;
    console.log(`${status} ${url}`);
    if (status >= 400) {
      hadError = true;
    }
  } catch (e) {
    console.error(`ERR ${url} -> ${e.message}`);
    hadError = true;
  } finally {
    await page.close();
    await browser.close();
  }
};

const run = async () => {
  for (const u of urls) {
    await check(u);
  }
  if (hadError) process.exit(1);
};

run();

Run in CI:

node scripts/check-link.js

4) What not to do

• Don’t try to bypass Cloudflare with header spoofing or unofficial “browser impersonation” libraries; these are brittle, may break suddenly, and can violate site terms.
• Don’t rely on raw curl against www.npmjs.com for machine checks; it will remain unreliable.

5) If you want an official stance or relaxation

Open an issue with npm Support describing:

• OSS use case (docs link checking)
• Example URLs and timestamps
• CI environment/IP ranges (if relevant)
  Ask if they can suggest a sanctioned path or clarify intended behavior for automated link checks.

---

Ready-to-commit snippets for open-telemetry/opentelemetry-js

A) Registry probe in a CI job (Bash):

set -euo pipefail
PKG="@opentelemetry/exporter-metrics-otlp-grpc"
ENC_PKG="%40opentelemetry%2Fexporter-metrics-otlp-grpc"

curl -sS -I "https://registry.npmjs.org/${ENC_PKG}" | head -n 1
curl -sS "https://api.npms.io/v2/package/${ENC_PKG}" | jq -r '.collected.metadata.name'

B) .lychee.toml:

accept = [200, 203, 206, 301, 302, 307, 308, 403]
exclude = ["^https://www\\.npmjs\\.com/"]

C) .markdown-link-check.json:

{
  "ignorePatterns": [
    { "pattern": "^https://www\\.npmjs\\.com/" }
  ]
}

D) Playwright-based check for website URLs: see the Node script above.

---

Why this matters for your career (and how to communicate it)

• This is intentional bot mitigation, not an outage.
• Your remediation demonstrates security-aware CI and avoids fragile scraping.
• Propose a PR that (1) adds the link-checker config, (2) switches existence checks to the registry/NPMS, and (3) optionally provides a Playwright path for must-check web pages.

You’ll ship reliable docs checks without fighting the bot wall, and your CI will be future‑proof when other sites adopt similar protections.

[trentm]

Thanks.

[crysislinux]

I am using a browser, however I have to use a VPN to access https://www.npmjs.com/, but now I got 403 error partially.

Could this be solved?