# pkg.go.dev Go Modules Scraper (`parseforge/go-pkg-modules-scraper`) Actor

Scrapes Go module listings from pkg.go.dev search results. Returns each module as a flat row with its import path, version, license, star count, and documentation link.

- **URL**: https://apify.com/parseforge/go-pkg-modules-scraper.md
- **Developed by:** [ParseForge](https://apify.com/parseforge) (community)
- **Categories:** Automation, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $19.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

[![ParseForge](https://raw.githubusercontent.com/ParseForge/apify-assets/main/banner.jpg)](https://apify.com/parseforge?fpr=vmoqkp)

### pkg.go.dev Go Modules Scraper

**Scrape Go module metadata from pkg.go.dev by keyword or package name, up to a million per run.** Every result includes the import path, version, license, documentation link, and star count. No API key required. Export to CSV, JSON, Excel, or XML.

Finding the right Go package means searching pkg.go.dev, reading docs, and comparing licenses and popularity by hand. This Actor reads the public search results directly, so you can collect structured metadata for hundreds of modules in one run. Feed it a keyword like "kafka client" or a package name, set a limit, and get a clean dataset back.

| Who uses it | What they scrape pkg.go.dev for |
|---|---|
| Go developers | Audit dependencies across a keyword space for license compliance and maintenance status. |
| DevOps engineers | Build an allowlist of vetted Go modules for internal registries. |
| Security researchers | Map the module landscape for a domain to identify abandoned or vulnerable packages. |
| Technical founders | Research the Go ecosystem around a problem space before building a new tool. |

### What it does

This Actor collects Go module listings from pkg.go.dev search results and returns each one as a flat row with its import path, version, license, star count, and documentation URL.

- 🔍 **Keyword search:** enter any free-text query, from broad terms like "http" to specific ones like "protobuf grpc".
- 📦 **Package name lookup:** search for an exact module path prefix to find related packages.
- 📊 **Structured metadata:** import path, version, license type, star count, and direct link to the docs page.
- ⚙️ **Configurable volume:** set a maximum number of modules to collect, from a single page up to a million per run.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

### What you can do with pkg.go.dev data

**📋 Build a dependency shortlist.**

A platform engineer searches for "service mesh" on pkg.go.dev, collects the top 50 results, and filters by license and star count to propose a vetted shortlist for the team.

**🛡️ Audit license compliance.**

A compliance officer runs a keyword search for every package category in production, exports the dataset, and flags any module with a missing or restrictive license.

**📈 Track ecosystem trends.**

A developer advocate scrapes modules for "observability" monthly, tracks new entries and star growth, and reports on the Go monitoring landscape.

**🔎 Discover alternatives.**

A backend engineer searches for "yaml parser", collects all matching modules, and compares their import paths, versions, and documentation links to pick the best-maintained option.

### Why choose this scraper

|  | What you get |
|---|---|
| **Import path** | The full module path you use in your go.mod file. |
| **License** | The detected open-source license for compliance checks. |
| **Star count** | Community popularity signal from the package's repository. |
| **Documentation link** | Direct URL to the package's pkg.go.dev documentation page. |

### How it compares

No other Store actor targets pkg.go.dev the same way, so the honest comparison is with the alternatives teams actually weigh.

| | pkg.go.dev Go Modules Scraper | Build it in-house | By hand |
|---|---|---|---|
| Setup | Run it now, zero config | Days of engineering | None, but hours per pull |
| When pkg.go.dev changes | Maintained for you | You fix it | You re-learn the page |
| Proxies, retries, anti-bot | Built in | Your problem | Browser only |
| Output | Fixed JSON schema, CSV/Excel export | Whatever you build | Copy-paste |
| Cost | Pay per result | Engineering time | Analyst hours |

### Configure the run

Drive the Actor with a free-text search query and a maximum item count, and it stops as soon as the limit is reached so only the modules you need land in your dataset. The Input tab lists every parameter.

A first run with the defaults:

```json
{
  "maxItems": 10
}
```

A larger pull:

```json
{
  "maxItems": 200
}
```

### Pricing

Pay-per-result: **$0.021 per result** collected. You pay only for the results written to your dataset.

| Results collected | Approximate cost |
|---|---|
| 100 results | $2.10 |
| 1,000 results | $21.00 |
| 10,000 results | $210.00 |

New Apify accounts start with $5 in free credit.

### Free users

Free-plan runs return up to 10 results as a preview. [Upgrade your Apify plan](https://console.apify.com/sign-up?fpr=vmoqkp) to collect up to 1,000,000 results per run.

### Run it

1. [Create a free Apify account with $5 in credit](https://console.apify.com/sign-up?fpr=vmoqkp).
2. Open the [pkg.go.dev Go Modules Scraper](https://apify.com/parseforge/go-pkg-modules-scraper?fpr=vmoqkp).
3. Set your inputs and any filters, then click **Start**.
4. Export the results as CSV, Excel, JSON, or XML from the **Dataset** tab.

Run it programmatically through the [Apify API](https://docs.apify.com/api/v2) (`run-sync-get-dataset-items`) or the [ApifyClient](https://docs.apify.com/api/client/js) for JavaScript and Python.

### Use with AI agents (MCP)

Give an AI agent live access to pkg.go.dev through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

```bash
claude mcp add --transport http apify "https://mcp.apify.com/?tools=parseforge/go-pkg-modules-scraper"
```

Then prompt it in plain language to run the scraper and read back the results.

### Troubleshooting

**Why am I getting no results?**

Your search query may be too specific or contain a typo. Try a broader keyword like "http" instead of a full package path, and check that the spelling matches what you see on pkg.go.dev.

**The run stopped before reaching my maxItems limit.**

pkg.go.dev returned fewer results than your limit for that query. This is expected when the search term is niche. Try a more general keyword to collect more modules.

**Some fields are empty in my dataset.**

Not every module on pkg.go.dev has a license detected or a star count available. Empty fields mean the information was not present on the page, not that the scrape failed.

**The Actor is running slowly.**

The Actor respects pkg.go.dev's response times. For large maxItems values, the run will take longer because it must fetch multiple pages. Reduce maxItems or split your search into narrower queries.

**Can I filter results by license or star count inside the Actor?**

The Actor collects all results that match your search query. To filter by license or stars, export the dataset to CSV or JSON and filter it in your spreadsheet or script.

### FAQ

| Question | Answer |
|---|---|
| Do I need a Google account or API key to scrape pkg.go.dev? | No. This Actor reads the public search pages directly. You do not need to register an app, create an API key, or log in. |
| What data does each result row contain? | Each row includes the module import path, latest version, detected license, star count, and a direct link to the documentation page on pkg.go.dev. |
| Can I search for an exact package name instead of a keyword? | Yes. The query field accepts both free-text keywords like "http router" and exact package paths like "github.com/gorilla/mux". |
| How many modules can I collect in one run? | You set the maximum with the maxItems field. The lower bound is 1 and the upper bound is 1,000,000 modules per run. |
| Does this Actor follow pagination? | Yes. It reads consecutive search result pages from pkg.go.dev until it reaches your maxItems limit or there are no more results. |
| What export formats are supported? | You can export your dataset in CSV, JSON, Excel, or XML format from the Apify platform. |
| Is this Actor affiliated with Google or the Go project? | No. This is an independent scraper that reads publicly available data from pkg.go.dev. It is not endorsed by or affiliated with Google LLC. |
| Can I schedule this Actor to run automatically? | Yes. Apify supports scheduled runs. You can set this Actor to run daily or weekly to track new module releases for your keywords. |
| Does the Actor handle rate limiting? | The Actor runs at a respectful request rate. If you need to collect a very large number of modules, consider spreading the work across multiple runs with different queries. |
| What happens if my search query returns no results? | The run completes with an empty dataset. Check your spelling or try a broader keyword. The Actor does not fail when there are no matching modules. |

### Related actors

Browse the full [ParseForge collection](https://apify.com/parseforge?fpr=vmoqkp) for more scrapers.

🆘 **Need help?** Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ **Disclaimer.** This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Google LLC. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.

# Actor input Schema

## `maxItems` (type: `integer`):

How many modules to collect per run.

## `query` (type: `string`):

Free-text search query, e.g. "http", "kafka client", "yaml".

## Actor input object example

```json
{
  "maxItems": 10,
  "query": "http"
}
```

# Actor output Schema

## `results` (type: `string`):

Scraped records.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "maxItems": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("parseforge/go-pkg-modules-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "maxItems": 10 }

# Run the Actor and wait for it to finish
run = client.actor("parseforge/go-pkg-modules-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "maxItems": 10
}' |
apify call parseforge/go-pkg-modules-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,parseforge/go-pkg-modules-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/rzZzA5Oz0ZM2ffPcw/builds/vjzSJQ6AkncAWsNCx/openapi.json