# San Francisco Open Data Scraper (`parseforge/sf-open-data-scraper`) Actor

Scrapes any dataset from data.sfgov.org by its 4x4 ID. Returns each row as a flat record with optional SoQL filtering, column selection, and full-text search.

- **URL**: https://apify.com/parseforge/sf-open-data-scraper.md
- **Developed by:** [ParseForge](https://apify.com/parseforge) (community)
- **Categories:** Business, Developer tools, Automation
- **Stats:** 3 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $25.72 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

[![ParseForge](https://raw.githubusercontent.com/ParseForge/apify-assets/main/banner.jpg)](https://apify.com/parseforge?fpr=vmoqkp)

### San Francisco Open Data Scraper

**Scrape any dataset from San Francisco's open data portal, up to a million rows per run.** Pull building permits, 311 cases, police incidents, business registrations, and hundreds more. Filter with SoQL, select columns, and export to CSV, JSON, Excel, or XML.

San Francisco's open data portal holds thousands of public datasets, but downloading them manually means clicking through a browser, hitting row limits, and repeating for every update. This Actor reads directly from the Socrata API behind data.sfgov.org, so you can pull the exact rows you need with a single run. No browser, no manual exports, no API key registration.

| Who uses it | What they scrape San Francisco Open Data for |
|---|---|
| Urban planners | Tracking new building permits and development pipeline changes across neighborhoods. |
| Data journalists | Pulling police incident reports or 311 cases for public safety investigations. |
| Real estate analysts | Monitoring eviction notices and registered business locations for market trends. |
| Civic hackers | Feeding live city data into dashboards and community tools. |

### What it does

This Actor collects any dataset hosted on data.sfgov.org by its 4x4 ID and returns each row as a flat record.

- 📋 **Any public dataset:** choose from over 500 datasets including building permits, 311 cases, police incidents, parking citations, food trucks, and business registrations.
- 🔍 **SoQL filtering:** pass a $where clause to filter rows server-side before they reach your dataset.
- 📊 **Column selection:** specify a $select list to return only the fields you need.
- 📈 **Sort control:** add an $order clause to sort results by any column, ascending or descending.
- 🔎 **Full-text search:** use the query parameter to search across all string columns in the dataset.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

### What you can do with San Francisco Open Data data

**🏗️ Monitor new construction.**

A real estate analyst pulls the building permits dataset weekly, filters by 'issued' status and neighborhood, and feeds the results into a market report.

**📰 Investigate public safety trends.**

A data journalist scrapes police incident reports filtered by date range and category to map crime patterns for a story.

**📞 Track city service requests.**

A neighborhood advocate collects 311 cases by district and complaint type to measure response times and identify recurring issues.

**🚚 Find food truck locations.**

A developer pulls the mobile food facility permit dataset to power a real-time map of active food trucks in the city.

### Why choose this scraper

|  | What you get |
|---|---|
| **No API key** | Reads public Socrata endpoints directly, no registration needed |
| **Server-side filters** | SoQL $where, $select, $order, and $query reduce data before transfer |
| **Fixed schema** | Every row from a given dataset arrives with the same columns every run |
| **Multi-format export** | Download as CSV, JSON, Excel, XML, or push to a database |

### How it compares

No other Store actor targets San Francisco Open Data the same way, so the honest comparison is with the alternatives teams actually weigh.

| | San Francisco Open Data Scraper | Build it in-house | By hand |
|---|---|---|---|
| Setup | Run it now, zero config | Days of engineering | None, but hours per pull |
| When San Francisco Open Data changes | Maintained for you | You fix it | You re-learn the page |
| Proxies, retries, anti-bot | Built in | Your problem | Browser only |
| Output | Fixed JSON schema, CSV/Excel export | Whatever you build | Copy-paste |
| Cost | Pay per result | Engineering time | Analyst hours |

### Configure the run

Drive the Actor with a dataset ID from the dropdown, and optionally narrow results with SoQL where, select, order, and full-text query parameters so only matching rows land in your dataset. The Input tab lists every parameter.

A first run with the defaults:

```json
{
  "datasetId": "3pee-9qhc",
  "maxItems": 10
}
```

A larger pull:

```json
{
  "datasetId": "3pee-9qhc",
  "maxItems": 200
}
```

### Pricing

Pay-per-result: **$0.0343 per result** collected. You pay only for the results written to your dataset.

| Results collected | Approximate cost |
|---|---|
| 100 results | $3.43 |
| 1,000 results | $34.30 |
| 10,000 results | $343.00 |

New Apify accounts start with $5 in free credit.

### Free users

Free-plan runs return up to 10 results as a preview. [Upgrade your Apify plan](https://console.apify.com/sign-up?fpr=vmoqkp) to collect up to 1,000,000 results per run.

### Run it

1. [Create a free Apify account with $5 in credit](https://console.apify.com/sign-up?fpr=vmoqkp).
2. Open the [San Francisco Open Data Scraper](https://apify.com/parseforge/sf-open-data-scraper?fpr=vmoqkp).
3. Set your inputs and any filters, then click **Start**.
4. Export the results as CSV, Excel, JSON, or XML from the **Dataset** tab.

Run it programmatically through the [Apify API](https://docs.apify.com/api/v2) (`run-sync-get-dataset-items`) or the [ApifyClient](https://docs.apify.com/api/client/js) for JavaScript and Python.

### Use with AI agents (MCP)

Give an AI agent live access to San Francisco Open Data through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

```bash
claude mcp add --transport http apify "https://mcp.apify.com/?tools=parseforge/sf-open-data-scraper"
```

Then prompt it in plain language to run the scraper and read back the results.

### Troubleshooting

**Why am I getting no results?**

Check your $where clause for syntax errors. SoQL is case-sensitive and uses single quotes for strings. Try removing the filter to confirm the dataset itself returns rows, then add conditions back one at a time.

**Why does my run time out?**

Large datasets with many columns can take time. Try reducing maxItems, using $select to request fewer columns, or adding a $where filter to narrow the result set.

**Why are some columns missing from my output?**

If you used the $select parameter, only those columns are returned. Remove $select to get all columns. Also note that some datasets have computed columns that only appear when requested by name.

**Why do I get an error about an invalid dataset ID?**

The 4x4 ID must match exactly. Copy it from the dataset URL on data.sfgov.org. The ID is the part after '/d/' in the URL, like 'i98e-djp9'.

**Why is my $order clause not working?**

Column names in $order are case-sensitive. Check the exact column name from the dataset's metadata. Use 'column\_name ASC' or 'column\_name DESC' syntax.

### FAQ

| Question | Answer |
|---|---|
| Do I need an API key or token to use this? | No. The Actor reads from the public Socrata API endpoints, so no registration or authentication is required. |
| How do I find the dataset ID I need? | Browse data.sfgov.org, open the dataset you want, and copy the 4x4 ID from the URL. It looks like 'i98e-djp9' for building permits. The input dropdown also lists over 500 common datasets by name. |
| Can I filter rows before downloading? | Yes. Use the $where field with a SoQL clause like "status='issued'" to filter server-side. You can also use $select to pick columns and $order to sort. |
| What is the maximum number of rows I can get? | You can set maxItems up to 1,000,000 per run. The Socrata API itself may impose its own limits on very large datasets, but the Actor will fetch as many as allowed. |
| Which datasets are available? | Over 500 datasets from data.sfgov.org, including building permits, 311 cases, police incidents, business registrations, eviction notices, parking citations, food trucks, and many more. The input dropdown lists them all. |
| Can I search within a dataset for specific text? | Yes. The query parameter performs a full-text search across all string columns in the dataset, returning only rows that match. |
| What output formats are supported? | You can export results as CSV, JSON, Excel, or XML from the dataset tab, or push them directly to a database or storage integration. |
| Does this work with datasets that have geographic columns? | Yes. Geographic columns like points and polygons are returned in their native format. You can filter them with SoQL spatial functions in the $where clause. |
| How often is the data updated? | The Actor pulls live data from the Socrata API on every run. The freshness depends on how often the City updates each dataset, which varies by department. |
| Can I scrape multiple datasets in one run? | The Actor processes one dataset ID per run. To collect multiple datasets, run the Actor multiple times with different dataset IDs, or use an Apify workflow. |

### Related actors

Browse the full [ParseForge collection](https://apify.com/parseforge?fpr=vmoqkp) for more scrapers.

🆘 **Need help?** Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ **Disclaimer.** This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by City and County of San Francisco. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.

# Actor input Schema

## `datasetId` (type: `string`):

Socrata 4x4 ID of any dataset hosted on data.sfgov.org. Find the ID in the dataset URL (e.g. 'i98e-djp9' for building permits, 'pyih-qa8i' for business locations, 'vw6y-z8j6' for 311 cases, '5cei-gny5' for parking citations, 'rqzj-sfat' for food trucks, 'wg3w-h783' for police incidents).

## `maxItems` (type: `integer`):

How many datasets to collect per run.

## `where` (type: `string`):

Optional Socrata SoQL $where clause. Example: "status='issued' AND zipcode='94110'". See https://dev.socrata.com/docs/queries/where.html

## `select` (type: `string`):

Comma-separated list of columns to return. Example: 'permit\_number,status,issued\_date'. Default: all columns.

## `order` (type: `string`):

Sort order. Example: 'issued\_date DESC' or 'filed\_date ASC'

## `query` (type: `string`):

Free-text search across all string columns of the dataset

## Actor input object example

```json
{
  "datasetId": "3pee-9qhc",
  "maxItems": 10
}
```

# Actor output Schema

## `overview` (type: `string`):

Overview of scraped data

## `fullData` (type: `string`):

Complete dataset

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "datasetId": "3pee-9qhc",
    "maxItems": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("parseforge/sf-open-data-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "datasetId": "3pee-9qhc",
    "maxItems": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("parseforge/sf-open-data-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "datasetId": "3pee-9qhc",
  "maxItems": 10
}' |
apify call parseforge/sf-open-data-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,parseforge/sf-open-data-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ggi3FePb1qgXc29H9/builds/0abLFGXvcrIeJltVZ/openapi.json