# Benchmarky — Test Actors Before You Buy (`skilak/benchmarky`) Actor

Rank & live-test similar Apify actors on the same input. Compare success rate, speed, cost per result & data completeness.

- **URL**: https://apify.com/skilak/benchmarky.md
- **Developed by:** [Skilak A](https://apify.com/skilak) (community)
- **Categories:** Developer tools, Automation, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $250.00 / 1,000 live benchmark of one actors

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Benchmarky — Test Scrapers Before You Buy

Compare similar Apify actors **before** committing to one. Benchmarky ranks every candidate for a niche from public quality signals, and can then **live-run the top contenders against the same input on your account** — so you pick a scraper based on measured success rate, speed, cost per result and data completeness, not on marketing copy.

### Why this exists

Search the Apify Store for "google maps scraper" and you get thousands of results — a dozen of them near-identical clones. Which one actually works? Which one is quietly deprecated, priced to make you go away, or succeeds on only 60% of its runs? The store won't tell you. This actor will.

- **Scorecard mode** (cheap, seconds): pulls public metadata for every candidate — review ratings, monthly users, 30-day run success rates, last-modified dates, normalized pricing — and outputs a ranked comparison table with red flags.
- **Live Benchmark mode**: actually runs the top N candidates head-to-head on the same test input, with hard budget and timeout caps, then compares what came back.

### Quick start

#### 1. Scorecard — rank the field (default)

```json
{
    "mode": "scorecard",
    "searchKeyword": "instagram profile scraper",
    "maxCandidates": 10
}
```

Returns one ranked row per candidate plus a Markdown report (`REPORT.md` in the run's key-value store).

#### 2. Live benchmark with per-actor input mappings

```json
{
    "mode": "benchmark",
    "searchKeyword": "instagram profile scraper",
    "benchmarkTopN": 3,
    "testInput": {"usernames": ["nasa", "natgeo"]},
    "inputMappings": {
        "apify/instagram-profile-scraper": {"usernames": ["nasa", "natgeo"]}
    },
    "perActorBudgetUsd": "0.50",
    "maxItemsPerRun": 20
}
```

`testInput` is sent to every candidate; `inputMappings` overrides it per actor when schemas differ.

#### 3. Benchmark with expected output fields

```json
{
    "mode": "benchmark",
    "actorIds": ["apify/instagram-profile-scraper", "coderx/instagram-profile-scraper-bio-posts"],
    "testInput": {"usernames": ["nasa", "natgeo"]},
    "expectedFields": ["username", "followersCount", "biography"]
}
```

`expectedFields` defines what "complete data" means for you. Leave it empty and the benchmark auto-derives it from the fields most candidates return.

### What it costs — full transparency

**This actor's own fees (pay-per-event):**

| Event | Price | Charged when |
|---|---|---|
| `actor-start` | $0.05 | Once per run |
| `scorecard-actor` | $0.02 | Per candidate scored |
| `benchmark-run` | $0.25 | Per candidate live-tested |
| `llm-mapping` | $0.00 | Reserved for auto-mapping (Phase 3) |
| `report-generated` | $0.05 | Once, when REPORT.md is saved |

Typical scorecard of 10 candidates: **$0.30**. Benchmark of the top 3: **$1.05**.

**⚠️ Live benchmark sub-runs additionally bill YOUR account** at each candidate actor's normal price — that's the whole point: you're buying a real test. Every sub-run is hard-capped by `perActorBudgetUsd` (default $0.50), `perActorTimeoutSecs` (default 300s) and `maxItemsPerRun` (default 20). The report itemizes every cent: our fees and each sub-run's cost, separately.

If your run hits your Apify max-total-charge budget, the actor stops starting new work, flushes everything it has, and exits cleanly with partial results — never a crashed run.

### Output

One dataset row per candidate:

```json
{
    "rowType": "candidate",
    "actorId": "compass/crawler-google-places",
    "title": "Google Maps Scraper",
    "url": "https://apify.com/compass/crawler-google-places",
    "compositeScore": 87.5,
    "rank": 1,
    "signals": {
        "rating": 4.75, "reviews": 1200, "monthlyUsers": 42000,
        "runs30d": 210000, "successRate30d": 0.94, "modifiedDaysAgo": 6,
        "pricePer1k": 4.0, "pricingModel": "PAY_PER_EVENT"
    },
    "redFlags": [],
    "strengths": ["100% success across 12,407,823 runs in the last 30 days", "4.83★ across 176 reviews"],
    "weaknesses": [],
    "benchmark": {
        "status": "SUCCEEDED",
        "wallTimeSec": 84.2,
        "costUsd": 0.31,
        "itemsReturned": 20,
        "fieldCompleteness": 0.94,
        "schemaConsistency": 1.0,
        "costPerItem": 0.0155,
        "missingFields": [],
        "strengths": ["Fastest successful run (84s)"],
        "weaknesses": [],
        "sampleRows": [{"...": "..."}]
    }
}
```

Plus one `{"rowType": "summary"}` row with the winner, itemized costs and methodology, and a shareable Markdown report at key `REPORT.md` in the run's key-value store.

### Neutrality — why you can trust the numbers

**Vendors cannot pay us.** Our only revenue is the per-event fee paid by you, the
person running the comparison. Developers of scored actors can't pay to rank higher,
suppress a result, or be excluded — and there are no affiliate links anywhere. Every
actor is scored by the same published formula from the same public data, benchmark
runs come from *your* account so they're indistinguishable from ordinary customer
traffic, and every report carries its generation timestamp. When the top candidates
are within single-run noise, the report says **"too close to call"** instead of
manufacturing a winner. Full details: [METHODOLOGY.md](docs/METHODOLOGY.md).

### Methodology — how the composite score works

Reliability weighs more than popularity, popularity more than price. All signals come from Apify's public API.

| Signal | Max points | How |
|---|---|---|
| Rating quality | 25 | Bayesian-averaged review rating (5-vote prior at 4.0 stars, so 3 five-star reviews don't beat 1,200 at 4.7) |
| Adoption | 20 | Log-scaled monthly active users, capped so giants don't drown challengers |
| Momentum | 10 | Log-scaled 30-day run count |
| Freshness | 15 | Last modified ≤30d: 15 · ≤90d: 10 · ≤180d: 5 · older: 0 |
| Health | 15 | Deprecated → score is 0 overall; under maintenance → 0 health points; otherwise scaled by public 30-day success rate |
| Price | 10 | Cohort-relative $ per 1,000 results (cheapest = 10) |
| Developer | 5 | Developer's most recent activity across the cohort |

**Red flags:** `DEPRECATED`, `UNDER_MAINTENANCE`, `STALE_BUILD_180D`, `SUSPICIOUS_PRICING` (event priced > $1), `LOW_RATING_HIGH_USERS`, `LOW_SUCCESS_RATE_30D`, `REQUIRES_EXTERNAL_KEY`, `CLONE_FARM` (same developer, 3+ near-identical listings), `EMPTY_SUCCESS` (benchmark "succeeded" with 0 items — the infinite-cost-per-item trap).

**Benchmark ranking:** success → field completeness → cost per item → speed.

### Limitations

- Candidate discovery is only as good as store keyword search; pass explicit `actorIds` for a precise cohort.
- Actors requiring logins or external API keys are skipped by default (`skipActorsRequiringSecrets`).
- Benchmarks are point-in-time: a scraper that won today can break tomorrow. Re-run before big commitments.
- Until LLM auto-mapping ships, candidates whose input schema doesn't accept your `testInput` need an entry in `inputMappings` (schema mismatches are logged as warnings).

### FAQ

**What happens when a candidate's schema doesn't match my testInput?**
With an `inputMappings` entry, that exact input is used. Without one, `testInput` is sent as-is and a schema-validation warning is logged; the run then succeeds or fails like any real run — which is itself signal.

**What if I hit my budget cap mid-run?**
The actor stops launching new work, writes every result it already has, and finishes successfully with a "partial results" note in the report.

**Is this allowed?**
Yes. Scorecard mode reads only public metadata from Apify's official API. Benchmark mode starts ordinary paid runs on your account, exactly as if you'd clicked "Start" yourself. Results are reported neutrally: facts, timestamps and methodology.

**Do you run my token through third parties?**
No. An optional `userApiToken` (needed only if the default run token can't start other actors) is stored encrypted by the platform and never logged, never pushed to the dataset, never included in the report.

***

*Every REPORT.md links back here — if a benchmark report helped you pick a scraper, that's the loop working.*

# Actor input Schema

## `mode` (type: `string`):

Scorecard ranks candidates from public metadata. Live Benchmark actually RUNS the top candidates on your account — you pay their normal prices for the test runs.

## `searchKeyword` (type: `string`):

Find candidate actors by store search, e.g. 'instagram profile scraper'. Ignored if actorIds provided.

## `actorIds` (type: `array`):

Compare exactly these actors (format: username/actor-name). Overrides searchKeyword.

## `maxCandidates` (type: `integer`):

How many candidates to pull from the store search and score (2–25). Ignored when you pass explicit actorIds.

## `benchmarkTopN` (type: `integer`):

How many of the top-ranked candidates to actually run in Live Benchmark mode (2–5). Each run charges you the candidate's normal price.

## `testInput` (type: `object`):

Canonical test payload. Phase 2: must match each candidate's schema via inputMappings. Phase 3: auto-mapped by LLM.

## `inputMappings` (type: `object`):

Optional map of actorId -> exact input object for that actor. Bypasses auto-mapping.

## `expectedFields` (type: `array`):

Field names to measure completeness against, e.g. \['username','followersCount','biography']. Leave empty to auto-derive from the fields most candidates return.

## `perActorBudgetUsd` (type: `string`):

Hard cap passed as maxTotalChargeUsd to each sub-run.

## `perActorTimeoutSecs` (type: `integer`):

Hard timeout per candidate run, in seconds (30–600). A run still going after this is aborted and marked DNF\_TIMEOUT.

## `maxItemsPerRun` (type: `integer`):

Max dataset items to pull from each candidate run (1–1000). Passed as maxItems to bound both spend and the completeness comparison.

## `userApiToken` (type: `string`):

Needed only for Live Benchmark if the default run token lacks permission to start other actors. Stored encrypted; never logged.

## `skipActorsRequiringSecrets` (type: `boolean`):

Skip candidates whose input schema requires a secret (e.g. an external API key) you'd have to supply. When off, provide an inputMappings entry for those actors or they are skipped anyway.

## Actor input object example

```json
{
  "mode": "scorecard",
  "searchKeyword": "google maps scraper",
  "actorIds": [
    "compass/crawler-google-places",
    "someuser/google-maps-extractor"
  ],
  "maxCandidates": 10,
  "benchmarkTopN": 3,
  "testInput": {
    "usernames": [
      "nasa",
      "natgeo"
    ]
  },
  "inputMappings": {
    "apify/instagram-profile-scraper": {
      "usernames": [
        "nasa",
        "natgeo"
      ]
    }
  },
  "expectedFields": [
    "username",
    "followersCount",
    "biography"
  ],
  "perActorBudgetUsd": "0.50",
  "perActorTimeoutSecs": 300,
  "maxItemsPerRun": 20,
  "skipActorsRequiringSecrets": true
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchKeyword": "google maps scraper"
};

// Run the Actor and wait for it to finish
const run = await client.actor("skilak/benchmarky").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "searchKeyword": "google maps scraper" }

# Run the Actor and wait for it to finish
run = client.actor("skilak/benchmarky").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchKeyword": "google maps scraper"
}' |
apify call skilak/benchmarky --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,skilak/benchmarky"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Aq4ntFwqKfBqHIOew/builds/6dHqMqslQksZsKmgV/openapi.json