# CSV Cleaner & Deduplication Tool (`taroyamada/csv-data-cleaner`) Actor

Clean user-supplied CSV tables by trimming whitespace, removing empty rows, deduplicating selected columns, and sorting records without external data collection.

- **URL**: https://apify.com/taroyamada/csv-data-cleaner.md
- **Developed by:** [naoki anzai](https://apify.com/taroyamada) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 3 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 cleaned csv rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🧹 CSV Data Cleaner

Data engineers and operations teams can clean user-supplied CSV tables without sending the contents to a third-party enrichment API. The actor trims whitespace, removes empty rows, deduplicates selected columns, sorts records, and returns both structured rows and normalized CSV.

Research teams and analytics pipelines depend on this automated data cleaner to prepare CSV datasets for downstream analysis. Instead of fighting with complex spreadsheet formulas to identify redundant entries, you can pass your raw CSV URL directly to this utility. It systematically scans the file to deduplicate rows based on specific columns—like usernames, email addresses, or phone numbers—ensuring you never analyse duplicate rows as separate records.

Beyond basic deduplication, the tool actively sanitizes the content. It trims invisible whitespace from text fields, drops blank lines generated during interrupted scraping runs, and sorts the final list for easy review. By automating this cleanup phase, you ensure that every exported spreadsheet contains perfectly formatted data. Your final files will have clean website URLs, properly structured bios, and deduplicated social media posts, ready to fuel your next marketing campaign.

### Store Quickstart

Start with the **Quickstart** template (direct CSV URL). For Apify pipelines, use **Pipeline Cleaner** with datasetId.

### Key Features

- 🧹 **Trim whitespace** — Remove leading/trailing spaces from all cells
- 🗑️ **Remove empty rows** — Drop rows where all columns are empty
- 🔁 **Deduplicate by columns** — Remove duplicate rows by specified key columns
- 📊 **Sort by column** — Output sorted by any column
- 🔗 **Dataset or URL input** — Apify dataset ID or direct CSV URL
- 🔑 **No API key needed** — Pure JS, zero dependencies

### Use Cases

| Who | Why |
|-----|-----|
| **Data engineers** | Clean scraper outputs before downstream processing |
| **BI analysts** | Standardize CSV imports from multiple sources |
| **Marketing ops** | Clean analyst CSVs before downstream pipeline ingestion |
| **Data migration** | Normalize CSV files during system migrations |
| **Apify pipelines** | Post-process actor output datasets |

### Input

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| csvUrl | string |  | Direct CSV URL (or use datasetId) |
| datasetId | string |  | Apify dataset ID (or use csvUrl) |
| dedupColumns | string\[] | \[] | Columns for dedup key |
| trimWhitespace | boolean | true | Trim whitespace |
| removeEmpty | boolean | true | Remove empty rows |
| sortBy | string |  | Column to sort by |

#### Input Example

```json
{
  "csvUrl": "https://example.com/data.csv",
  "dedupColumns": ["email"],
  "trimWhitespace": true,
  "removeEmpty": true,
  "sortBy": "created_at"
}
```

### Input Examples

#### Example: Type detection only

```json
{
  "datasetId": "abc123",
  "detectTypesOnly": true
}
```

#### Example: Full cleanup pass

```json
{
  "datasetId": "abc123",
  "trimWhitespace": true,
  "normalizeNulls": true,
  "dedupeRows": true
}
```

#### Example: Column-specific transformation

```json
{
  "datasetId": "abc123",
  "transformations": [
    {
      "column": "email",
      "op": "lowercase"
    },
    {
      "column": "phone",
      "op": "e164"
    }
  ]
}
```

### Output

| Field | Type | Description |
|-------|------|-------------|
| `rowNumber` | integer | Original row index |
| `data` | object | Cleaned row as key-value pairs |
| `changes` | string\[] | List of cleanings applied to this row |
| `dropped` | boolean | Whether the row was removed |
| `dropReason` | string|null | Why the row was dropped (empty, duplicate, etc.) |

#### Output Example

```json
{
  "inputRows": 1250,
  "outputRows": 1180,
  "duplicatesRemoved": 45,
  "emptyRowsRemoved": 25,
  "cleanedData": [
    {"email": "user1@example.com", "name": "Alice", "created_at": "2026-01-01"},
    {"email": "user2@example.com", "name": "Bob", "created_at": "2026-01-02"}
  ]
}
```

### API Usage

Run this actor programmatically using the Apify API. Replace `YOUR_API_TOKEN` with your token from [Apify Console → Settings → Integrations](https://console.apify.com/account/integrations).

#### cURL

```bash
curl -X POST "https://api.apify.com/v2/acts/taroyamada~csv-data-cleaner/run-sync-get-dataset-items?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "csvUrl": "https://example.com/data.csv", "dedupColumns": ["email"], "trimWhitespace": true, "removeEmpty": true, "sortBy": "created_at" }'
```

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("taroyamada/csv-data-cleaner").call(run_input={
  "csvUrl": "https://example.com/data.csv",
  "dedupColumns": ["email"],
  "trimWhitespace": true,
  "removeEmpty": true,
  "sortBy": "created_at"
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

#### JavaScript / Node.js

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('taroyamada/csv-data-cleaner').call({
  "csvUrl": "https://example.com/data.csv",
  "dedupColumns": ["email"],
  "trimWhitespace": true,
  "removeEmpty": true,
  "sortBy": "created_at"
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

### Tips & Limitations

- Set `removeDuplicates: true` to deduplicate based on all columns.
- Use `delimiter` to handle TSV (`\t`) or semicolon-separated files.
- Combine with Phone Validator and Email Checker for full lead-data cleansing.
- Output dataset is ready for direct import into CRMs or databases.

### FAQ

**What CSV dialects are supported?**

Standard RFC 4180 CSV: comma-delimited, quoted fields, CRLF line endings. TSV not supported directly.

**Max CSV file size?**

In-memory processing. Works well up to ~100 MB / 1M rows. Larger files need chunking.

**Does it validate data types?**

No — cleaning operations only. For type validation, combine with validation libraries.

**Can I use this in Apify pipelines?**

Yes — provide datasetId from a prior actor run to clean that dataset directly.

**What's the max file size?**

Limited by actor memory (1024 MB by default). Tested up to 100k rows.

**Can I upload a local CSV?**

Provide a public URL via `csvUrl`. Use a service like file.io or S3 presigned URLs for private files.

### Related Actors

DevOps & Tech Intel cluster — explore related Apify tools:

- [🌐 DNS Propagation Checker](https://apify.com/taroyamada/dns-propagation-checker) — Check DNS propagation across 8 global resolvers (Google, Cloudflare, Quad9, OpenDNS).
- [🔍 Subdomain Finder](https://apify.com/taroyamada/subdomain-finder) — Discover subdomains for any domain using Certificate Transparency logs (crt.
- [📦 NPM Package Analyzer](https://apify.com/taroyamada/npm-package-intelligence) — Analyze npm packages: download stats, dependencies, licenses, deprecation status.
- [💬 Reddit Scraper](https://apify.com/taroyamada/reddit-data-scraper) — Scrape Reddit posts and comments from any subreddit via official JSON API.
- [GitHub Release & Changelog Monitor API](https://apify.com/taroyamada/github-release-monitor) — Track GitHub releases, tags, release notes, and changelog drift over time with one summary-first repository row per repo.
- [Docs & Changelog Drift Monitor API](https://apify.com/taroyamada/docs-changelog-drift-monitor) — Monitor release notes, changelog pages, migration guides, and key docs pages with one summary-first target row per monitored repo, SDK, or product.
- [Tech Events Calendar API | Conferences + CFP](https://apify.com/taroyamada/tech-events-intelligence) — Aggregate tech conferences and CFPs across multiple sources into a deduplicated event calendar for DevRel and recruiting workflows.
- [🔒 OSS Vulnerability Monitor](https://apify.com/taroyamada/oss-vulnerability-monitor) — Monitor open-source packages for known security vulnerabilities using OSV and GitHub Security Advisories.

### Cost

**Pay Per Event**:

- `actor-start`: $0.01 (flat fee per run)
- `dataset-item`: $0.001 per output item

**Example**: 1,000 items = $0.01 + (1,000 × $0.001) = **$1.01**

No subscription required — you only pay for what you use.

### ⭐ Was this helpful?

If this actor saved you time, please [**leave a ★ rating**](https://apify.com/taroyamada/csv-data-cleaner/reviews) on Apify Store. It takes 10 seconds, helps other developers discover it, and keeps updates free.

Bug report or feature request? Open an issue on the [Issues tab](https://apify.com/taroyamada/csv-data-cleaner/issues) of this actor.

# Actor input Schema

## `csvUrl` (type: `string`):

URL to fetch CSV from.

## `csvData` (type: `string`):

Raw CSV content (alternative to URL).

## `delimiter` (type: `string`):

Field delimiter character (comma, tab, etc.)

## `trimWhitespace` (type: `boolean`):

Remove leading/trailing whitespace from values

## `removeEmpty` (type: `boolean`):

Remove rows where all fields are empty

## `dedupColumns` (type: `array`):

Columns to deduplicate by.

## `sortBy` (type: `string`):

Column to sort by.

## `sortOrder` (type: `string`):

Sort direction: ascending or descending

## `delivery` (type: `string`):

Where to send results: dataset or webhook

## `webhookUrl` (type: `string`):

Webhook URL to POST results to (if delivery=webhook)

## `dryRun` (type: `boolean`):

Run without saving results (for testing)

## Actor input object example

```json
{
  "csvData": "name,email,city\nAlice,alice@example.com,Tokyo\nBob,bob@example.com,New York\nAlice,alice@example.com,Tokyo\n,,\nCharlie,charlie@example.com,London",
  "delimiter": ",",
  "trimWhitespace": true,
  "removeEmpty": true,
  "sortOrder": "asc",
  "delivery": "dataset",
  "dryRun": false
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "csvData": `name,email,city
Alice,alice@example.com,Tokyo
Bob,bob@example.com,New York
Alice,alice@example.com,Tokyo
,,
Charlie,charlie@example.com,London`
};

// Run the Actor and wait for it to finish
const run = await client.actor("taroyamada/csv-data-cleaner").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "csvData": """name,email,city
Alice,alice@example.com,Tokyo
Bob,bob@example.com,New York
Alice,alice@example.com,Tokyo
,,
Charlie,charlie@example.com,London""" }

# Run the Actor and wait for it to finish
run = client.actor("taroyamada/csv-data-cleaner").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "csvData": "name,email,city\\nAlice,alice@example.com,Tokyo\\nBob,bob@example.com,New York\\nAlice,alice@example.com,Tokyo\\n,,\\nCharlie,charlie@example.com,London"
}' |
apify call taroyamada/csv-data-cleaner --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,taroyamada/csv-data-cleaner"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Rrl9isk1pUMDbShKc/builds/vDFvLZTzwjpU3EwlO/openapi.json