# RAG Post Processor - Text Cleaner & Chunker for LLM Pipelines (`jalicia/rag-post-processor`) Actor

Clean and chunk scraped text into embedding-ready chunks, sized in tokens (tiktoken) rather than characters. Preserves headings, tables, code blocks and links. Every chunk carries its token count, heading path and content hash. Chain after any scraper. Billed per KB of text processed.

- **URL**: https://apify.com/jalicia/rag-post-processor.md
- **Developed by:** [Jordan Wagner](https://apify.com/jalicia) (community)
- **Categories:** AI, Automation, Other
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 input kb processeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RAG Post Processor — token-aware, structure-preserving chunker

Turn scraped pages into chunks you can embed without surprises: **sized in
tokens**, **split on document structure**, and **carrying the metadata a
retrieval pipeline actually needs**.

Drop it after any scraper, or feed it items directly.

***

### Why tokens, not characters

Character budgets are the default in most chunkers, and they don't mean
anything to an embedding model. Measured with `cl100k_base`, one
1000-character chunk is:

| Content | chars/token | tokens in 1000 chars |
|---|---|---|
| English prose | 6.08 | ~160 |
| Python source | 4.47 | ~220 |
| JSON | 2.41 | ~415 |
| German | 3.27 | ~300 |
| Japanese | 0.89 | **~1080** |

A single character setting gives a 7× spread. `max_tokens` here is a hard
ceiling under the encoding you select — no emitted chunk ever exceeds it.

> `text-embedding-3-small` and `text-embedding-3-large` use **`cl100k_base`**,
> not `o200k_base`, despite being newer models. That's the default here.

***

### Why structure

Chunking by slicing a character stream cuts through the things that make a
passage interpretable. This Actor parses the document into blocks first —
headings, paragraphs, fenced code, tables, lists, quotes — and packs whole
blocks into chunks.

- **Code that must be split keeps its fence and language on every part**, so
  every chunk is still valid, highlightable code.
- **A table that must be split repeats its header row on every part**, so no
  chunk is an anonymous wall of pipe-separated values.
- **Lists split between items, never inside one.**
- **Headings lead their content**, never trail at the end of the previous chunk.
- **Every chunk carries its heading path**, both inline (optional breadcrumb)
  and as structured `heading_path` / `section` fields.
- **Chunks begin and end on whole words and whole sentences.**

### Why the metadata

Every row carries what an ingestion pipeline would otherwise have to recompute:

- `token_count` — exact, under your chosen encoding, so you can batch embedding
  calls without re-tokenizing
- `content_hash` + `chunk_id` — stable across runs, so you can upsert unchanged
  chunks instead of paying to re-embed them
- `heading_path` / `section` — for metadata filtering and for showing users
  where an answer came from
- `block_kinds` — filter code out of prose retrieval, or the reverse
- `source_field` — which field of the input item was actually read
- `overlap_tokens` — how much of this chunk is repeated context

The run log also reports **token amplification** (output tokens ÷ input
tokens), which is the multiplier your embedding provider will bill you for.

***

### Input

Four input paths, tried in order. Chaining reads **every page** of the source
dataset, not just the first.

| Field | Type | Default | Description |
|---|---|---|---|
| `datasetId` | string | — | Dataset ID from a previous Actor run. Chain after any scraper. |
| `data` / `items` | array | — | Items passed inline. |
| `text_field` | string | auto | Read this field instead of auto-detecting. |
| `max_tokens` | integer | `512` | Hard token ceiling per chunk (16–8191). |
| `overlap_tokens` | integer | `64` | Whole-sentence overlap, clamped to 50% of `max_tokens`. |
| `min_tokens` | integer | `24` | Below this, merge into the previous chunk. Text is never dropped. |
| `encoding` | enum | `cl100k_base` | `cl100k_base`, `o200k_base`, `p50k_base`, `r50k_base`. |
| `split_on_heading_level` | integer | `0` | `0` packs sections; `1`–`6` forces a break at that heading level. |
| `include_heading_context` | boolean | `true` | Prefix each chunk with its heading breadcrumb. |
| `preserve_links` | boolean | `true` | Keep URLs; convert anchors to markdown links. |
| `drop_nav` | boolean | `true` | Drop `<nav>`/`<menu>`. Script/style/comments are always dropped. |

Text is auto-detected from `markdown`, `text`, `content`, `body`,
`page_content`, `html` and ~25 other common scraper field names. The field
chosen is reported on every row as `source_field`.

Chunk sizes moved from characters to tokens in v1.0. The old `chunk_size`,
`overlap` and `min_chunk_chars` inputs are still accepted: they are converted
to their token equivalents at ~4 characters per token, and the conversion is
written to the run log.

#### Example input

```json
{
  "datasetId": "YOUR_SCRAPER_DATASET_ID",
  "max_tokens": 512,
  "overlap_tokens": 64,
  "encoding": "cl100k_base",
  "split_on_heading_level": 2
}
```

***

### Output

````json
{
  "chunk_id": "9f2c1a77b4e3d018-1",
  "original_id": "https://docs.example.com/sdk/install",
  "source_url": "https://docs.example.com/sdk/install",
  "source_field": "markdown",
  "chunk_index": 1,
  "total_chunks": 4,
  "chunk_text": "Widget SDK > Quick start\n\n```python\nfrom widget import Client\n\nclient = Client(api_key=\"sk-test\")\n```",
  "token_count": 67,
  "token_count_method": "tiktoken:cl100k_base",
  "chunk_length_chars": 231,
  "heading_path": ["Widget SDK", "Quick start"],
  "section": "Widget SDK > Quick start",
  "block_kinds": ["code", "heading"],
  "content_hash": "9f2c1a77b4e3d018…",
  "overlap_tokens": 0,
  "cleaned_at": "2026-08-02T04:31:00+00:00"
}
````

***

### Cleaning

The cleaner is HTML-aware (stdlib parser, not regex) and removes the contents —
not just the tags — of `<script>`, `<style>`, `<noscript>`, `<template>`,
`<svg>` and HTML comments, so tracking snippets and CSS rules never reach your
index.

It **decodes** HTML entities rather than deleting them (`&pound;99` stays
`£99`, `&#8220;` becomes `“`), handles numeric and named forms, and **keeps
URLs**, converting `<a href>` into markdown links so retrieved passages remain
citable. HTML headings, lists, tables and `<pre>` blocks are converted to their
markdown equivalents so the chunker can see the structure.

***

### Pricing

**$0.0005 per KB of text processed** (rounded up), charged once per run,
computed from the bytes of text this Actor actually reads — not from whole-item
JSON, and not from how many chunks the text turns into. No chunking setting can
change your bill.

Items skipped because they had no readable text are not charged for.

***

### Chaining with other actors

Works after **Website Content Crawler**, **Cheerio Scraper**, or any Actor
emitting a text-like field. Pass its dataset ID as `datasetId`, or use it as an
Actor-to-Actor trigger target — the `resource.defaultDatasetId` chaining format
is handled too.

Common destinations for the output: Pinecone, Qdrant, Weaviate, pgvector,
Chroma, Milvus, or any LangChain / LlamaIndex ingestion step.

***

### Reliability notes

- The tokenizer's BPE tables are baked into the Docker image at build time, so
  no run depends on a network fetch to tokenize. If a tokenizer is somehow
  unavailable anyway, the run continues with a conservative character estimate
  and **every row is marked** `token_count_method: "estimated:chars"` so you can
  tell the difference.
- Dataset reads are paginated, retried with backoff on `429`/`5xx`, and
  time-limited. `X-Apify-Pagination-Total` is compared against what was read,
  and an incomplete read is reported in the log rather than passing silently.
- A failed dataset read stops the run before anything is charged.
- Credentials present in the Actor input are redacted before any fallback path
  can put them into the output dataset.

# Actor input Schema

## `datasetId` (type: `string`):

Apify dataset ID from a previous Actor run. Use this to chain directly after any scraper. All pages are read, not just the first.

## `data` (type: `array`):

Array of objects from a previous scraper. Each needs a text field — markdown, text, content, body, html and around 25 other common names are recognised automatically.

## `text_field` (type: `string`):

Name the field to read instead of relying on automatic detection. Useful when your scraper uses a custom field name, or when an item has several text fields and you want a specific one. The field actually used is reported on every output row as source\_field.

## `max_tokens` (type: `integer`):

Hard ceiling on tokens per chunk, measured with the encoding below. No emitted chunk ever exceeds this. 8191 is the input limit for OpenAI's text-embedding-3 models.

## `overlap_tokens` (type: `integer`):

Tokens of trailing context repeated at the start of the next chunk. Only whole sentences are carried, so a chunk never begins mid-word. Clamped to 50% of max\_tokens; 50% is fully supported. Overlap duplicates tokens, and the run log reports the resulting amplification so you can budget your embedding spend.

## `min_tokens` (type: `integer`):

Chunks below this are merged into the previous chunk rather than emitted as their own row. No text is ever dropped.

## `encoding` (type: `string`):

Which tokenizer to measure with. Pick the one your embedding model uses. Note that text-embedding-3-small and -large use cl100k\_base, not o200k\_base, despite being newer models. The BPE tables are bundled into the image, so no download happens at run time.

## `split_on_heading_level` (type: `integer`):

0 packs sections together to fill the budget. Set 1 to 6 to start a new chunk at every heading of that level or higher, so one section never bleeds into the next. Use 2 for typical documentation pages.

## `include_heading_context` (type: `boolean`):

Prepend the breadcrumb (for example "Widget SDK > Quick start") to each chunk's text, so an isolated passage still carries the context a retriever needs. The path is also emitted separately as heading\_path and section on every row.

## `preserve_links` (type: `boolean`):

Keep URLs in the text and convert HTML anchors to markdown links, so retrieved passages can still be cited. Turn off only if you want URLs stripped entirely.

## `drop_nav` (type: `boolean`):

Remove nav and menu contents. Script, style, noscript, template, svg and comment contents are always removed.

## Actor input object example

```json
{
  "max_tokens": 512,
  "overlap_tokens": 64,
  "min_tokens": 24,
  "encoding": "cl100k_base",
  "split_on_heading_level": 0,
  "include_heading_context": true,
  "preserve_links": true,
  "drop_nav": true
}
```

# Actor output Schema

## `chunks` (type: `string`):

Dataset of cleaned, chunked text items ready for embedding. See the dataset schema for the item fields.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("jalicia/rag-post-processor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("jalicia/rag-post-processor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call jalicia/rag-post-processor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jalicia/rag-post-processor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/rPRQKJP9bsGxsU9Ed/builds/1BCjpZulBdhwc1G1h/openapi.json