# Rag Embedding Generator (`labrat011/rag-embedding-generator`) Actor

Generate vector embeddings from text or chunked datasets using OpenAI or Cohere. Chains with RAG Content Chunker for end-to-end RAG pipelines. Outputs raw vectors ready for any vector database.

- **URL**: https://apify.com/labrat011/rag-embedding-generator.md
- **Developed by:** [mick\_](https://apify.com/labrat011) (community)
- **Categories:** AI, Agents, Automation
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RAG Embedding Generator

Apify Actor that generates vector embeddings from text or chunked datasets using OpenAI or Cohere. Chains directly with RAG Content Chunker or any crawler output. Outputs flat embedding objects with pass-through metadata, ready for any vector database. No vendor lock-in. MCP-ready for AI agent integration.

### Features

- Two embedding providers: OpenAI (text-embedding-3-small/large, ada-002) and Cohere (embed-english/multilingual-v3.0, light variants)
- Three input modes: single text, text list, or dataset chaining from any previous actor
- Pass-through metadata from RAG Content Chunker (chunk\_id, source\_url, page\_title, section\_heading)
- Batched API requests for throughput (up to 2048 texts per OpenAI call, 96 per Cohere call)
- Exponential backoff retry on rate limits and transient failures (3 attempts)
- API key marked `isSecret` -- never logged, never stored, never included in output
- Hardcoded API base URLs to prevent SSRF attacks
- Input validation and sanitization (key format checks, dataset ID regex, text length limits)
- Output: raw float arrays compatible with any vector DB (Pinecone, Qdrant, Weaviate, Chroma, etc.)

### Requirements

- Python 3.11+
- Apify platform account (for running as Actor)
- OpenAI or Cohere API key

Install dependencies:

```bash
pip install -r requirements.txt
```

### Configuration

#### Actor Inputs

Defined in `.actor/INPUT_SCHEMA.json`:

- `api_key` (string, required) -- your OpenAI or Cohere API key. Marked `isSecret`
- `provider` (string, optional) -- `"openai"` (default) or `"cohere"`
- `model` (string, optional) -- embedding model to use. Default: `"text-embedding-3-small"`
- `text` (string, optional) -- a single text string to embed, max 100,000 characters
- `texts` (array, optional) -- a list of text strings to embed, max 10,000 items
- `dataset_id` (string, optional) -- Apify dataset ID from a previous actor run (e.g., RAG Content Chunker). Takes priority over text/texts
- `dataset_field` (string, optional) -- field to read from each dataset item. Default: `"text"`. Supports dot notation
- `batch_size` (integer, optional) -- texts per API request. Default: 128. Max: 2048 (OpenAI) or 96 (Cohere)
- `include_text` (boolean, optional) -- include original text in output. Default: false

At least one of `text`, `texts`, or `dataset_id` must be provided, plus `api_key`.

#### Supported Models

| Provider | Model | Dimensions | Notes |
|----------|-------|-----------|-------|
| OpenAI | `text-embedding-3-small` | 1536 | Default. Cheapest, good quality |
| OpenAI | `text-embedding-3-large` | 3072 | Best quality, higher cost |
| OpenAI | `text-embedding-ada-002` | 1536 | Legacy, widely deployed |
| Cohere | `embed-english-v3.0` | 1024 | English-optimized |
| Cohere | `embed-multilingual-v3.0` | 1024 | 100+ languages |
| Cohere | `embed-english-light-v3.0` | 384 | Faster, smaller vectors |
| Cohere | `embed-multilingual-light-v3.0` | 384 | Faster, multilingual |

### Usage

#### Local (CLI)

```bash
APIFY_TOKEN=your-token apify run
```

#### Single Text Input

```json
{
  "api_key": "sk-your-openai-key",
  "provider": "openai",
  "model": "text-embedding-3-small",
  "text": "This is a sample text to embed into a vector representation."
}
```

#### Text List Input

```json
{
  "api_key": "sk-your-openai-key",
  "texts": [
    "First document to embed.",
    "Second document to embed.",
    "Third document to embed."
  ]
}
```

#### Dataset Chaining (from RAG Content Chunker)

```json
{
  "api_key": "sk-your-openai-key",
  "dataset_id": "abc123XYZ",
  "dataset_field": "text",
  "model": "text-embedding-3-small",
  "batch_size": 256
}
```

#### Example Output

Each embedding is a separate dataset item:

```json
{
  "index": 0,
  "embedding": [0.0123, -0.0456, 0.0789, "...1536 floats total"],
  "dimensions": 1536,
  "token_count": 12,
  "chunk_id": "a1b2c3d4e5f67890",
  "source_url": "https://example.com/page",
  "page_title": "Example Page",
  "section_heading": "Introduction"
}
```

A summary item is appended at the end:

```json
{
  "_summary": true,
  "total_embeddings": 42,
  "total_tokens": 8374,
  "provider": "openai",
  "model": "text-embedding-3-small",
  "dimensions": 1536,
  "processing_time": 3.241,
  "billing": {
    "total_embeddings": 42,
    "amount": 0.0126,
    "rate_per_embedding": 0.0003
  }
}
```

### Pipeline Position

This actor fills the embedding step in a standard RAG pipeline:

```
Crawl (Website Content Crawler, 101K+ users)
  -> Clean (optional preprocessing)
    -> Chunk (RAG Content Chunker)
      -> Embed (this actor)
        -> Store (Pinecone, Qdrant, Weaviate integrations)
```

#### Chaining with RAG Content Chunker

1. Run RAG Content Chunker on your text or crawler output
2. Copy the output dataset ID from the chunker run
3. Pass it as `dataset_id` to this actor
4. This actor reads each chunk, skips `_summary` rows, and passes through `chunk_id`, `source_url`, `page_title`, and `section_heading` metadata

The output vectors include all the metadata needed to store them in a vector database with proper source attribution.

### Architecture

- `src/agent/main.py` -- Actor entry point, input routing (text/texts/dataset), dataset loading, output
- `src/agent/embedder.py` -- Core embedding engine, OpenAI + Cohere API calls, batching, retry logic
- `src/agent/validation.py` -- Input validation, API key format checks, provider/model whitelist, sanitization
- `src/agent/pricing.py` -- PPE billing calculator ($0.0003/embedding)
- `skill.md` -- Machine-readable skill contract for agent discovery

### Security

- **API key handling**: Marked `isSecret` in input schema, validated for format only, never logged or stored, stripped from error messages
- **SSRF prevention**: Outbound requests hardcoded to `api.openai.com` and `api.cohere.ai` only -- no user-supplied URLs
- **Provider/model whitelist**: Only known provider+model combinations accepted, prevents arbitrary endpoint injection
- **Input sanitization**: Control characters stripped, dataset IDs and field names regex-validated, text length bounded
- **Error safety**: All error messages pass through `_sanitize_error()` to ensure API keys are never leaked in logs or output
- **No data retention**: Texts and embeddings exist only in memory during the run

### Pricing

Pay-Per-Event (PPE): **$0.0003 per embedding** ($0.30 per 1,000 embeddings).

This is the actor's platform fee only. You also pay the embedding provider (OpenAI or Cohere) directly via your own API key.

| Content Size | Approx. Embeddings | Actor Fee | Provider Fee (OpenAI 3-small) |
|-------------|-------------------|-----------|-------------------------------|
| Single blog post | 10-20 | $0.003-$0.006 | ~$0.001 |
| 10-page website | 50-100 | $0.015-$0.03 | ~$0.005 |
| 100-page docs site | 500-1,000 | $0.15-$0.30 | ~$0.05 |
| Large knowledge base | 5,000-10,000 | $1.50-$3.00 | ~$0.50 |

### Troubleshooting

- **"API key is required"**: Provide your OpenAI or Cohere API key in the `api_key` field
- **"Invalid OpenAI API key format"**: OpenAI keys start with `sk-` followed by alphanumeric characters
- **"Invalid model for provider"**: Check the supported models table above. Model names are case-sensitive
- **"No input provided"**: Supply at least one of `text`, `texts`, or `dataset_id`
- **"Text exceeds maximum length"**: Individual texts are limited to 100K characters. Use `texts` or `dataset_id` for bulk
- **"Invalid dataset\_id format"**: Must be alphanumeric with hyphens/underscores, 1-64 characters
- **"API key is invalid or expired"**: Your provider API key was rejected. Verify it in your OpenAI/Cohere dashboard
- **"Failed after 3 attempts"**: Transient API error. Try again, or reduce `batch_size` if hitting rate limits
- **Dataset errors**: Verify the dataset ID exists and the actor has access to it

### License

See `LICENSE` file for details.

***

### MCP Integration

This actor works as an MCP tool through Apify's hosted MCP server. No custom server needed.

- **Endpoint:** `https://mcp.apify.com?tools=labrat011/rag-embedding-generator`
- **Auth:** `Authorization: Bearer <APIFY_TOKEN>`
- **Transport:** Streamable HTTP
- **Works with:** Claude Desktop, Cursor, VS Code, Windsurf, Warp, Gemini CLI

**Example MCP config (Claude Desktop / Cursor):**

```json
{
    "mcpServers": {
        "rag-embedding-generator": {
            "url": "https://mcp.apify.com/?tools=labrat011/rag-embedding-generator",
            "headers": {
                "Authorization": "Bearer <APIFY_TOKEN>"
            }
        }
    }
}
```

AI agents can use this actor to generate vector embeddings from text using OpenAI or Cohere, embed chunked documents, and prepare data for vector database storage -- all as a callable MCP tool.

# Actor input Schema

## `api_key` (type: `string`):

Your embedding provider API key (OpenAI or Cohere). Never logged or stored beyond this run.

## `provider` (type: `string`):

Which embedding provider to use.

## `model` (type: `string`):

Which embedding model to use. OpenAI: 'text-embedding-3-small' (1536 dims, cheapest), 'text-embedding-3-large' (3072 dims, best quality), 'text-embedding-ada-002' (1536 dims, legacy). Cohere: 'embed-english-v3.0' (1024 dims), 'embed-multilingual-v3.0' (1024 dims), 'embed-english-light-v3.0' (384 dims), 'embed-multilingual-light-v3.0' (384 dims).

## `text` (type: `string`):

A single text string to embed. For bulk processing, use dataset\_id instead.

## `texts` (type: `array`):

A list of text strings to embed. For bulk processing from another actor, use dataset\_id instead.

## `dataset_id` (type: `string`):

ID of an existing Apify dataset (e.g., output from RAG Content Chunker or Website Content Crawler). Each item's text field will be embedded. Takes priority over text/texts inputs.

## `dataset_field` (type: `string`):

Which field to read from each dataset item. Default: 'text'. Supports dot notation for nested fields (e.g., 'metadata.content').

## `batch_size` (type: `integer`):

Number of texts to send per API request. Higher values are faster but use more memory. OpenAI supports up to 2048. Cohere supports up to 96.

## `include_text` (type: `boolean`):

Whether to include the original text in each output item alongside the embedding vector. Useful for debugging but increases output size.

## Actor input object example

```json
{
  "provider": "openai",
  "model": "text-embedding-3-small",
  "text": "The quick brown fox jumps over the lazy dog.",
  "dataset_field": "text",
  "batch_size": 128,
  "include_text": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "text": "The quick brown fox jumps over the lazy dog."
};

// Run the Actor and wait for it to finish
const run = await client.actor("labrat011/rag-embedding-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "text": "The quick brown fox jumps over the lazy dog." }

# Run the Actor and wait for it to finish
run = client.actor("labrat011/rag-embedding-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "text": "The quick brown fox jumps over the lazy dog."
}' |
apify call labrat011/rag-embedding-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,labrat011/rag-embedding-generator"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/CLgzlqLl3bFadCJ8E/builds/EAdQOmeheLg6YCMg5/openapi.json