Pricing

from $5.00 / 1,000 results

Try for free

Go to Apify Store

RAG Pipeline

Try for free

One-click RAG pipeline: chunks text, generates embeddings, and stores vectors in Pinecone or Qdrant. Provide your content and API keys -- the orchestrator handles the rest.

Pricing

from $5.00 / 1,000 results

Rating

0.0

(0)

Developer

mick_

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Unified RAG Pipeline

One-click RAG pipeline: chunk text, generate embeddings, and store vectors — all in a single actor.

Merges 4 separate actors (rag-content-chunker, rag-embedding-generator, rag-vector-store-writer, rag-pipeline) into one unified pipeline. No sub-actor calls — everything runs in-process for maximum reliability and simplicity.

Features

3 chunking strategies: recursive (paragraph→sentence→word), markdown (header-aware), sentence (boundary-only)
Token-aware splitting: Uses tiktoken with cl100k_base encoding (same as text-embedding-3-small/large)
2 embedding providers: OpenAI (text-embedding-3-small, text-embedding-3-large, ada-002) and Cohere (v3 models)
2 vector DB providers: Pinecone and Qdrant Cloud (auto-creates collections)
Exponential backoff retry: All API calls retry on rate limits and server errors
SSRF protection: Hardcoded API endpoints, validated Qdrant URLs, blocked private IPs
File URL support: Fetch .txt, .md, .html, and .pdf files over HTTPS
Dataset chaining: Read from any Apify crawler/crawler dataset
Pay-per-event pricing: $5.00 per 1,000 chunks processed end-to-end

Input

Field	Type	Default	Required
`text`	string	—	No*
`source_dataset_id`	string	—	No*
`file_url`	string	—	No*
`file_urls`	string[]	—	No*
`chunking_strategy`	enum	`recursive`	No
`chunk_size`	int	512	No
`chunk_overlap`	int	64	No
`embedding_api_key`	string	—	Yes
`embedding_provider`	enum	`openai`	No
`embedding_model`	string	`text-embedding-3-small`	No
`embedding_batch_size`	int	128	No
`vector_db_api_key`	string	—	Yes
`vector_db_provider`	enum	`pinecone`	No
`index_name`	string	—	Yes
`qdrant_url`	string	—	If Qdrant
`pinecone_namespace`	string	`""`	No
`qdrant_distance_metric`	enum	`Cosine`	No

*One content source required: text, file_url, file_urls, or source_dataset_id.

Pricing

This actor uses pay-per-event (PPE) pricing.

$5.00 per 1,000 chunks processed end-to-end. This single fee replaces the previous 3 separate charges (chunker + embedder + writer) and the orchestrator fee. You only pay for chunks actually produced.

Actor Start: $0.00005 per run (charged once per run)
Platform usage (proxy/compute): billed to your account

How It Works

Input text → [Chunker] → chunks → [Embedder] → vectors → [Vector DB Writer] → Pinecone/Qdrant
                    All in-process, single actor run

PDF to RAG Markdown Chunks for Embeddings

awesome_highboy/docforge

Convert PDFs into token-bounded Markdown chunks for RAG, embeddings, and vector databases (Pinecone, Chroma, Weaviate, Qdrant). Set maxTokens + overlap; get clean chunks with page number, token count, and SHA-256 content hash for dedup. JSON dataset ready for any LLM pipeline.

Adam

Rag Pipeline Manager Mcp

bronze_quarterback/rag-pipeline-manager-mcp

Segun Zubair

Website to Text & Markdown — AI / RAG Content Crawler

inexhaustible_glass/rag-website-crawler

Scrape any website into clean text & Markdown with RAG-ready chunks and token counts for LLMs, vector databases (Pinecone, Qdrant) and AI chatbots. Also extracts linked PDF/Word/Excel. Anti-block, robots.txt-aware. Website-to-text for beginners, full RAG pipeline for pros. CPU only, no API key.

Hitman studio

AI Data Pipeline — Crawl, Chunk & Export to Vector DB

ozapp/ai-data-pipeline

Crawl any website, extract clean text, split into chunks with quality scoring, and export to JSON, Pinecone, or Qdrant. Built for RAG pipelines and AI training data. Includes language detection, content type classification, and token counting.

Ozapp

Rag Vector Store Writer

labrat011/rag-vector-store-writer

Apify Actor that writes embedding vectors to Pinecone or Qdrant vector databases. Chains directly with RAG Embedding Generator output or accepts raw vectors with metadata. Handles batching, retries, collection creation, metadata mapping, and ID generation. Bring your own vector DB API key.

mick_

Tender RAG Chunker — Text Chunks for LLM

adobeflex/tender-rag-chunker

Chunk tender text for RAG/embeddings with stable ids and metadata.

Yahor

RAG Ingestor: Multi-Source Chunks for Vector DBs

aitoolbreakdown/atb-rag-ingestor

Ingest URLs, sitemaps, and GitHub READMEs into uniform chunks with titles, source URLs, and stable IDs. Ready to push straight into Pinecone, Weaviate, or any RAG pipeline.

AI Tool Breakdown

Rag Embedding Generator

labrat011/rag-embedding-generator

Generate vector embeddings from text or chunked datasets using OpenAI or Cohere. Chains with RAG Content Chunker for end-to-end RAG pipelines. Outputs raw vectors ready for any vector database.

mick_

Website & Docs to Markdown + RAG Chunks

awesome_highboy/aeo-rag-ready-content-structurer

Turn websites & docs into clean Markdown plus token-bounded, embeddings-ready RAG chunks (heading lineage + sha256) ready for Pinecone, Weaviate, Qdrant or pgvector. Optional no-hallucination field extraction and AEO mode (FAQ, answer-first, llms.txt, citations). Robots honored; ownership required.

Adam

YouTube Transcript API - RAG Chapters, Summary & Chunks

webdatalabs/youtube-transcript-rag

Turn any YouTube video, playlist, or channel into RAG-ready data: clean transcript, timestamped segments, AI chapters, summary, key quotes, and embeddings-ready chunks. Built for AI agents and RAG pipelines.