RAG Freshness - Incremental Vector Updates

Update a RAG index only where a web page changed

RAG Freshness turns a public HTML page into stable chunks, stores a private baseline, and returns only the upsert and delete operations required after a change. Your workflow chooses the embedding model and vector database.

Use it to keep documentation assistants, support bots, internal search, product catalogs, policy libraries, and other retrieval-augmented generation systems current while avoiding repeated embedding calls for unchanged content.

Try it in 30 seconds

1. Click Try for free and keep the prepared https://example.com input.
2. Start the Actor. The first run creates a baseline and returns the page as actions.upsert chunks.
3. Run the same input again. An unchanged page returns no writes and reports the number of embeddingsAvoided.
4. Open the dataset. The table shows whether content changed and the number of upserts, deletes, and avoided embeddings.

You can also start from the public Reduce RAG embedding costs task. One successful page emits one page-analyzed event at the price shown above; failed pages do not emit that event.

A verified repeat-run record shows one unchanged chunk, zero vector writes, and one avoided embedding on https://example.com. It records one test, not a savings guarantee.

Importable n8n workflow

The public OmniAPI Apify examples repository includes a ready-to-import RAG Freshness n8n workflow, a step-by-step tutorial, and a verified repeat-run demonstration. The example prepares upserts, deletes, and embeddingsAvoided for a downstream vector database while keeping the commercial Actor implementation private.

What you get

Each successfully analyzed URL produces one dataset item with:

• actions.upsert: new or modified chunks with stable IDs, text, position, and hashes.
• actions.delete: chunk IDs that disappeared from the source.
• stats.embeddingsAvoided: unchanged chunks that do not need another embedding call.
• baselineCreated: whether this was the first stored snapshot for the URL.
• contentHash: a deterministic hash for auditing and idempotency.
• sourceId and processedAt: traceability fields for downstream automation.

The Actor is vector-database neutral. Connect the upserts and deletes to Pinecone, Qdrant, Weaviate, Supabase/pgvector, Elasticsearch, Milvus, Chroma, or your own storage layer.

Typical use cases

Incremental documentation updates

Schedule the Actor after a documentation release. Embed only actions.upsert, then delete the IDs in actions.delete. Unchanged sections retain their IDs, which helps prevent duplicate vectors and unnecessary churn.

RAG content-change monitoring

Run the same task daily or weekly and alert only when the returned actions are non-empty. This gives AI and knowledge-management teams a reviewable change feed without rebuilding the entire index.

Cost control for embedding pipelines

Track stats.embeddingsAvoided to estimate how many embedding requests the incremental workflow avoided. The Actor does not call an embedding provider, so you choose the model, provider, vector database, and retry policy.

Input example

{
  "urls": [{ "url": "https://example.com" }],
  "maxChunkChars": 1800,
  "overlapChars": 120,
  "includeUnchanged": false,
  "dryRun": false,
  "snapshotStoreName": "omniapi-rag-freshness-demo"
}

Run it again with the same URL and snapshotStoreName to compare against the saved baseline. Use a different store name for an independent environment, customer, or experiment.

Output example

{
  "success": true,
  "url": "https://example.com/",
  "baselineCreated": false,
  "changed": false,
  "upsertCount": 0,
  "deleteCount": 0,
  "embeddingsAvoided": 1,
  "stats": { "previousChunks": 1, "currentChunks": 1, "added": 0, "modified": 0, "removed": 0, "unchanged": 1, "embeddingsAvoided": 1 },
  "actions": { "upsert": [], "delete": [] },
  "processedAt": "2026-09-08T20:18:40.963Z"
}

Modified chunks use actions.upsert[].content for the extracted Markdown and retain their stable id when matched to the previous baseline. Treat the dataset schema as the source of truth when generating integrations.

Use with n8n, Make, Zapier, or the API

Call the Actor synchronously when a downstream workflow needs the dataset immediately, or start an asynchronous run and consume the dataset after completion. A common n8n flow is:

1. Schedule Trigger.
2. HTTP Request to the Apify synchronous dataset endpoint.
3. Split the returned items.
4. Embed each actions.upsert item.
5. Upsert vectors by stable ID.
6. Delete the IDs in actions.delete.

Keep your Apify token in a credential or secret environment variable, never directly in a shared workflow.

Pricing and cost control

The launch price is $0.02 per successfully analyzed page. Failed URL items are returned for diagnosis without charging the page-analyzed event. Your Apify platform usage is billed according to your own plan and the pricing shown before each run.

Start with a small URL list, inspect the output, and use Apify's maximum-charge controls before scheduling a large production job.

Limits and responsible use

• Public HTML pages only, up to 5 MB and 50 URLs per run.
• It does not bypass authentication, paywalls, CAPTCHA, robots restrictions, or anti-bot controls.
• Private-network and unsafe targets are blocked to reduce SSRF risk.
• It extracts page content; it does not execute a full browser-rendered application.
• Deterministic similarity should be evaluated on your content before allowing unattended destructive vector deletes.
• Website layouts and content can change. Monitor scheduled runs and review unexpected output.

Only process pages you are authorized to access and use. You remain responsible for source-site terms, privacy requirements, retention, and the downstream use of extracted content.

FAQ

Does it generate embeddings?

No. It returns the smallest set of chunk changes so you can use your preferred embedding model and vector database.

Where is the baseline stored?

In an Apify key-value store under the running user's account. Use snapshotStoreName to separate environments or workflows.

Will unchanged chunks preserve their IDs?

The matching strategy is designed to preserve stable IDs for unchanged or matched chunks. Test it with representative content before production use.

Can I use it for a single page?

Yes. One URL is a good way to validate the result before creating a scheduled task for a larger knowledge base.