We have covered the fundamentals in part I, and we will walk through how to implement these ideas into an actual RAG system, from a design aspect.

This is a 4-week rapid prototype we built for a client demonstrating our capabilities to quickly develop a use case with IBM watsonx.

Core features

Mockups

Wireframes and markup illustrations were instrumental in aligning with the client team on the final deliverables for the POC. The design and approach underwent several iterations before we reached consensus on the final user interface.

We use the Carbon Design System from IBM to build the mockups for a more professional touch and feel. There’s also a React component library for quick visual explorations.

Finding information

We want to create a similar experience with Google, where users will be able to use it as a normal search engine, but with AI-generated response as a summary to user’s question.

As we know, responses from LLM may contain hallucination, and we should always verify the underlying data sources. By displaying all relevant sections in a list, we can examine if there’s any irrelevant results, and exclude them if we want a more concise response from LLM.

Deep-dive

When user clicks on the section in retrieved results, they will be directed to the deep-dive page, where we implement an auto-scroll for them, based on page number extracted in the meta data, so they will be looking at the exact page number of the section, instead of from page 1. This is implemented with query parameters from the URL, which got carried over to the next page.

On the right, we can perform another LLM call by selecting full document or only specific pages, if we want a more overall summary, instead of only specific sections from the previous search page.

Managing files

A separate tab is created for users to upload documents for ingestion and extraction. Users can see a quick overview of all uploaded documents, as well as deleting obsolete ones. When they delete documents, a cleanup action on the vector database is also performed to wipe out the index.

Backend API

Authentication

• Accepts OAuth2PasswordRequestForm; verifies credentials against Redis hashes seeded at startup
• Issues a signed HS256 JWT (sub = username); all other endpoints require Bearer token via OAuth2PasswordBearer
• Stateless — no session storage; token expiry relies on JWT TTL

Document ingestion

• Accepts one or more multipart/form-data files; streams each directly to MinIO via upload_fileobj
• Generates a UUID-prefixed object key ({uuid}_{filename}) per file to avoid collisions
• Pushes each object key onto the Redis job_queue list via RPUSH — decoupling upload latency from processing time

Hybrid search

• Runs two Elasticsearch queries concurrently via asyncio.gather:
  BM25: multi_match across combined_text, content, source, headings, content_with_context
  Semantic: cosine similarity script score against combined_text_vector using a watsonx-generated query embedding
• Both queries filter strictly by username (keyword term); results are tenant-isolated per user
• Embedding call offloaded to a thread pool via run_in_executor to avoid blocking the async event loop
• Results merged with Reciprocal Rank Fusion (RRF, k=60); top-10 documents fetched via mget and returned

Streaming summarization

• Accepts prompt, model_id, and context (retrieved chunks) from the UI
• Exchanges a watsonx API key for a short-lived IAM Bearer token before each request
• Proxies the watsonx generation_stream endpoint as a text/event-stream StreamingResponse; tokens reach the browser as they are generated
• LLaMA 3.3 70B prompt formatted with Llama 3 chat tokens (<|begin_of_text|>, headers, <|eot_id|>)

Task lifecycle

• store_task: writes task metadata to a Redis hash keyed {username}:{task_id} and a reverse-lookup hash {task_id}-username
• task_status: reads {id}-status and {id}-error keys set by the job worker; returns running, ready, or failed
• get_tasks: scans Redis with KEYS {username}:* and returns all task hashes for the user
• delete_task : removes Redis keys and issues an Elasticsearch delete_by_query on source == task_id to purge all indexed chunks

PDF streaming

• Fetches the raw file body from MinIO and returns it as a chunked application/pdf StreamingResponse
• Used by the in-browser PDF viewer to render source documents alongside search results

Job worker

Queue consumer loop

• Blocks on BLPOP job_queue 0: zero CPU usage between jobs; wakes instantly when a key arrives
• Single-threaded by design: one document processed at a time, preventing GPU/memory contention on constrained hardware
• Crash-resistant: outer try/except logs the full traceback, marks the job failed in Redis, and resumes listening; the worker never exits

Document parsing

• Fetches the raw bytes from MinIO and wraps them in a DocumentStream for in-memory processing (no temp files)
• Uses Docling DocumentConverter with PdfPipelineOptions: table structure extraction enabled, OCR disabled (native text PDFs)
• Supports PDF, DOCX, PPTX, HTML, image, AsciiDoc, and Markdown inputs through a single converter
• HierarchicalChunker splits the converted document into heading-aware chunks preserving document structure

Contextual enrichment

• Adjacent chunks sharing the same heading are merged before embedding, which reduces over-segmentation of tightly related content
• For each merged chunk, a contextualization prompt is sent to llama-3-3-70b-instruct: the full document markdown + the chunk text → a short retrieval-optimized context sentence
• IAM token cached in-process with a 50-minute refresh interval to avoid redundant token-exchange calls
• LLM errors are caught silently: chunks are indexed with an empty context rather than failing the entire job

Embedding

• Four 1024-dimensional vectors generated per chunk via intfloat/multilingual-e5-large hosted on watsonx.ai:
  content_vector: raw chunk text
  content_with_context_vector: LLM context prepended to chunk text
  headings_vector: section heading text
  combined_text_vector: source key + heading + context + content (broadest semantic signal)
• Input truncated to 512 tokens server-side; input_text not returned to minimize payload size

Elasticsearch indexing

• Index auto-created on first job with explicit mappings: dense_vector (dims=1024, cosine similarity) for all vector fields; text for full-text fields; keyword for source and username
• Existing chunks for the same source file are deleted via delete_by_query before re-indexing — supports clean re-processing of updated documents
• Chunks written in batches of 100 using the Elasticsearch helpers.bulk API for throughput
• Job status written to Redis as {key}-status = ready on success; UI polls /task_status to surface completion

Cloud architecture

This post walks through designing and building a self-hosted platform prototype (in 4 weeks), that takes an audio file, transcribes it with Whisper, and uses a locally-deployed large language model to produce a structured, timestamp-anchored meeting summary, and streamed in real time to a React frontend. Every component runs in a container; no data leaves your infrastructure.

1 · Overall architecture & data flow

The stack is 5 containers wired together on a private bridge network, orchestrated with podman compose (or docker compose).

• stt-ui (Node 20 + React): Single-page app with upload, playback, transcript view, and streaming summary funcions
• stt-api (Python 3.11 + FastAPI): REST API, JWT auth, MinIO orchestration, LLM gateway
• stt-worker (python 3.11 + Celery): Async transcription via faster-whisper (CPU, int8). mem_limit: 16g prevents runaway jobs from OOMing the host.
• redis: Celery broker + result backend + user credentials + transcript cache (also seeding the first user into Redis on first boot via redis-cli HSET)
• minio: S3-compatible object store with raw uploads and processed MP3s in separate buckets

Audio upload & transcription: two-phase flow

Phase 1 : File storage & job dispatch
User selects an audio file in the browser. Before sending it to the server, the browser creates a temporary Audio object from the file to read its duration without server round-trip needed for that metadata. The file is then uploaded to the API, which saves it to MinIO’s raw bucket under a randomly generated UUID key and immediately queues a background transcription task. The API responds in milliseconds; no transcription has happened yet.

Phase 2 : Metadata persistence
Only after MinIO confirms the file was stored successfully does the client make a second call to record the task metadata in Redis, including filename, file size, duration, and the UUID . This two-step sequence is intentional: if the file storage fails, no orphan metadata record is ever created. The two operations are kept independent so each can fail or be retried without affecting the other.

Background transcription
Meanwhile the Celery worker picks up the queued task. It fetches the raw file from MinIO, converts it to MP3, writes the MP3 back into a separate processed bucket, then runs faster-whisper to produce timestamped segments. When transcription completes, the results are written to Redis and the task status flips from running to ready. The UI polls the status endpoint every 15 seconds and fetches the transcript once it sees ready.

On-prem LLM inference

The /summarize flow calls any OpenAI-compatible endpoint via LiteLLM. To run fully on-prem, point the env vars at an Ollama instance and set the model prefix to ollama/.

Whisper model

An optimized faster-whisper version of the original Whisper released by OpenAI. Note that depending on your hardware specifications, you may need to select suitable model size and precision accordingly. For example, if you are running on a MacBook Pro with 16gb of RAM (same as me), then you need to go with a medium model instead of large-V2 and running on CPU. Note that having a dedicated GPU significantly speeds up the transcription process.

2 · API design

FastAPI auto-generates OpenAPI docs at http://localhost:8082/docs (Swagger UI) and /redoc. All routes except POST /token require a Bearer JWT.

Authentication

The API uses FastAPI’s built-in OAuth2PasswordBearer. POST /token fetches the user hash from Redis, verifies the password, and signs a JWT. Every protected route injects current_user via Depends(get_current_user).

3 · Structured output with LiteLLM + Instructor

Rather than asking the LLM to produce free-form markdown, Instructor enforces a typed Pydantic schema and streams partial results as they arrive.

The schema

Field descriptions are serialized into the JSON schema sent to the model, giving the LLM explicit format guidance without extensive prompt engineering:

class BulletPoint(BaseModel):
    content: str
    reference_timestamp: str = Field(
        description="Timestamp in HH:MM:SS format (e.g. '00:12:34')"
    )

class SummaryResponse(BaseModel):
    bullet_points: list[BulletPoint] = Field(
        description="Key points, each with the timestamp where it appears"
    )

SUMMARIZE_SYSTEM_PROMPT = """You are an AI assistant specializing in writing detailed meeting summaries.
Your task is to provide a detailed and relevant summary based on the transcript provided.
For each key point, identify the timestamp from the transcript (in HH:MM:SS format) where it was mentioned.
Do not copy sentences verbatim from the transcript — synthesize and rephrase.
"""

Streaming partial results

create_partial yields progressively more complete SummaryResponse instances as tokens arrive. The bullet_points list grows one item at a time:

async def run_summary_stream(messages, model):
    async for partial in instructor_client.chat.completions.create_partial(
        model=model, # openai/gpt-oss-120b
        messages=messages,
        api_key=WATSONX_API_KEY,
        api_base=WATSONX_URL,
        project_id=WATSONX_PROJECT_ID,
        response_model=SummaryResponse,
        max_tokens=2000,
        stream=True
    ):
        yield partial

Filtering incomplete partials

During streaming, a timestamp arrives character-by-character ("00" → "00:0" → "00:01:07"). A regex gate holds each bullet back until its timestamp is fully formed, and an event is only emitted when the list actually grows, suppressing the flood of empty partials:

TS_RE = re.compile(r"^\d{2}:\d{2}:\d{2}$")

last_count = 0
async for partial in run_summary_stream(messages, model):
    points = [
        {"content": bp.content, "reference_timestamp": bp.reference_timestamp}
        for bp in (partial.bullet_points or [])
        if bp.content and TS_RE.match(bp.reference_timestamp or "")
    ]
    if len(points) > last_count:   # only emit when a new bullet completes
        last_count = len(points)
        yield f"data: {json.dumps({'type':'partial','bullet_points':points})}\n\n"

yield f"data: {json.dumps({'type': 'complete'})}\n\n"

4· Frontend (audio sync)

The transcript view synchronizes a live audio player with both the raw transcript and the AI-generated summary. Three interactions are wired together:

Real-time transcript highlighting

An HTML5 Audio object is created from a blob URL fetched from the API. A timeupdate listener continuously writes the current playback position to state. Each transcript row compares that value against its [start, end] window — the matching row turns bold and auto-scrolls into view:

a.addEventListener("timeupdate", () => setCurrentAudioTime(a.currentTime));

// in JSX — each transcript row:
const isActive = currentAudioTime >= item.start && currentAudioTime < item.end;
if (isActive) element.scrollIntoView({ behavior: "instant" });

Click-to-seek from transcript and summary

Clicking any transcript row seeks the audio to that segment’s start time. Clicking the clock icon on a summary bullet point does the same, but anchored to the LLM-produced timestamp, parsed from HH:MM:SS to seconds:

// transcript row click
const handleTranscriptClick = (seconds) => {
    a.currentTime = seconds;
    a.play();
    setButtonName("Pause");
};

// summary bullet — parse HH:MM:SS
const [h, m, s] = bp.reference_timestamp.split(":").map(Number);
const seconds = h * 3600 + m * 60 + s;

Client-side transcript export

The download button formats each segment as [HH:MM:SS-HH:MM:SS]: text, creates a Blob, triggers a synthetic anchor click, then immediately calls revokeObjectURL to release the memory:

const text = transcription
    .map(({ start, end, text }) =>
        `[${toTime(start)}-${toTime(end)}]: ${text}`)
    .join("\n");

const blob = new Blob([text], { type: "text/plain" });
const link = document.createElement("a");
link.href = URL.createObjectURL(blob);
link.download = `transcription_${taskId}.txt`;
link.click();
URL.revokeObjectURL(link.href);  // clean up blob memory

Key takeaways

• Decoupled transcription: Celery + Redis keeps the API non-blocking; Whisper loads once per worker process via worker_process_init.
• Two-phase upload: object storage write and metadata persistence are separate calls, keeping them atomic and independently retryable.
• Structured streaming: instructor’s create_partial + regex gate delivers clean, complete bullet points one at a time, never a half-formed timestamp.
• Provider-agnostic LLM: LiteLLM prefix routing targets Ollama, watsonx.ai, Azure OpenAI, or OpenAI with a single env var.
• Frontend without a state library: React Hooks + Context handles auth, language, sort/filter/pagination, and real-time streaming cleanly at this scale.
• Audio synchronization: timeupdate + segment windows provide real-time transcript highlighting; both the transcript and AI summary seek the audio on click.
• Zero external dependencies: MinIO, Redis, and Ollama all run in containers on the same host. No data leaves the machine.

Getting a RAG demo working takes an afternoon. Getting it to work reliably at enterprise scale is a different challenge entirely. This post walks through the four components that make up a production RAG system, the tradeoffs that matter in each one, and the tools we’ve found most useful along the way.

The four components are:

1. Ingestion & Extraction: reading your documents and turning them into searchable pieces
2. Retrieval: finding the most relevant pieces for any given question
3. Generation: using an AI model to craft an answer from those pieces
4. Evaluation: measuring quality so you can keep improving

Let’s go through each one.

1. Ingestion and Extraction

Before an AI can answer questions from your documents, those documents need to be read, understood, and organized. This sounds straightforward until you’re dealing with a mix of scanned PDFs, Word files, PowerPoint decks, spreadsheets, and web pages. All with different layouts, tables, and embedded images.

Parsing documents with Docling

Docling is an open-source library from IBM Research that handles this diversity well. You point it at a document and it figures out the structure and hands you back clean, organized content.

Under the hood, Docling routes each file type to the right parser: PDFs go through a specialized backend that understands layouts, while Word and Excel files go through OpenXML. For tricky content like scanned pages or charts, it applies OCR and vision models. The end result is a structured document you can export to Markdown, JSON, or HTML.

For most RAG pipelines, Markdown export is the sweet spot. It keeps headings, lists, and tables intact without the overhead of more complex formats. Docling also integrates directly with LangChain and LlamaIndex, so it fits cleanly into most existing pipelines.

Breaking documents into chunks

A language model can only read so much text at once. After parsing, you need to break documents into smaller pieces called chunks, that can be stored, searched, and passed to the model efficiently.

There’s no one-size-fits-all approach here. The right strategy depends on your documents and your questions.

Fixed-size chunking is the simplest approach: split every document into pieces of a set length (say, 512 tokens), with a small overlap between adjacent chunks so context isn’t lost at the edges. It’s quick to set up and works well as a starting point. The downside is that finding the right size takes trial and error. A chunk that’s too small loses context; one that’s too large buries the relevant sentence. Tools like IBM watsonx AutoRAG can automate this search, which saves a lot of manual tuning.

Hierarchical chunking is smarter: instead of counting characters, it respects the document’s own structure. Section breaks become chunk breaks. This produces chunks that align with how the document was actually written, which tends to mean better search results. Docling’s HybridChunker does this out of the box and can also merge small adjacent chunks so nothing ends up too short.

Custom chunking is for when you know exactly what your documents look like. Legal contracts, product data sheets, and internal knowledge-base articles often follow a predictable structure. Writing a custom parser for that structure gives you the most precise and well-labeled chunks, at the cost of needing to maintain it as document formats evolve.

Making chunks easier to find

Two enrichment steps make a big difference in retrieval quality:

• Contextual summaries add a brief summary of the surrounding section to each chunk. A chunk on its own might be too narrow to match a broad question; the summary gives it extra surface area for search.
• Metadata store the page number, document title, section heading, and source alongside each chunk. This lets you filter results at query time. For example, restricting a search to documents from the latest version, a specific department or date range.

2. Retrieval

With documents ingested and indexed, the retrieval layer’s job is to find the handful of chunks most relevant to each question. Getting this right matters more than almost anything else; even a perfect AI model can’t give a good answer if it’s given the wrong context.

Why hybrid search wins

There are two main ways to search a document index:

• Keyword search (BM25) looks for documents that contain the exact words from your query. It’s fast, reliable for precise terms like product codes or proper nouns, and has been the backbone of enterprise search for decades. Its weakness is that it doesn’t understand meaning so it can miss relevant documents that use different words to say the same thing.
• Semantic search uses AI embeddings to compare the meaning of a query to the meaning of each chunk. It handles synonyms and paraphrasing well, but can be thrown off when exact terminology matters. Searching for “Section 4.2(b)” semantically is unlikely to find exactly what you need.

Hybrid search runs both in parallel and combines the scores. The result set captures both precise matches and conceptually similar content, and it consistently outperforms either approach used alone. There are two common ways to merge the scores: a tunable weighted average, or the simpler Reciprocal Rank Fusion (RRF) algorithm, which requires no tuning and is a solid default.

OpenSearch as the retrieval engine

OpenSearch is a well-proven open-source search engine that supports both BM25 and vector search natively, making it a natural fit for hybrid RAG.

When a query arrives, a coordinating node fans it out to all data shards simultaneously to run keyword and semantic searches in parallel. Results come back from each shard, are normalized and merged in the coordinating node, and then the top matches are fetched and returned. The parallel architecture keeps response times low even over large document collections.

Reranking: a second pass for precision

Retrieval often casts a wide net. Before passing 50 results to the model, a reranker does a more careful job of sorting them. A cross-encoder model reads the query and each chunk together (rather than independently) and scores how well they match. You then pass only the top 3–5 results to the model, which keeps context windows small and answers focused.

3. Generation

Once you have the right context, the generation step is where the AI model actually writes the answer. This stage has two key decisions: which model to use, and how to control what it gives back.

Choosing a model

• Proprietary APIs such as ChatGPT or Gemini has the highest capability, and can be chosen when data can leave your environment.
• Open-weight models like Granite or Llama are self-hosted, and particularly good for use cases with data-residency requirements.
• Fine-tuned models shine in specialized domains with unique terminology or tone.

For most enterprise teams, open-weight models deployed on their own infrastructure strike the right balance between capability and control.

Getting structured output

Free-form prose is useful for some use cases, but many enterprise workflows need the AI to return data in a predictable, machine-readable format. Examples include extracting fields from a contract, classifying a support ticket, or filling a form from an email.

The Instructor library makes this straightforward. You define the shape of the output you want using a Pydantic model, and Instructor ensures the AI returns exactly that, with automatically retrying if the output doesn’t validate.

The example below shows how to pull structured information from a sentence:

import os

import litellm
from litellm import completion
from pydantic import BaseModel, Field

import instructor
from instructor import Mode

litellm.drop_params = True  # watsonx.ai doesn't support `json_mode`

os.environ["WATSONX_URL"] = "https://us-south.ml.cloud.ibm.com"
os.environ["WATSONX_API_KEY"] = ""
os.environ["WATSONX_PROJECT_ID"] = ""
# Additional options: https://docs.litellm.ai/docs/providers/watsonx


class Company(BaseModel):
    name: str = Field(description="name of the company")
    year_founded: int = Field(description="year the company was founded")


client = instructor.from_litellm(completion, mode=Mode.JSON)

resp = client.create(
    model="watsonx/meta-llama/llama-3-8b-instruct",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": """\
Given the following text, create a Company object:

IBM was founded in 1911 as the Computing-Tabulating-Recording Company (CTR), a holding company of manufacturers of record-keeping and measuring systems.
""",
        }
    ],
    project_id=os.environ["WATSONX_PROJECT_ID"],
    response_model=Company,
)

print(resp.model_dump_json(indent=2))
"""
{
  "name": "IBM",
  "year_founded": 1911
}
"""

This pattern works especially well with IBM watsonx, which adds enterprise governance, audit logging, and data-residency controls on top of the model inference layer.

A few prompt engineering tips that consistently help:

• Be explicit in the system prompt about the model’s role and the format you expect.
• Keep temperature low (0.0–0.2) for factual questions where you want consistent, reliable answers.
• Set a max token limit so responses stay concise and costs stay predictable.

Agentic approach

Another interesting way is to build the retrieval system into a tool and connect it to agentic platforms like IBM watsonx Orchestrate, either via OpenAPI or MCP server, and have the AI agent reason and decide the query term to retrieve knowledge based on chat history. The advantage of the agentic approach is that if retrieved result is not relevant, the AI agent can reason and fix the query term to retrieve better results in a loop.

4. Evaluation

Building a RAG system without measuring it is like driving without a speedometer. You need to know whether changes are actually improvements. And if something breaks, you need to find out before your users do.

There are two complementary ways to evaluate a RAG pipeline:

Algorithm-based checks: fast and cheap

These metrics compare tokens between the generated answer and a reference answer, with no AI calls required. That makes them fast enough to run on every deployment.

These are best used as automated gates in a CI/CD pipeline — if the score drops below a threshold, flag it for review before it reaches production.

LLM-as-a-judge: deeper quality checks

For a more nuanced view, you can use a second AI model to evaluate the outputs. This captures things that token counts miss — like whether the answer actually makes sense given the question.

LLM-as-a-judge evaluation typically covers two areas:

Did we retrieve the right context?

• Did the relevant document appear in the results at all? (hit rate)
• Was it near the top? (reciprocal rank, NDCG)
• Was most of what we retrieved actually useful? (precision, context relevance)

Did we generate a good answer?

• Is every claim backed by the retrieved context? (faithfulness)
• Does the answer actually address what was asked? (answer relevance)
• How close is it to the ideal answer? (answer similarity)

LLM-as-a-judge takes longer and costs more, so a practical approach is to run algorithm-based checks on every release and LLM-as-a-judge periodically against a curated benchmark set.

Closing thoughts

If you’re starting from scratch, this is the order of impact we’d recommend focusing on:

1. Get parsing right first: downstream quality is limited by the quality of your extracted text. Spend time configuring Docling for your document types before touching anything else.
2. Pick a chunking strategy that fits your content: hierarchical chunking works well for structured documents; custom chunking gives the best results when you control the format.
3. Use hybrid search: it consistently outperforms either keyword or semantic search alone and takes only moderate extra effort to set up.
4. Add structured output: if your use case needs machine-readable responses, Instructor + Pydantic removes a lot of fragile prompt engineering.
5. Automate evaluation from day one: it’s much easier to maintain quality when you’re catching regressions automatically rather than reactively.

The tools covered above are a solid starting point for building something that scales.

Part II of this post covers the design aspect of such enterprise RAG system.

IBM watsonx Orchestrate exposes a REST API that lets you integrate AI agents directly into your applications and workflows. Whether you are building an internal automation pipeline or a customer-facing product, the API gives you programmatic access to agent orchestration, streaming responses, and an OpenAI-compatible chat completions endpoint.

This guide walks through the four essential operations you need to get up and running: obtaining an authentication token, listing your agents, sending a message with streaming, and calling the chat completions endpoint. All examples are shown in both cURL and Python.

Prerequisites

Set these environment variables once and every snippet below will work without modification.

export IBM_API_KEY="your_api_key_here"
export WXO_REGION="your_region"
export WXO_INSTANCE_ID="your_instance_id"

Step 1 : Obtain an IAM Access Token

POST https://iam.cloud.ibm.com/identity/token

All API calls are authenticated with a short-lived IBM IAM bearer token. Exchange your API key for a token before making any other call. Tokens are valid for 3600 seconds (1hr) and must be refreshed periodically.

curl --location 'https://iam.cloud.ibm.com/identity/token' \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --header 'Accept: application/json' \
  --data-urlencode 'grant_type=urn:ibm:params:oauth:grant-type:apikey' \
  --data-urlencode 'apikey=IBM_API_KEY'

import requests, os

response = requests.post(
    "https://iam.cloud.ibm.com/identity/token",
    headers={"Content-Type": "application/x-www-form-urlencoded"},
    data={
        "grant_type": "urn:ibm:params:oauth:grant-type:apikey",
        "apikey": os.getenv("IBM_API_KEY")
    }
)

print(response.json())

{
  "access_token": "eyJhbGciOiJIUz......",
  "refresh_token": "not_supported",
    "ims_user_id": 15031254,
    "token_type": "Bearer",
    "expires_in": 3600,
    "expiration": 1774596742,
    "scope": "ibm openid"
}

Note: The response includes access_token, expires_in (3600), and token_type: Bearer. Cache and reuse the token across calls until it expires.

Step 2 : List Available Agents

GET /instances/{instance_id}/v1/orchestrate/agents

Retrieve all agents provisioned in your instance. The id field from this response is required for all subsequent calls.

curl 'https://api.$WXO_REGION.watson-orchestrate.cloud.ibm.com/instances/$WXO_INSTANCE_ID/v1/orchestrate/agents' \
  --header 'Accept: application/json' \
  --header 'Authorization: Bearer YOUR_ACCESS_TOKEN'

region = os.getenv("WXO_REGION")
instance_id = os.getenv("WXO_INSTANCE_ID")

response = requests.get(
    f"https://api.{region}.watson-orchestrate.cloud.ibm.com"
    f"/instances/{instance_id}/v1/orchestrate/agents",
    headers={"Authorization": f"Bearer {token}"}
)

print(response.json())

[
    {
        "id": "4ba981cb...",
        "tenant_id": "b319196991014918aeee542a3527dbd7...",
        "name": "VoiceAgentTest_95874q",
        "description": "testing voice agent",
        "instructions": "",
        "tools": [],
        "collaborators": [],
        "knowledge_base": [],
        "llm": "groq/openai/gpt-oss-120b",
        "style": "default",
        "supported_apps": [],
        "created_by": "IBMid-696000C5O6",
        "created_on": "2026-03-20T06:22:49.593367Z",
        "updated_by": "IBMid-696000C5O6",
        "updated_at": "2026-03-25T08:33:10.818168Z",
        "deleted_by": null,
        "deleted_at": null,
        "environments": [],
        ...
    }
]

Note: Each agent object includes its id, name, llm model binding, tools, collaborators, and audit timestamps. Store the id of the agent you want to interact with.

Step 3 : Send a Message with Streaming

POST /instances/{instance_id}/v1/orchestrate/runs?stream=true

The /runs endpoint starts an agent run and streams server-sent events back to the client — ideal for real-time UIs and long-running tasks.

curl 'https://api.$WXO_REGION.watson-orchestrate.cloud.ibm.com/instances/$WXO_INSTANCE_ID/v1/orchestrate/runs?stream=true&stream_timeout=120000&multiple_content=true' \
  --header 'IAM-API-KEY: IBM_API_KEY' \
  --header 'Authorization: Bearer YOUR_ACCESS_TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{
    "message": { "role": "user", "content": "Hello, how can you help me?" },
    "agent_id": "YOUR_AGENT_ID"
  }'

response = requests.post(
    f"https://api.{region}.watson-orchestrate.cloud.ibm.com"
    f"/instances/{instance_id}/v1/orchestrate/runs",
    params={"stream": "true", "stream_timeout": "120000", "multiple_content": "true"},
    headers={
        "IAM-API-KEY":   os.getenv("IBM_API_KEY"),
        "Authorization": f"Bearer {token}",
        "Content-Type":  "application/json"
    },
    json={
        "message": {"role": "user", "content": "Hello, how can you help me?"},
        "agent_id": agent_id
    },
    stream=True
)
response.raise_for_status()

for line in response.iter_lines():
    if line:
        print(line)   # process each SSE event

Raw: b'{"id": "177459...", "event": "run.started", "data": {"run_id": "6001dfb7...", "agent_id": "4ba981cb...", "thread_id": "f05f080a..."}}'
Raw: b'{"id": "177459...", "event": "message.started", "data": {"thread_id": "f05f080a...", "run_id": "6001dfb7..."}}'

Event sequence: run.started → message.started → _…chunks…_ → run.completed. Each event carries a run_id and thread_id for correlating messages across a conversation.

Step 4 : Chat Completions (optional)

POST /instances/{instance_id}/v1/orchestrate/{agent_id}/chat/completions

For applications already built on the OpenAI Chat Completions interface, watsonx Orchestrate provides a drop-in compatible endpoint per agent — no client library changes required.

curl 'https://api.$WXO_REGION.watson-orchestrate.cloud.ibm.com/instances/$WXO_INSTANCE_ID/v1/orchestrate/$WXO_AGENT_ID/chat/completions' \
  --header 'Authorization: Bearer YOUR_ACCESS_TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{
    "messages": [{ "role": "human", "content": "What services do you offer?" }],
    "model": "open-ai/gpt-oss-120b",
    "stream": false
  }'

response = requests.post(
    f"https://api.{region}.watson-orchestrate.cloud.ibm.com"
    f"/instances/{instance_id}/v1/orchestrate/{agent_id}/chat/completions",
    headers={
        "Authorization": f"Bearer {token}",
        "Content-Type":  "application/json"
    },
    json={
        "messages": [{"role": "human", "content": "What services do you offer?"}],
        "model":    "open-ai/gpt-oss-120b",
        "stream":   False
    }
)

print(response.json())

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1234567890,
  "model": "open-ai/gpt-oss-120b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "We offer a wide range of services including..."
      },
      "finish_reason": "stop"
    }
  ],
  ...
}

Note: The response mirrors the OpenAI ChatCompletion schema, choices[0].message.content holds the reply and finish_reason indicates why generation stopped.

What’s Next

With authentication, agent discovery, and two interaction patterns in hand, you have everything required to build production-grade integrations on top of watsonx Orchestrate. From here, explore tool-calling to connect agents to external APIs, multi-agent collaboration, and context management to maintain conversation state across turns.

Your team already has a knowledge management tool — and it’s well-used. Years of how-to articles, troubleshooting guides, and step-by-step procedures live inside it, many of them rich with visual aids, including annotated screenshots showing where to click, diagrams of network topology, photos of physical printer setup steps.

Today, that means opening a portal, typing a search query, hoping the right article surfaces, and then reading through it. That workflow was fine before AI became mainstream. Now users expect more: they want to type a question in plain English: How do I connect the office printer on Windows? and get back a direct, conversational answer that includes the relevant screenshots from the original article, not just a link to go read it themselves.

The gap isn’t the content. The gap is the interface and the richness of the response. Text-only chatbot answers fail when the source material is inherently visual. A step-by-step printer setup guide without its annotated images is significantly less useful.

This writeup solves that gap by keeping your existing knowledge base exactly as it is, and adding an AI layer on top that:

1. Understands natural language questions
2. Retrieves the right article from your existing system
3. Extracts and surfaces the embedded images alongside the text answer, so users see the visual aids, not just words describing them

Solution overview

The system adds an AI layer on top of an existing knowledge base without modifying it. IBM watsonx Orchestrate is an agentic platform that acts as the AI brain — the LLM receives the user’s natural language question, decides to call an external search tool, and synthesises a rich response from the result. Critically, that response includes images extracted from the source articles, served to the chat widget with correct CORS headers so they render inline.

The tool-km (km, short for knowledge management) is a FastAPI micro service we control. It handles authenticating against the knowledge base API, fetching articles, parsing HTML bodies, extracting both text and embedded images, and resolving file attachments (PDFs, DOCX) based on a single keyword argument that the AI agent decides based on user query. Nothing in the existing knowledge base changes.

Everything runs as containers on IBM Cloud Code Engine and communicates over HTTPS.

Key design decisions

• Why not modify the existing knowledge base?
  The entire point of this architecture is non-invasive augmentation. The existing knowledge management system keeps running unchanged. The AI layer reads from it as a consumer. No schema changes, no data migration, no disruption to existing workflows or users.
• Why extract images rather than just link to the article?
  A link sends the user back to the original portal, which is the exact experience we’re replacing. Extracting images from the HTML and surfacing them inline in the chat widget means the user gets a complete, self-contained answer without context-switching. For visually-guided tasks like hardware setup, this is the difference between a useful response and a frustrating one.
• Why a dedicated image proxy service?
  Browser security (CORS, mixed-content) makes it impossible to render private COS images directly in a chat widget. A dedicated proxy sidecar solves this cleanly without coupling image-serving concerns into tool-km. It also scales independently — and to zero when idle.

Closing thoughts

Multiple knowledge sources could also be added alongside tool-km, such as SharePoint, ServiceNow, or Confluence. The LLM routes queries to the right tool automatically and each tool can follow the similar image extraction strategy via batch jobs.

Introduction

The rapid advancement of multimodal Large Language Models (LLMs) has opened new possibilities for AI to process both textual and visual information. These models can now understand images, extract text, and even perform more complex tasks like reasoning about the content of images in a way that was previously only possible with separate Optical Character Recognition (OCR) tools. But what if we could combine the power of LLM and OCR to create a more robust solution for image text extraction and understanding?

In this blog post, we explore how multimodal LLM, specifically Meta’s Llama 3.2, can complement OCR systems like Tesseract to enhance text extraction and interpretation from images.

Llama 3.2: Meta’s Multimodal Marvel

Llama 3.2 is Meta’s open-source LLM that can process both text and images. This marks a significant step forward in multimodal AI systems, where the model is capable of understanding high-resolution images, extracting textual content, and even relating those visual inputs to textual knowledge.

Key Features of Llama 3.2:

• Multimodal capabilities: Llama 3.2 can handle both text and image inputs
• Scale: model comes in 11B and 90B parameter sizes, with robust vision models
• Resolution support: Llama 3.2 can process high-resolution images (up to 1120x1120 pixels)
• Adapter weights: to integrate image understanding into the pre-existing language model, Llama 3.2 uses adapter weights, which consist of cross-attention layers that feed image encoder representations into the language model

This integration allows the model to process images in conjunction with text while preserving the language-only capabilities of the Llama model. Developers can leverage Llama 3.2 as a drop-in replacement for earlier versions, such as Llama 3.1, while gaining the benefit of image understanding.

OCR with Tesseract: A Tried-and-Tested Solution

While Llama 3.2 provides a powerful, multimodal solution, the integration of traditional OCR tools like Tesseract can still be highly valuable, especially when extracting text from images with varying qualities or formats.

What is Tesseract OCR?

Tesseract is an open-source OCR engine developed by Google, capable of recognizing text from images. It supports a wide range of image types (e.g., JPEG, PNG, TIFF) and is used in a variety of applications, from scanning documents to text extraction from screenshots.

For Python developers, the python-tesseract library is a popular wrapper around the Tesseract engine, enabling easy integration into Python-based workflows. With Tesseract, you can:

• Extract text from images
• Retrieve the text along with bounding box coordinates (for localization)
• Handle a variety of image formats via the Pillow and Leptonica libraries

This makes Tesseract a robust tool for text extraction tasks, but it is not without its limitations — particularly in cases where the image quality is poor, or the text is embedded in complex visual contexts.

How LLM and OCR Can Complement Each Other

By combining the strengths of OCR (text extraction) and multimodal LLMs (image/text understanding), we can create more advanced solutions for image-text interactions. Here are a few ways in which these technologies can work together:

1. Improved LLM Responses with OCR

Approach:

One of the simplest yet most powerful ways to leverage OCR with LLMs is to use OCR-extracted text as part of the prompt when interacting with the model. The extracted text can provide context for the LLM, allowing it to refer back to the OCR data when necessary.

Use Case:

For example, if you’re working with an image containing text in different sections, you can pass the OCR result into the prompt and ask the model to interpret or analyze it accordingly. The LLM can use this extracted text to generate more accurate and context-aware responses.

Result:

• OCR-only: Extracted text from the image
• LLM-only: Model interpretation without OCR input
• OCR + LLM: Text extraction combined with context from LLM analysis

# example prompt for OCR + LLM
prompt = f"""
You are an AI assistant specialized in understanding images
 
user question:
{query}
 
use following text extracted from object character recognition:
{ocr_text}
"""

This approach provides a way for the LLM to reference OCR results directly, improving response quality and relevance.

2. Correcting and Enhancing OCR Results with LLM

Approach:

OCR engines like Tesseract may occasionally misinterpret or miss certain details in text extraction, particularly when images are noisy or contain unusual fonts. By passing the OCR-extracted text (along with localization data, such as bounding box coordinates) into an LLM, you can ask the model to correct or enhance the extracted text.

Use Case:

You could pass the OCR results to the LLM and ask it to make corrections based on the surrounding context or additional information provided by the image. For instance, if the OCR engine fails to recognize certain words or misinterprets characters, the LLM can make intelligent corrections based on the context of the image or the surrounding text.

Result:

By comparing the results from:

• OCR-only: The raw, potentially flawed OCR output
• OCR + LLM: The improved version of the text, as corrected by the LLM

# example prompt for OCR + LLM
prompt = f"""
You are an AI assistant specialized in understanding images.
Your task is to review text extracted from OCR.
 
OCR extracted text with location:
{ocr_text}
 
Correct any errors from OCR and return text
"""
   
Copy code
      # example ocr_text json
# left/top/width/height are boundries/locations for text boxes
orc_text = [
    {
        "index": 13,
        "text": "Tax Invoice",
        "left": 177,
        "top": 198,
        "width": 225,
        "height": 22
    },
    {
        "index": 16,
        "text": "Ahmedabad Shopping Mall",
        "left": 170,
        "top": 220,
        "width": 178,
        "height": 17
    },
    ...
  ]

The LLM’s ability to correct and improve OCR results can lead to more accurate text extraction, especially in challenging image conditions.

3. Highlighting Specific Text in Images

Approach:

When working with an image that contains multiple blocks of text, it may be useful to extract and highlight specific portions of the image. By passing both OCR-extracted text and its bounding box coordinates into an LLM, you can request the model to return the relevant areas of the image or highlight the text corresponding to a user query.

Use Case:

For example, if you’re processing a document image and need to find the portion of the image that contains a specific keyword or phrase, you can use OCR to identify and localize the text, then prompt the LLM to identify the relevant text and return its bounding box coordinates.

The role of the LLM here is to understand user query and relate them to text extracted from OCR, without LLM, we may need to use other techniques such as fuzzy match or sematic search (via embeddings)

Result:

• Box area: exact location of the highlighted text
• Image with highlight: section of the image that corresponds to the user query, visually highlighted

prompt = f"""
You are an AI assistant specialized in analyzing images.
Here's the result from OCR: {ocr_text}
 
Return the following JSON object with 2 keys:
- answer: contains the answer to find {query}, note that you should correct mistakes made by OCR
- reference: a single numeric index used from OCR as references that are relevant
 
Return only valid JSON. Nothing else.
"""

# example JSON output from LLM with query: date
{
    "text": "The date is 25/8/17. The reference index is 43.\n\n```json\n{\n    \"answer\": \"25/8/17\",\n    \"reference\": 43\n}\n```",
    "json": {
        "answer": "25/8/17",
        "box": {
            "index": 43,
            "text": ":25/8/17",
            "left": 358,
            "top": 303,
            "width": 68,
            "height": 15
        }
    }
}

Conclusion

The integration of multimodal LLMs like Llama 3.2 and OCR technologies like Tesseract opens up a new world of possibilities for text extraction and understanding from images. By combining the strengths of both approaches, you can enhance the quality of OCR results, generate more context-aware responses, and even extract and highlight specific parts of an image based on textual content.

As AI models continue to evolve, we can expect even more sophisticated solutions that seamlessly integrate vision and language, driving forward the capabilities of image-text interactions in various applications, from document processing to interactive AI assistants.

Overview

This blog post is to demonstrate the studying and interpretation of the relationships between machine learning model output and various input variables — as a starting point to optimize lead conversion.

Business context-wise, we will be learning about an online education company called X Education that sells online courses to industry professionals. With limited resources, it is important for the company to identify which users are more likely to be converted into leads.

We will mostly be using R to perform the analysis and visualization.

Live demo: https://jameschen4.shinyapps.io/leads/

library(data.table)
library(caret)
library(xgboost)
library(pROC)
library(xgboostExplainer)
library(dplyr)
library(tidyr)
library(reactable)
library(htmltools)

Data

The data is available on Kaggle. One good thing about this data set is that it comes with a data dictionary, which explains what key columns mean. There are 37 variables and around 10k records.

Preprocessing

There is already a well-written EDA by Ashish with Python. We will be taking the cleaned dataset directly from him, and focus on the simulation and visualization part.

We will also be dropping these columns, which are mainly related to leads:

• Lead Quality (Low in Relevance, Might be, Not Sure, Worst)
• Lead Origin (Landing Page Submission, Lead Add Form, Lead Import)
• Lead Source (Facebook, Google, Olark Chat, Organic Search..etc)

Training

Here we will train an XGBoost model with 10-fold cross-validation, where the target variable is if the user is converted into a lead (0- No, 1-Yes). The code below are taken directly from David Foster.

Turns out the baseline model already gives good result: AUC 0.9650633.

Using the xgboostExplainer package, we are able to identify few important variables that could help lead conversion.

The package creates a waterfall chart which can help us identify the most important contributing factors when it comes to lead conversion.

For the record below (taken from the test set):

Prediction: 0.9630512

Weight: 3.260574

After reviewing some more cases, we have identified a list of 6 variables that may help convert a lead:

• Total.Time.Spent.on.Website
• Last.Activity_SMS.Sent/Last.Notable.Activity_SMS.Sent
• Tags_Will.revert.after.reading.the.email
• Specialization_Other_Specialization
• Tags_Lost.to.EINS
• TotalVisits

Simulation

The core idea here is that if the variables are independent, we could play around with different inputs to see how the predictions will change.

We already know from Ashish’s EDA that there isn’t any serious collinearity (usually when VIF > 5):

Numerical Variables

First we will create a list of percentages and a list of seeds to iterate though. The percentage here refers to the percentage of user base that could be affected. For example, if perc = 0.2, that means we will randomly select 20% of the records (again, from the test data), and add 600 seconds to the Total.Time.Spent.on.Website variable, while other variables remain unchanged for those 20% records.

The diversified seeding here is just to make sure the simulation is more robust. For every iteration, we will take the median of the predictions. Note that here baseline of median predictions before we simulate anything is around 0.152.

A side note is that, despite there are no serious collinearity present, when TotalVisits = 0, Total.Time.Spent.on.Website must also be 0 as well. For simplicity, we are ignoring this for now.

Another note is that here we are only demonstrating a fixed boost on the continuous variables (for example, add 600 seconds), and we can always try different combinations as well.

One-hot Variables

For one-hot variables, one way is to split up the 0s and 1s first, and then randomly take a percentage on the 0’s (target) and convert it into 1’s — which means that the SMS Sent becomes the last activity for those selected records.

We then repeat the process for the remaining 4 variables, and the aggregated conversion rates are as below:

Visualization

Before visualization, let’s also add more details to the final data:

For the actual visualization, we will construct a heatmap in a table format with the reactable package — best part? It is compatible with R Shiny.

Live demo: https://jameschen4.shinyapps.io/leads/

Some thoughts behind this visualization:

• Upward and download arrow icons are to illustrate increase/decrease objectives — which is more eye-catching
• Baseline categorical and numerical are just reference points on current test data (before any simulation) — and will be used to evaluate how feasible it is to set up said objective. For example, collecting 50% more users from EINS Education (a competitor company) may not be realistic as current baseline is only 2%
• Numbers often are aligned right, and texts are aligned left. However, since we only have a few handful of numbers, aligning center looks better with % and #
• Emphasize X=10% on both the first column of the lead simulations as well as overhead description with X% of user…

High priority questions to ask next:

• What can we do to attract users from EINS Education?
• What strategies EINS education are using?
• Why do users buy from us and EINS both?
• Are our email and SMS campaigns successful — where can we get more data on this?
• Why users without specialization are less likely to convert? Is it because they actually don’t have one, or that they don’t want to tell us this information?

Final Thoughts

We focus on objectives here (what we want to achieve), as opposed to the actual approach we are taking, because it may require deeper analysis and domain knowledge of Online Education to come up with solid optimization plan (for example, to study why certain users are spending 2000+ seconds on the website — is it because of certain content?).

Also note that in proper setting, there should be A/B testing to validate correlations revealed by the XGBoost model. But again, the analysis in this blog should only be served as the starting point of lead conversion optimization, and it is an iterative process to refine the analysis with different stakeholders.

Machine learning model is often perceived as a blackbox, and the community is well aware of this (hence we see packages like xgboostexplainer and shap). Sometimes even the simplest linear regression is hard for others to understand — and this is where simulation or sensitivity-based analysis can add value. There are also other ways to perform such analysis:

From variables A, B, C and D; which combination of values of A, B and C (without touching D) increases the target y value by 10, minimizing the sum of A, B and C.

Questions?

jchen6912@gmail.com

What can we do to help convert leads? was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Overview

The original article with links to technical papers can be found here. Part I of this blog will cover overview and challenges of machine learning, particularly within the realm of labeling training data. Part II will then focus more on the technical implementations of Snorkel, based on technical papers published.

Audience

The audience of this blog post are data scientists and data analysts. We will create new visualizations on key concepts covered within the original article published by Stanford in Part I. Advanced analytics background is not required to understand this blog post, but is nice to have.

Objective

The objective of Part I blog post is to communicate the background and potential challenges data science practitioners might face in supervised machine learning — training data labeling. In real world data science, training data sets might not always be properly labeled.

The problem statement here is clear: both Classical and Modern ML require large amount of training data sets to be manually labeled, in order to obtain promising predictive results. However, common labeling process often takes domain experts months (if not years) to do so, in a consistent way. Not to mention that when analytic objectives change, labels require re-tagging too.

Exhibit 1 is based on below section from the original article, along with additional information for new perspectives.

In Traditional Supervision, it is common to identify data points that lie closer to the decision boundary, and prioritize domain experts’ time on those data first, hoping that learning can be based on more valuable information.

There are also ways to prioritize labeling data points over outliers in the cyber security setting, where attack signals only account for a tiny fraction compared to normal web traffics. The paper can be found here.

Snorkel, however, utilizes Weak Supervision, which not only includes such prioritization method, but also with other label generation approaches.

Accordingly to the article, Weak Supervision is capable of leverage both manual and automatic labeling methods to get more labeled data, in a less painful way, while maintaining promising results.

Weak Supervision can take inputs from both domain experts (high-level abstracts similar to Expert Systems) and non-experts, as there are internal mechanisms to deal with conflicts and correlations (we will cover in Part II). Varies input formats are also supported, including task-specific heuristics, regular express patterns (mostly for NLP), and generic rules of thumb.

Exhibit 2 is based on below section from the original article.

This sums up for the Part I. We will cover the technical aspects of Weak Supervision in Part II. Stay tuned!

Questions?

jchen6912@gmail.com

This blog post is a quick introduction to interactive data visualization with D3. The focus of this post is to walk to basic ideas/steps of visualizing data, as opposed to actual technical implementation.

Live Demo(Chrome on PC for best experience).

Data

Available here.

It is consisted of the daily top 200 most listened songs in 50+ countries from 2017 and 2018 by Spotify users. It contains more than 2 million rows, which includes more than 6,600 artists, and close to 20,000 songs.

Problem

While there could be a number of different ways to frame the problem, here wewould like to know that what is the lifespan of popular songs. As a result, we will use first day’s data (top 200 songs listened on 2017–01–01) as the baseline and see changes on the percentage of songs remaining on the top 200 list over time. No surprise here that the percentage will drop as people are getting tired of listening to the same songs, and new songs will appear.

Processing

Since this is rather large amount of data, we will import the CSV file to SQLite3 for easier manipulation. Also for data visualization, usually we don’t want to bombard audience with the full data; and hence, instead of showing daily results, we will go for weekly (so roughly 53 data points for each region).

Visuals

Highly recommend looking for visual inspirations from online sources, including Pinterest, McKinsey Insight, and Wall Street Journey.

This visualization is inspired from here.

Development

The actual development is through D3.js, where the official gallery/guide provides excellent demo and instructions on how to implement heat map visualization, along with interactive tool tips.

Deployment

Highly recommend Netlify, a free hosting service platform. Once you have created in the index.html file, you can drag and drop the entire folder onto the platform, and it would take less than 3 seconds for your site to be up and running; alternatively, you can also connect Git to it.

Questions?

jchen6912@gmail.com

Collecting

The key here is the format of data storage.

Intuitively we think of time-series data in the format below, also known as the wide format.

However, the wide format is bad for SQL-based storage, as when we add new dates, we need to add another column to the table, and thus, best practice is storing the data in long format, as shown below.

Another thing worth mentioning here is that since it is highly likely we are forecasting multiple products here, a convenient way is to use Python dictionary, where key is product ID, and value is the long data format (basically a dataframe). So we end up with a dictionary of dataframes.

Cleaning

Cleaning here is pretty standard. Basically we look for outliers. Common practice is that if we have weird values such as a fixed negative number for demand (which it usually is greater than or equal to 0), we drop them. Another thing we need to look for here is missing values. We could use pandas fillna function, in which we specify if we want to replace them with 0’s or with a method such as the last demand (backfill).

Testing

Also known as back-testing. Basically two most common ways of windowing, one is sliding (training and forecast time are fixed), and the other one is expanding (training data length increased by x at a time). Uber does a great job visualizing the two methods as below.

Common evaluation metrics include MAPE, wMAPE, sMAPE, as well as RMSE and RMLSE (if the data distribution is skewed).

Modeling

The most important take-away here is that often simple methods such as mean of all data points outperform more sophisticated ARIMA. Also FB’s prophet package (in both R and Python) could be a go-to algorithm as well.

Evaluating

When evaluating performance, we could try different training and forecasting length to better understand our data, as well as the algorithms used. For example, Naive method performs extremely well when training data is 1 week and forecasting data is the next week; and ARIMA shines when we have longer data length, in terms of both training and forecasting. In other words, we need to determine our business use case first, whether it is short-term forecast or long-term forecast, so that we can better evaluate the results from different algorithms.

Questions?

jchen6912@gmail.com

Time-series Forecasting Flow was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.