<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Himadri Talukder on Medium]]></title>
        <description><![CDATA[Stories by Himadri Talukder on Medium]]></description>
        <link>https://medium.com/@himadricuet?source=rss-b09579f13f8a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OKD2WtJVct0CYcOHNRYbbA.jpeg</url>
            <title>Stories by Himadri Talukder on Medium</title>
            <link>https://medium.com/@himadricuet?source=rss-b09579f13f8a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:34:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@himadricuet/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Running Apache Spark Jobs on IBM watsonx.data]]></title>
            <link>https://medium.com/ibm-data-ai/running-apache-spark-jobs-on-ibm-watsonx-data-1467f3b7b1f0?source=rss-b09579f13f8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1467f3b7b1f0</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[watsonx]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[watsonxdata]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Himadri Talukder]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 13:04:56 GMT</pubDate>
            <atom:updated>2026-07-10T13:04:56.777Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn how to build, submit, monitor, and troubleshoot Spark applications using the watsonx.data Spark Engine with real world examples and best practices on watsonx.data lakehouse.</h4><p>If you’re coming from a traditional Spark background, your first question about IBM watsonx.data is probably: <em>“Where’s </em><em>spark-submit?&quot;</em></p><p>The answer: there isn’t one. Watsonx.data exposes its Spark engine as a REST API. You POST a JSON payload, keep your spark job in a COS bucket, get back a job ID, and poll for the result. That’s the entire mental model shift.</p><p>This guide walks through the complete end-to-end workflow using a real-world example: reading a CSV file from Cloud Object Storage (COS), loading it into an Iceberg table in the watsonx.data catalog, and then running a second Spark job to analyze the ingested data.</p><p>Along the way, we’ll cover the practical pitfalls that aren’t in the official docs, the ones that will actually break your jobs in production.</p><p>All the code referenced here lives in this GitHub repository: <a href="https://github.com/hitalukder/data-engineering/tree/main/watsonx.data/spark-engine"><strong>github.com/hitalukder/data-engineering</strong></a></p><pre>├── submit_job.py             ← runs locally; talks to the watsonx REST API<br>├── csv_to_iceberg.py         ← runs inside the Spark engine; reads CSV, writes Iceberg<br>├── sales_analysis_2026_q1.py ← runs inside the Spark engine; analyzes the ingested CSV<br>├── .env.example              ← credential template (never commit the real .env)<br>└── requirements.txt</pre><h3>What We’re Building</h3><p>The CSV lives in a COS bucket you own. A Spark job reads it and writes it out as an Iceberg table into the watsonx.data catalog, which is backed by a separate, watsonx-managed COS bucket. Both buckets need credentials wired up correctly, this is where most people get stuck, and we’ll cover exactly how to avoid that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iEnJfBaAgT9D6-V7aF5WxQ.png" /><figcaption>Architecture diagram</figcaption></figure><h4>How a Job Flows Through the Engine, End to End</h4><p>Here’s a concise breakdown of what actually happens when you submit a job:</p><ol><li><strong>API key exchange:</strong> submit_job.py calls get_access_token(), which POSTs your IBM_API_KEY to iam.cloud.ibm.com. IBM IAM returns a short-lived bearer token (about one hour) that authorizes every subsequent call to the watsonx.data engine.</li><li><strong>Upload Spark scripts to COS: </strong>Before submitting anything, your PySpark scripts (csv_to_iceberg.py / sales_analysis_2026_q1.py) are uploaded from your machine to spark-home-bucket via cos s3 cp. The engine can&#39;t receive code inline it has to pull it from a COS path at runtime.</li><li><strong>POST the job:</strong>submit_job.py sends a single POST /applications request carrying three things: the s3a:// path to the script, the CLI arguments, and the conf dictionary (HMAC credentials plus Hive auth). This returns a job_id. <strong>This POST is never retried in-process</strong> — the API has no idempotency key, and duplicate POSTs create duplicate jobs.</li><li><strong>Poll for state:</strong>poll_until_done() calls GET /applications/{job_id} every 10 seconds and logs the state transitions: accepted → running → finished. The poll loop tolerates transient 500s from the control plane — the Spark job keeps running regardless of whether the status endpoint answers cleanly.</li><li><strong>The engine executes: </strong>The watsonx Spark engine pulls the script from spark-home-bucket and runs it. On startup, it merges its own pre-configured defaults (catalogs, Hive metastore URI) with the conf you submitted (your HMAC credentials, Hive PLAIN auth). The script then reads the source CSV from raw-data-bucket using the HMAC credentials registered for that bucket.</li><li><strong>writeTo().overwritePartitions()</strong>The Spark executor writes the processed DataFrame to the target Iceberg table in the watsonx.data lakehouse catalog (bronze_catalog.bronze_schema). The write lands in bronze-bucket (the managed lakehouse COS bucket) using plain S3v4 HMAC signing instead of the default signer, because the default WatsonxAWSV4Signer rejects the API key at write time.</li><li><strong>Logs:</strong> Once the job reaches a terminal state, the engine flushes all driver stdout/stderr to spark-home-bucket at spark/{engine_id}/logs/{job_id}/. download_driver_logs() lists that prefix and downloads every file locally to logs/{job_id}/. This is your only window into what happened inside the engine — stack traces, row counts, and print statements all land here.</li></ol><p>Now let’s run the sample program from the <a href="https://github.com/hitalukder/data-engineering/blob/main/watsonx.data/spark-engine/README.md">GitHub repo.</a></p><h3>Prerequisites</h3><p>Before writing any code, you need four things set up in IBM Cloud:</p><p><strong>1. A watsonx.data instance with a Spark engine provisioned.</strong> Find your engine in the watsonx.data console. Note the engine ID from the URL, it looks like spark669.</p><p><strong>2. An IBM Cloud API key.</strong> Create one at <a href="https://cloud.ibm.com/iam/apikeys">cloud.ibm.com/iam/apikeys</a>. This is how you authenticate to both the watsonx REST API and the Hive metastore.</p><p><strong>3. HMAC credentials for your COS instance.</strong> This is the one step people miss. When creating a COS service credential, you must pass {&quot;HMAC&quot;: true} in the parameters. IAM-only credentials don&#39;t work with Spark&#39;s S3A filesystem driver.</p><p><strong>4. The Spark application script uploaded to COS.</strong> Watsonx pulls your PySpark script directly from a COS path, there’s no way to send it inline. We’ll cover this below.</p><h3>Step-by-Step: Running the Sample Job</h3><p>Here are the steps to run the program</p><h4>Step 1: Clone the Repo and Configure Credentials</h4><pre>git clone git@github.com:hitalukder/data-engineering.git<br>cd data-engineering.git/watsonx.data/spark-engine<br>uv init<br>uv venv<br>source .venv/bin/activate<br>uv pip install -r requirements.txt</pre><p>requirements.txt contains the three packages submit_job.py needs to run locally (these are <strong>not</strong> installed inside the Spark engine itself):</p><pre>boto3&gt;=1.30<br>requests&gt;=2.28<br>python-dotenv&gt;=1.0</pre><p>Now copy the credential template and fill it in:</p><pre>cp .env.example .env</pre><p>Open .env and populate every field:</p><pre># IBM Cloud identity<br>IBM_API_KEY=&lt;your API key from cloud.ibm.com/iam/apikeys&gt;<br>IBM_INSTANCE_ID=crn:v1:bluemix:public:lakehouse:us-south:a/&lt;bss&gt;:&lt;instance&gt;::<br>SPARK_ENGINE_URL=https://us-south.lakehouse.cloud.ibm.com/lakehouse/api/v3/spark_engines/SAPRK_ENGINE_ID<br><br># COS — must be HMAC credentials, not IAM-only<br>S3_ENDPOINT=https://s3.us-south.cloud-object-storage.appdomain.cloud<br>S3_ACCESS_KEY=&lt;HMAC access key&gt;<br>S3_SECRET_KEY=&lt;HMAC secret key&gt;<br><br># The bucket where submit_job.py will upload the Spark script<br>SPARK_JOB_BUCKET=spark-home-bucket/jobs<br><br># your csv file bucket<br>CSV_DATA_BUCKET=spark-home-bucket/jobs/sales_2026_q1.csv</pre><h4>Step 2: Upload the Spark Application to COS</h4><p>Watsonx has no way to receive your script inline, it must pull it from a COS path at submission time.</p><p>Upload the Spark application script to the desired cos bucket.</p><ul><li>csv_to_iceberg.py to s3://SPARK_JOB_BUCKET</li><li>sales_analysis_2026_q1.py to s3://SPARK_JOB_BUCKET</li></ul><p>Your sample CSV also needs to be in COS. Upload it to the data bucket. A data file is provided for your reference. <a href="https://github.com/hitalukder/data-engineering/blob/main/data/structured/sales/sales_2026_q1.csv">Data file</a></p><ul><li>sales_2026_q1.csv to s3://CSV_DATA_BUCKET</li></ul><h4>Step 3: Understand submit_job.py</h4><p>This is the local script that drives everything. It has four distinct responsibilities, each in its own function.</p><h4>3a. Get an IAM Bearer Token</h4><pre>def get_access_token(api_key: str) -&gt; str:<br>    resp = requests.post(<br>        &quot;https://iam.cloud.ibm.com/identity/token&quot;,<br>        headers={&quot;Content-Type&quot;: &quot;application/x-www-form-urlencoded&quot;},<br>        data={&quot;grant_type&quot;: &quot;urn:ibm:params:oauth:grant-type:apikey&quot;,<br>              &quot;apikey&quot;: api_key},<br>        timeout=30,<br>    )<br>    resp.raise_for_status()<br>    return resp.json()[&quot;access_token&quot;]</pre><p>Tokens expire after roughly one hour. In any long-running orchestrator, call this fresh before each submission rather than caching across jobs.</p><h4>3b. Build the Spark conf</h4><p>This is the most important part — and the most common place submissions fail.</p><pre>def build_conf() -&gt; dict:<br>    bucket = SPARK_JOB_BUCKET   # spark-home-bucket<br>    conf = {}<br><br>    # Register HMAC credentials for every non-lakehouse bucket you touch.<br>    # The per-bucket key pattern overrides the global spark.hadoop.fs.s3a.*<br>    # defaults, which means the engine&#39;s pre-configured lakehouse signing<br>    # is unaffected for buckets you don&#39;t list here.<br>    for b in [bucket, &quot;raw-data-bucket&quot;]:<br>        conf[f&quot;spark.hadoop.fs.s3a.bucket.{b}.aws.credentials.provider&quot;] = \<br>            &quot;org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider&quot;<br>        conf[f&quot;spark.hadoop.fs.s3a.bucket.{b}.endpoint&quot;]   = S3_ENDPOINT<br>        conf[f&quot;spark.hadoop.fs.s3a.bucket.{b}.access.key&quot;] = S3_ACCESS_KEY<br>        conf[f&quot;spark.hadoop.fs.s3a.bucket.{b}.secret.key&quot;] = S3_SECRET_KEY<br><br>    # If the engine&#39;s default CAS signer rejects your API key on a lakehouse<br>    # bucket (you&#39;ll see IbmWxdDasException: Please provide valid api key),<br>    # bypass it with these two keys:<br>    #   conf[&quot;spark.hadoop.fs.s3a.bucket.bronze-bucket.custom.signers&quot;] = &quot;&quot;<br>    #   conf[&quot;spark.hadoop.fs.s3a.bucket.bronze-bucket.s3.signing-algorithm&quot;] = &quot;&quot;<br><br>    # Hive metastore PLAIN auth.<br>    # Must be set under BOTH prefixes — spark.hive.metastore.* is read by<br>    # Spark&#39;s HiveExternalCatalog; spark.hadoop.hive.metastore.* is<br>    # propagated to Hadoop Configuration, which Iceberg&#39;s HiveCatalog reads.<br>    # If you only set one, Iceberg writes silently fail with &quot;user not authorized.&quot;<br>    # Note: the username must be ibmlhapikey_&lt;your-email&gt;, not bare ibmlhapikey.<br>    hive_auth = {<br>        &quot;client.plain.username&quot;: &quot;ibmlhapikey_&lt;your-email@example.com&gt;&quot;,<br>        &quot;client.plain.password&quot;: API_KEY,<br>        &quot;use.SSL&quot;:               &quot;true&quot;,<br>        &quot;client.auth.mode&quot;:      &quot;PLAIN&quot;,<br>        &quot;truststore.type&quot;:       &quot;JKS&quot;,<br>        &quot;truststore.path&quot;:       &quot;file:///opt/ibm/jdk/lib/security/cacerts&quot;,<br>        &quot;truststore.password&quot;:   &quot;changeit&quot;,<br>    }<br>    for k, v in hive_auth.items():<br>        conf[f&quot;spark.hive.metastore.{k}&quot;]        = v<br>        conf[f&quot;spark.hadoop.hive.metastore.{k}&quot;] = v<br><br>    return conf</pre><p>Two rules that save most first-timers:</p><ul><li>Don’t set spark.driver.memory, spark.executor.memory, or catalog keys. The engine pre-configures all of these. Overriding them causes either an HTTP 400 on submission or silent metastore auth failure.</li><li>Use the per-bucket credential pattern (spark.hadoop.fs.s3a.bucket.&lt;name&gt;.*) so you&#39;re only scoping your credentials to the buckets you own, leaving the engine&#39;s signed access to managed lakehouse buckets untouched.</li></ul><h4>3c. Submit the Job</h4><pre>def submit_job(headers: dict, args: argparse.Namespace) -&gt; str:<br>    payload = {<br>        &quot;application_details&quot;: {<br>            # The engine pulls this path from COS at runtime<br>            &quot;application&quot;: f&quot;s3a://{SPARK_JOB_BUCKET}/csv_to_iceberg.py&quot;,<br>            # These become sys.argv inside csv_to_iceberg.py<br>            &quot;arguments&quot;: [<br>                &quot;--source-path&quot;,  &quot;s3a://raw-data-bucket/sales/sales_2026_q1.csv&quot;,<br>                &quot;--target-table&quot;, f&quot;bronze_catalog.{args.iceberg_db}.{args.iceberg_table}&quot;,<br>            ],<br>            &quot;conf&quot;: build_conf(),<br>        }<br>    }<br>    resp = requests.post(f&quot;{SPARK_URL}/applications&quot;,<br>                         headers=headers, json=payload, timeout=60)<br>    resp.raise_for_status()<br>    return resp.json()[&quot;id&quot;]</pre><blockquote><strong><em>POST exactly once. Never retry this call inside your client.</em></strong><em> The watsonx API has no idempotency key. A 5xx from the gateway can arrive </em>after<em> the engine has already accepted your request. In-process retries can create multiple duplicate Spark jobs from a single logical submission. Let your orchestrator (Airflow, Argo, etc.) handle retries at the task level.</em></blockquote><h4>3d. Poll for Completion</h4><pre>TERMINAL_STATES = {&quot;finished&quot;, &quot;failed&quot;, &quot;killed&quot;, &quot;stopped&quot;, &quot;error&quot;}<br><br>def poll_until_done(job_id: str, headers: dict, max_minutes: int = 30) -&gt; dict:<br>    deadline = time.time() + max_minutes * 60<br>    while time.time() &lt; deadline:<br>        try:<br>            resp = requests.get(f&quot;{SPARK_URL}/applications/{job_id}&quot;,<br>                                headers=headers, timeout=90)<br>            resp.raise_for_status()<br>            state = (resp.json().get(&quot;state&quot;) or &quot;&quot;).lower()<br>            log.info(f&quot;  state → {state}&quot;)<br>            if state in TERMINAL_STATES:<br>                return resp.json()<br>        except requests.RequestException as e:<br>            # The poll endpoint can return 500s intermittently even when the<br>            # Spark job is running fine. Treat as a warning, not a failure.<br>            log.warning(f&quot;  poll error (will retry): {e}&quot;)<br>        time.sleep(10)<br>    raise TimeoutError(f&quot;Job {job_id} did not finish within {max_minutes} minutes&quot;)</pre><p>The key insight here: a 500 on the poll endpoint does not mean your Spark job failed. The job is running regardless of whether the control plane can answer your status request. Keep polling, eventually a successful poll will reveal the terminal state.</p><h4>Step 4: Run It</h4><p>Run the data loading job first</p><pre>python3 submit_job.py \<br>  --iceberg-db   bronze_schema \<br>  --iceberg-table sales_q1</pre><p>Run the analysis job</p><pre>python submit_job.py \<br>        --iceberg-db   bronze_schema \<br>        --iceberg-table sales_q1 --script sales_analysis_2026_q1</pre><p>Expected output:</p><pre>2026-06-30 19:31:03,790 - INFO - Script      : csv_to_iceberg<br>2026-06-30 19:31:03,790 - INFO - Job bucket  : s3://spark-home-bucket/jobs<br>2026-06-30 19:31:03,790 - INFO - Engine      : spark669<br>2026-06-30 19:31:04,149 - INFO - Submitting  : s3a://spark-home-bucket/jobs/csv_to_iceberg.py<br>2026-06-30 19:31:04,149 - INFO - Arguments   : [&#39;--source-path&#39;, &#39;s3a://spark-home-bucket/jobs/sales_2026_q1.csv&#39;, &#39;--target-table&#39;, &#39;bronze_catalog.bronze_schema.sales_q1&#39;, &#39;--load-mode&#39;, &#39;full_refresh&#39;]<br>2026-06-30 19:31:05,361 - INFO - Submitted — job_id=079de639-82e0-4ed6-9160-59d5d96f39b7<br>2026-06-30 19:31:06,372 - INFO -   state → accepted<br>2026-06-30 19:31:49,887 - INFO -   state → running<br>2026-06-30 19:32:54,303 - INFO -   state → finished<br>2026-06-30 19:32:54,305 - INFO - Final state : finished</pre><p>If the final state is anything other than finished, the logs will be downloaded automatically to logs/&lt;job-id&gt;/. Read the driver log tail for the stack trace.</p><h4>Step 5: Verify the Data Landed</h4><p>Run these queries from the watsonx.data query editor:</p><pre>-- Catalog view (source of truth — always use this, not raw COS listing)<br>SELECT COUNT(*) FROM bronze_catalog.bronze_schema.sales_q1;<br><br>-- Spot-check rows<br>SELECT * FROM bronze_catalog.bronze_schema.sales_q1 LIMIT 10;</pre><p>To confirm storage and catalog are in sync, compare that row count against the number of Parquet files under the table prefix in the COS console. They should match the Iceberg snapshot’s total-data-files count. If the file count is higher than expected, you have orphan files left over from a previous overwrite.</p><h3>Common Errors and Recommended Configuration</h3><p>This is where most submissions fail silently or with cryptic errors. The watsonx engine pre-configures a lot for you. Hive metastore connectivity, default catalogs, internal signing for the lakehouse buckets and your job’s conf either complements those defaults or accidentally overrides them. The six common hiccup, below cover nearly everything that trips people up.</p><h4>1. don’t override what the engine already provides</h4><p>The engine ships pre-configured spark.hive.metastore.*, spark.sql.catalog.*, and signer config for the lakehouse buckets (bronze-bucket, gold-bucket). Setting your own values for these can silently disable metastore login or break catalog resolution. Add only what the engine doesn’t already know about.</p><h4>2. register HMAC credentials for any non-lakehouse buckets you touch</h4><p>We needed spark-home-bucket for shipping the Spark app + GCP key + BQ JAR. That&#39;s not a managed lakehouse bucket, so the engine doesn&#39;t have credentials for it. Tell it:</p><pre>conf = {<br>    f&quot;spark.hadoop.fs.s3a.bucket.{bucket}.aws.credentials.provider&quot;:<br>        &quot;org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider&quot;,<br>    f&quot;spark.hadoop.fs.s3a.bucket.{bucket}.endpoint&quot;: S3_ENDPOINT,<br>    f&quot;spark.hadoop.fs.s3a.bucket.{bucket}.access.key&quot;: S3_ACCESS_KEY,<br>    f&quot;spark.hadoop.fs.s3a.bucket.{bucket}.secret.key&quot;: S3_SECRET_KEY,<br>}</pre><p>The pattern is spark.hadoop.fs.s3a.bucket.&lt;bucket-name&gt;.&lt;setting&gt; these are per-bucket overrides that take precedence over the global spark.hadoop.fs.s3a.* defaults.</p><h4>3. when the engine’s default signer rejects you, bypass it</h4><p>The lakehouse buckets (bronze-bucket, gold-bucket) are accessed by default through a custom watsonx signer that calls out to CAS (Common Access Service) to mint signatures. CAS validates your IBM Cloud API key against fine-grained watsonx permissions.</p><p>In practice, even with the Manager role on the watsonx.data instance, CAS can reject submissions outright. The fix is to bypass the custom signer for those buckets and route them through plain S3v4 signing with your COS HMAC credentials instead, which do work</p><pre>&quot;spark.hadoop.fs.s3a.bucket.bronze-bucket.aws.credentials.provider&quot;:<br>    &quot;org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider&quot;,<br>&quot;spark.hadoop.fs.s3a.bucket.bronze-bucket.endpoint&quot;: S3_ENDPOINT,<br>&quot;spark.hadoop.fs.s3a.bucket.bronze-bucket.access.key&quot;: S3_ACCESS_KEY,<br>&quot;spark.hadoop.fs.s3a.bucket.bronze-bucket.secret.key&quot;: S3_SECRET_KEY,<br># Disable the Watsonx custom signer:<br>&quot;spark.hadoop.fs.s3a.bucket.bronze-bucket.custom.signers&quot;: &quot;&quot;,<br>&quot;spark.hadoop.fs.s3a.bucket.bronze-bucket.s3.signing-algorithm&quot;: &quot;&quot;,</pre><p>Do the same for gold-bucket. Before blaming anything else, verify your HMAC keys work end-to-end with plain boto3.</p><h4>4. ship code via spark.files, JARs via spark.jars</h4><pre>&quot;spark.files&quot;: f&quot;s3a://{bucket}/{GCP_KEY_FILENAME}&quot;,<br>&quot;spark.jars&quot;:  f&quot;s3a://{bucket}/{BQ_JAR_FILENAME}&quot;,</pre><p>Inside the Spark app, spark.files artifacts are reachable via SparkFiles.get(filename). <strong>Important:</strong> SparkFiles.getRootDirectory() asserts that a SparkContext is already initialized, so any code that imports a shipped Python module via SparkFiles must run <em>after</em> SparkSession.builder.getOrCreate().</p><p>If you’re reading from BigQuery, the BigQuery Spark connector needs both a JAR (shipped via spark.jars) and a service-account key (shipped via spark.files):</p><pre>&quot;spark.files&quot;: f&quot;s3a://{bucket}/bq-service-account.json&quot;,<br>&quot;spark.jars&quot;:  f&quot;s3a://{bucket}/spark-4.0-bigquery-0.44.0.jar&quot;,<br>&quot;spark.hadoop.google.cloud.auth.service.account.enable&quot;: &quot;true&quot;,<br>&quot;spark.datasource.bigquery.parentProject&quot;: GCP_PROJECT,<br>&quot;spark.datasource.bigquery.project&quot;:       GCP_PROJECT,</pre><p>Inside the Spark app:</p><pre>from pyspark import SparkFiles<br>key_path = SparkFiles.get(&quot;bq-service-account.json&quot;)<br>hadoop_conf = spark.sparkContext._jsc.hadoopConfiguration()<br>hadoop_conf.set(&quot;google.cloud.auth.service.account.enable&quot;, &quot;true&quot;)<br>hadoop_conf.set(&quot;google.cloud.auth.service.account.json.keyfile&quot;, key_path)<br><br>df = (spark.read.format(&quot;bigquery&quot;)<br>        .option(&quot;table&quot;, &quot;project.dataset.table&quot;)<br>        .option(&quot;credentialsFile&quot;, key_path)<br>        .option(&quot;parentProject&quot;, GCP_PROJECT)<br>        .load())</pre><h4>5. don’t try to set driver/executor sizing in conf</h4><p>Watsonx.data Spark validates spark.driver.memory / spark.executor.memory as a pair with CPU. Setting only memory throws an HTTP 400 at submission. On most plan tiers, you also can&#39;t set sizing on individual application submissions at all — engine sizing is configured at the engine level via &quot;t-shirt sizes&quot; in the watsonx console.</p><p>So don’t include any spark.driver.* or spark.executor.* resource keys in your conf unless you&#39;ve confirmed they&#39;re accepted for your specific engine configuration.</p><h4>6. Hive metastore auth, the silent failure mode</h4><p>If you’re writing Iceberg tables backed by the watsonx Hive metastore, your Spark application needs to authenticate to that metastore itself. The engine pre-configures this only for its own default credentials — when you submit a job under your own identity, you need to provide PLAIN auth credentials explicitly:</p><pre>hive_plain = {<br>    &quot;client.plain.username&quot;: &quot;ibmlhapikey_&lt;your-email&gt;&quot;,  # NOT just &quot;ibmlhapikey&quot;<br>    &quot;client.plain.password&quot;: IBM_API_KEY,<br>    &quot;use.SSL&quot;:               &quot;true&quot;,<br>    &quot;client.auth.mode&quot;:      &quot;PLAIN&quot;,<br>    &quot;truststore.type&quot;:       &quot;JKS&quot;,<br>    &quot;truststore.path&quot;:       &quot;file:///opt/ibm/jdk/lib/security/cacerts&quot;,<br>    &quot;truststore.password&quot;:   &quot;changeit&quot;,<br>}<br># Set under BOTH prefixes:<br>for k, v in hive_plain.items():<br>    conf[f&quot;spark.hive.metastore.{k}&quot;]         = v   # Spark HiveExternalCatalog reads this<br>    conf[f&quot;spark.hadoop.hive.metastore.{k}&quot;]  = v   # Iceberg HiveCatalog reads this (via Hadoop Config)</pre><p>Two non-obvious things in there:</p><ol><li><strong>Both prefixes</strong>. spark.hive.metastore.* is read by Spark&#39;s HiveExternalCatalog; spark.hadoop.hive.metastore.* is propagated to the Hadoop Configuration object, which is what Iceberg&#39;s HiveCatalog reads. If you only set one, half your queries break — typically writes succeed but Iceberg-specific operations (like CALL system.expire_snapshots) fail with a permission error.</li><li><strong>ibmlhapikey_&lt;your-email&gt;</strong>, not bare ibmlhapikey. The email suffix is part of the SASL principal. Without it, the metastore accepts the connection but downstream Iceberg writes silently fail with &quot;user not authorized.&quot;</li></ol><p>The watsonx.data Spark engine works well once you internalize one shift: you’re not launching a Spark cluster, you’re calling an API that manages one for you. Almost every hard-to-diagnose failure in this workflow traces back to the same root cause either fighting the engine’s pre-configured defaults (memory, catalogs, signers) or missing one half of a two-part config (Hive auth under both prefixes, HMAC creds for every non-lakehouse bucket).</p><p>If your job fails, start in this order:</p><ol><li>Check the driver logs in logs/&lt;job-id&gt;/ — they&#39;re your only real window into the engine.</li><li>Confirm your HMAC credentials work with plain boto3 before suspecting Spark.</li><li>Double-check Hive auth is set under <em>both</em> spark.hive.metastore.* and spark.hadoop.hive.metastore.*.</li><li>Make sure you haven’t accidentally set spark.driver.memory, spark.executor.memory, or catalog keys.</li></ol><p>Get these fundamentals right once, and the rest of the workflow submitting, polling, and verifying is refreshingly boring, which is exactly what you want from a production data pipeline.</p><p><em>Did I use AI? Yes, I did. I used AI to help improve the writing. I asked it to fix the grammar and make the content simple, clear, and easy to follow. The ideas, code and content are mine; AI only helped improve the wording.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1467f3b7b1f0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/running-apache-spark-jobs-on-ibm-watsonx-data-1467f3b7b1f0">Running Apache Spark Jobs on IBM watsonx.data</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IBM watsonx.data Intelligence: Metadata Enrichment & Text-to-SQL]]></title>
            <link>https://medium.com/ibm-data-ai/ibm-watsonx-data-intelligence-metadata-enrichment-text-to-sql-bf03eef1451b?source=rss-b09579f13f8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf03eef1451b</guid>
            <category><![CDATA[intelligence]]></category>
            <category><![CDATA[metadata-management]]></category>
            <category><![CDATA[watsonxdata]]></category>
            <category><![CDATA[data-intelligence]]></category>
            <category><![CDATA[text-to-sql]]></category>
            <dc:creator><![CDATA[Himadri Talukder]]></dc:creator>
            <pubDate>Mon, 08 Jun 2026 15:03:50 GMT</pubDate>
            <atom:updated>2026-06-08T15:03:50.341Z</atom:updated>
            <content:encoded><![CDATA[<h4>From Natural Language to SQL (Text2SQL): watsonx.data Intelligence with DB2</h4><p>IBM watsonx.data Intelligence helps you turn raw database tables into governed, semantically rich data assets that can be queried accurately with natural language. In this tutorial, you will load data into Db2, enrich metadata in watsonx.data Intelligence, test the Text2SQL API, and package the workflow into a FastAPI application.</p><blockquote><strong><em>Core insight:</em></strong><em> Data without context is just bytes. Metadata enrichment gives your data </em>meaning <em>business terms, display names, descriptions, relationships, quality scores. Text2SQL then uses that meaning to let anyone ask questions in plain English and get trusted, governed, accurate answers.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLp0hUImLEn__rp5nDmLmA.png" /><figcaption>Reference Architecture</figcaption></figure><h3>What you will build</h3><p>By the end of this guide, you will have:</p><ul><li>A Db2 database loaded with a university dataset.</li><li>A watsonx.data Intelligence project connected to Db2.</li><li>Imported and enriched data assets with business-friendly metadata.</li><li>Text2SQL working against your governed assets.</li><li>A simple FastAPI service that sends natural-language questions to the Text2SQL endpoint and returns SQL.</li></ul><h3>Prerequisites</h3><p>Before you start, make sure you have:</p><ul><li>An IBM Cloud account with access to watsonx.data Intelligence</li><li>A Db2 instance with permission to create tables and load data</li><li>An IBM Cloud API key for generating IAM tokens</li><li>A watsonx.data Intelligence project</li><li>Connection details for Db2: host, port, database, username, and password</li></ul><h4>Supported SQL dialects</h4><p>The Text2SQL API supports multiple SQL dialects: For this tutorial, we use db2.</p><ul><li>db2</li><li>presto</li><li>presto_sql</li><li>postgresql</li><li>mssql</li><li>oracle</li><li>snowflake</li></ul><h4>API base URLs by region</h4><p><a href="https://cloud.ibm.com/apidocs/data-semantic-automation">Semantic API endpoint base URL</a> Choose the base URL that matches your watsonx.data Intelligence instance location:</p><ul><li>Dallas: <a href="https://api.dataplatform.cloud.ibm.com/">https://api.dataplatform.cloud.ibm.com</a></li><li>Frankfurt: <a href="https://api.eu-de.dataplatform.cloud.ibm.com/">https://api.eu-de.dataplatform.cloud.ibm.com</a></li><li>London: <a href="https://api.eu-gb.dataplatform.cloud.ibm.com/">https://api.eu-gb.dataplatform.cloud.ibm.com</a></li><li>Tokyo: <a href="https://api.jp-tok.dataplatform.cloud.ibm.com/">https://api.jp-tok.dataplatform.cloud.ibm.com</a></li><li>Sydney: <a href="https://api.au-syd.dai.cloud.ibm.com/">https://api.au-syd.dai.cloud.ibm.com</a></li><li>Toronto: <a href="https://api.ca-tor.dai.cloud.ibm.com/">https://api.ca-tor.dai.cloud.ibm.com</a></li></ul><p>My instance is in Dallas <em>us-south</em>. Set your environment variables now, you will reuse them throughout the tutorial:</p><pre>export DIALECT=db2<br>export BASE_URL=https://api.dataplatform.cloud.ibm.com<br>export IAM_API_KEY=&lt;your-ibm-cloud-api-key&gt;<br>export WXD_PROJECT_ID=&lt;your-project-id&gt;</pre><blockquote><strong><em>Note for on-premises deployments:</em></strong><em> If you are running IBM watsonx.data Intelligence on IBM Software Hub (on-prem), the Text-to-SQL feature is delivered via a separate Semantic Automation component. Make sure you have applied the latest patch for your version (for example, the IKC 5.3.0 / watsonx.data intelligence 2.3.0 patch) before proceeding. The UI path to enable Text-to-SQL may differ slightly from the SaaS steps described in this guide.</em></blockquote><h3>Steps for This Experiment</h3><ul><li>Step 1: Load Data into Db2</li><li>Step 2: Generate an IBM Cloud access token</li><li>Step 3: Configure watsonx.data intelligence</li><li>Step 4: Create a DB2 connection</li><li>Step 5: Import DB2 Assets</li><li>Step 6: Add data classification categories</li><li>Step 7: Enrich metadata using AI</li><li>Step 8: Test the Text2SQL API endpoint</li><li>Step 9: Build a FastAPI application to package the solution</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*57TIMC53H5ouiIf-PHuhuw.png" /><figcaption>Architecture diagram text to sql</figcaption></figure><h3>Step 1: Load Data into Db2</h3><p>Load your source data into Db2 first. For this tutorial, we use a university dataset with five tables: students, courses, teachers, enrollment, and grades. The CSV files are available in the <a href="https://github.com/hitalukder/data-engineering/tree/main/data/structured/academic">GitHub repository.</a></p><p>You can load data into Db2 in several ways:</p><ul><li>Db2 web UI (simplest for demos — shown below)</li><li>Object storage import</li><li>Db2 import utilities</li><li>Python with pandas</li><li>Other database ingestion tools</li></ul><h4><strong>Using the Db2 web UI:</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W75rH5SUUg7omFFvr01Lpg.png" /><figcaption>DB2 Data Upload — 1</figcaption></figure><ul><li>Click <strong>Data</strong> and drag a CSV file or browse to select it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mYjqXAF7gScB-ZKPRhLcqg.png" /><figcaption>DB2 Data Upload — 2</figcaption></figure><ul><li>Select a target schema.</li><li>If no table exists, click <strong>New Table</strong> , columns are created automatically from the CSV headers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aqlwEytLf3qTdRJ6c47KmQ.png" /><figcaption>DB2 Data Upload — 3</figcaption></figure><p>Verify the load by checking that the tables exist and the row counts look correct. This is important because metadata enrichment and Text2SQL both depend on the underlying schema being loaded properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X9G4j7JVTwZ3C0F3oOf7EQ.png" /><figcaption>DB2 Data Upload — 4</figcaption></figure><p>Click Next-&gt; Begin load</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f90LcwITPjytBk4fVZlrjQ.png" /><figcaption>DB2 Data Upload — 5</figcaption></figure><ul><li>Repeat for all five CSV files.</li></ul><p><strong>Verify:</strong> After loading, confirm each table exists and the row counts look correct. Metadata enrichment and Text2SQL both depend on a properly loaded schema.</p><blockquote><strong><em>Tip:</em></strong><em> Save your Db2 connection credentials now from the </em><strong><em>Service Credentials</em></strong><em> tab. You will need the host, port, database name, username, and password in Step 4.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C8rWl7AID2fKydGUwDawHw.png" /><figcaption>DB2 Service Credentials for login</figcaption></figure><h3>Step 2: Generate an IBM Cloud access token</h3><p>The Text2SQL API uses IAM Bearer tokens for authentication. Tokens expire, so you may need to regenerate one if you get a 401 error later.</p><h4>2.1 Create an IBM Cloud API Key</h4><ul><li>In the IBM Cloud console, click your user avatar (top-right) and select <strong>Manage → Access (IAM)</strong>.</li><li>Go to <strong>API keys → Create</strong>.</li><li>Enter a name, then copy and save the generated key immediately — it is shown only once.</li></ul><blockquote><strong><em>Security:</em></strong><em> Never commit your API key to source control. Store it in an environment variable or a secrets manager.</em></blockquote><pre>export IAM_API_KEY=&lt;your-api-key&gt;</pre><h4>2.2 Generate an Access Token</h4><pre>curl -X POST &quot;https://iam.cloud.ibm.com/identity/token&quot; \<br>  -H &quot;Content-Type: application/x-www-form-urlencoded&quot; \<br>  -d &quot;grant_type=urn:ibm:params:oauth:grant-type:apikey&amp;apikey=$IAM_API_KEY&quot;</pre><p>The response contains an access_token. Save it:</p><pre>export BEARER_TOKEN=&lt;access_token&gt;</pre><blockquote>Use Authorization: Bearer $BEARER_TOKEN in all subsequent API requests.</blockquote><h3>Step 3: Configure watsonx.data intelligence</h3><p>From the IBM Cloud resource list, open your watsonx.data Intelligence instance.</p><h4>3.1. Create a new project:</h4><p>Create a new watsonx.data Intelligence project that will be used to manage data assets, metadata enrichment, governance artifacts, and Text2SQL capabilities.</p><p>If no project exists, create one: enter a name and description.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T7BVumZxEZ3QbHUhgXSkKw.png" /><figcaption>Watsonx.data intelligence new project create</figcaption></figure><p>Go to the General tab and copy the Project ID. This information will be used in later steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vrv20O0VcnnajCXTm05hPg.png" /><figcaption>Watsonx.data intelligence project Id</figcaption></figure><pre>export WXD_PROJECT_ID=&lt;Project ID&gt;</pre><h4>3.2 Enable Text-to-SQL</h4><p>IBM requires the project to be onboarded for Text-to-SQL before the API will accept requests for it.</p><ul><li>In the left panel, select <strong>Data Intelligence</strong>.</li><li>Enable the <strong>Natural Language</strong> (Text-to-SQL) feature.</li></ul><p>can enable the feature from the project UI under the Data Intelligence or Natural Language settings, depending on your tenant version.</p><ol><li>Go to <strong>Projects</strong> and select your project.</li><li>Open the <strong>Manage</strong> tab.</li></ol><blockquote><strong><em>On-premises note:</em></strong><em> On IBM Software Hub deployments, this feature is part of the Semantic Automation component. If you do not see the option, verify the component is installed and the patch has been applied.</em></blockquote><h3>Step 4: Create a DB2 connection</h3><ol><li>Enable the <strong>Natural Language</strong> feature.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JVlNgEBTPJ0ajdWEclWOzA.png" /><figcaption>Watsonx.data intelligence enable text to sql service</figcaption></figure><h3>Step 4: Create a DB2 connection</h3><p>Create a connection from watsonx.data Intelligence to your Db2 database. This connection will be used to discover and access database assets.</p><ol><li>Open your Data Intelligence project.</li><li>Go to the Assets tab.</li><li>Click New Asset</li><li>In the dialog window, select Connect to a data source</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9lFO7vxgLV17Rp5qhRRceQ.png" /><figcaption>Watsonx.data intelligence create new connection</figcaption></figure><p>Select IBM Db2 and click Next</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7OWV7zuAQF-Q_GCMDFcShg.png" /><figcaption>Watsonx.data intelligence create new connection db2</figcaption></figure><ol><li>Enter the connection details.</li><li>Before saving, click Test Connection to make sure everything is working correctly.</li><li>Set the connection name to db2-connection</li></ol><p>Import tables from Db2 into the project. Imported assets become available for metadata enrichment, governance, and Text2SQL.</p><ul><li>Go to the <strong>Assets</strong> tab and click <strong>Import assets</strong>.</li></ul><h3>Step 5: Import DB2 Assets</h3><p>Import tables from Db2 into the project. These imported assets become available for metadata analysis, enrichment, and governance.</p><ul><li>Go to the Assets tab and click on the Import assets</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iokJ2F_heQYfLVaL0UJxAg.png" /><figcaption>Data intelligence new asset import — 1</figcaption></figure><ul><li>Select the db2-connection you created.</li><li>Click <strong>Next</strong>, then select your schema.</li><li>Choose all five tables (students, courses, teachers, enrollment, grades).</li><li>Click <strong>Import</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ndd3JSV6ZMHEnYl_wC6Osg.png" /><figcaption>Data intelligence new asset import — 2</figcaption></figure><p>After import, all five tables should appear as data assets in the project’s data directory. If an asset is missing, check that the table was loaded successfully in Step 1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yHvCNadjQ7TTe3DXwlt3Ew.png" /><figcaption>Data intelligence new asset import — 3</figcaption></figure><h3>Step 6: Add data classification categories</h3><p>Create business categories and classifications that reflect your domain. These categories are used during enrichment to suggest and assign business terms to columns and assets. Well-structured categories with clear descriptions significantly improve the quality of AI-generated metadata.</p><p>Navigate to <strong>Hamburger menu → </strong><strong>Governance → </strong><strong>All artifacts</strong>, then create the following hierarchy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ECEQP4zMhthZ0WOW8AoZg.png" /><figcaption>Data intelligence add new classified terms</figcaption></figure><p>Add short, specific descriptions to each subcategory. The enrichment process uses these descriptions as semantic context when assigning terms to columns. Vague or empty descriptions produce weaker suggestions.</p><pre>Academic Institution              ← Root category<br>├── Student Information           ← Subcategory<br>│     Description: Student ID, GPA, Enrollment Year, Major, Student Status<br>├── Academic Staff                ← Subcategory<br>│     Description: Teacher ID, Department, Title, Specialization, Salary Grade<br>├── Course Management             ← Subcategory<br>│     Description: Course Code, Credits, Course Level, Schedule, Capacity<br>├── Enrollment &amp; Registration     ← Subcategory<br>│     Description: Enrollment ID, Enrollment Status, Payment Status, Attendance<br>└── Academic Performance          ← Subcategory<br>      Description: Grade, Letter Grade, Grade Points, Pass/Fail, Remarks</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GRc-JJOjVdbIe2ll-LV5_Q.png" /><figcaption>Data intelligence add new catagory terms</figcaption></figure><h3>Step 7: Enrich metadata using AI</h3><p>Metadata enrichment is the step that makes Text2SQL work well. IBM watsonx.data Intelligence analyzes your imported assets , their column names, data types, sample values, and surrounding context and automatically generates descriptions, business term assignments, and relationship mappings.</p><p>This enriched metadata helps users understand their data more easily and improves data discovery, governance, and usability</p><p>Go to the Assets tab, click New Asset and then select Enrich data assets with metadata</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qvNFj5fAxA9_t27T9DC4jA.png" /><figcaption>Data intelligence enrich metadata — 1</figcaption></figure><ul><li>Enter a name and optional description and <strong>tags</strong>.</li><li>Then select data from Project → Data Asset, choose all the data, and click Select.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i_eWY6_R5k1Uull8Vqy3KQ.png" /><figcaption>Data intelligence enrich metadata — 2</figcaption></figure><p>In the next step, select the <strong>enrichment objective</strong> and <strong>categories</strong>. Choose the options based on your data, patterns, and how you want to proceed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f930b9d83ea105b7eb764b5112d190a2/href">https://medium.com/media/f930b9d83ea105b7eb764b5112d190a2/href</a></iframe><p>For this tutorial, select: <strong>Profile data</strong>, <strong>Expand metadata</strong>, <strong>Assign terms and classifications</strong>, and <strong>Set relationships</strong>.</p><p>Also, select a <strong>sampling option</strong> depending on your dataset size, and your needs for speed and accuracy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3b6c41b0d05d932fe78c5849ab10db2/href">https://medium.com/media/a3b6c41b0d05d932fe78c5849ab10db2/href</a></iframe><p>For this tutorial, <strong>Basic</strong> is a good starting point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0_hyfL8tXxrH4CG9lLkShw.png" /><figcaption>Data intelligence enrich metadata — 3</figcaption></figure><p>Select the <strong>Academic Institution</strong> root category (and its subcategories) you created in Step 6.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HpC5Fz99keIH7EEJ6uJSWA.png" /><figcaption>Data intelligence enrich metadata — 4</figcaption></figure><p>Next, you will see the <strong>metadata enrichment scheduled job</strong> option. You can keep the default settings unless you want to change them. Then click Next, and finally click Create.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*swusLab7jEwE0lH_85XbvA.png" /><figcaption>Data intelligence enrich metadata — 5</figcaption></figure><p><strong>Enrichment typically takes 15–30 minutes</strong> depending on data size and the objectives selected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fisYgXaXPXU8MgC5V0tziQ.png" /><figcaption>Data intelligence enrich metadata — 6</figcaption></figure><p>Once the job completes, open the enrichment results. You will see:</p><ul><li><strong>Suggested Description</strong> and <strong>Display Name</strong> columns for each table and column</li><li>Business term assignments</li><li>Detected key relationships (click <strong>Review Key Relationships</strong> to verify foreign keys)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Cn-z1O3ca4mrkS0o9JDJg.png" /><figcaption>Data intelligence enrich metadata — 7</figcaption></figure><blockquote><strong><em>Tip:</em></strong><em> If suggestions look weak or generic, add more detail to your governance category descriptions and re-run the enrichment. You can also provide SQL examples as additional context — IBM documents this as a supported method to improve Text2SQL generation.</em></blockquote><p>Next, review the generated descriptions and metadata. If you are satisfied, click <strong>Accept</strong>. If not, edit and add more details, then click <strong>Accept</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sNsVv-5HWT0rFBvBkTe1hw.png" /><figcaption>Data intelligence enrich metadata — 8</figcaption></figure><p>If you want to accept suggestions in bulk use the below option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZtesiJwBYVRvCSVv3EIpQ.png" /><figcaption>Data intelligence enrich metadata — 10</figcaption></figure><p>Metadata enrichment will also identify foreign key relationships. You can verify this by selecting <strong>Review Key Relationship</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j4aM1bYqUZOkkCS7YlrOJQ.png" /><figcaption>Data intelligence enrich metadata — 11</figcaption></figure><h3>Step 8: Test the Text2SQL API endpoint</h3><p>Submit natural-language questions to the Text2SQL API and verify the generated SQL against your expectations. Good test queries should cover:</p><ul><li>A simple aggregation</li><li>A multi-table join</li><li>A business-term query (uses terms from your governance categories)</li><li>A complex query that depends on detected relationships</li></ul><pre>Example questions:<br>- What is the average GPA for each major?<br>- Which courses have more than 90 percent of their seats filled?<br>- List all teachers hired after 2010 and their departments.<br>- Show student names along with the courses they are enrolled in.<br>- Find honor roll students based on GPA.</pre><h4>API call:</h4><pre>export TEXT_QUERY=&quot;Show student names along with the courses they are enrolled in.&quot;<br><br>curl --location \<br>  &quot;$BASE_URL/semantic_automation/v1/text_to_sql?container_id=$WXD_PROJECT_ID&amp;container_type=project&amp;dialect=$DIALECT&amp;model_id=meta-llama%2Fllama-3-3-70b-instruct&quot; \<br>  --header &quot;accept: application/json&quot; \<br>  --header &quot;Content-Type: application/json&quot; \<br>  --header &quot;Authorization: Bearer $BEARER_TOKEN&quot; \<br>  --data &quot;{<br>    \&quot;query\&quot;: \&quot;$TEXT_QUERY\&quot;,<br>    \&quot;raw_output\&quot;: false<br>  }&quot; | jq</pre><p><strong>The </strong><strong>raw_output field:</strong></p><ul><li>false (recommended): Returns a structured JSON object with the cleaned, parsed SQL and a confidence score.</li><li>true: Returns the raw model output. Useful for debugging when the generated SQL looks unexpected.</li></ul><h4>Output:</h4><pre>{<br>  &quot;input_query&quot;: &quot;Show student names along with the courses they are enrolled in.&quot;,<br>  &quot;generated_sql_queries&quot;: [<br>    {<br>      &quot;sql&quot;: &quot;SELECT \&quot;s\&quot;.\&quot;FIRST_NAME\&quot;, \&quot;s\&quot;.\&quot;LAST_NAME\&quot;, \&quot;c\&quot;.\&quot;COURSE_NAME\&quot; FROM \&quot;BGD20177\&quot;.\&quot;STUDENTS\&quot; \&quot;s\&quot; INNER JOIN \&quot;BGD20177\&quot;.\&quot;ENROLLMENT\&quot; \&quot;e\&quot; ON \&quot;s\&quot;.\&quot;STUDENT_ID\&quot; = \&quot;e\&quot;.\&quot;STUDENT_ID\&quot; INNER JOIN \&quot;BGD20177\&quot;.\&quot;COURSES\&quot; \&quot;c\&quot; ON \&quot;e\&quot;.\&quot;COURSE_ID\&quot; = \&quot;c\&quot;.\&quot;COURSE_ID\&quot;;&quot;,<br>      &quot;score&quot;: 100<br>    }<br>  ],<br>  &quot;model_id&quot;: &quot;meta-llama/llama-3-3-70b-instruct&quot;,<br>  &quot;resource_usage&quot;: {<br>    &quot;token_count&quot;: 5828,<br>    &quot;capacity_unit_hours&quot;: 0.0005410380,<br>    &quot;capacity_unit_hours_billed&quot;: 0,<br>    &quot;model_usages&quot;: [<br>      {<br>        &quot;model_type&quot;: &quot;embedding&quot;,<br>        &quot;model_id&quot;: &quot;granite-embedding-278m-multilingual&quot;,<br>        &quot;input_token_count&quot;: 16,<br>        &quot;output_token_count&quot;: 0<br>      },<br>      {<br>        &quot;model_type&quot;: &quot;foundation&quot;,<br>        &quot;model_id&quot;: &quot;meta-llama/llama-3-3-70b-instruct&quot;,<br>        &quot;input_token_count&quot;: 5325,<br>        &quot;output_token_count&quot;: 487<br>      }<br>    ]<br>  }<br>}</pre><p>The score field (0–100) reflects the model&#39;s confidence in the generated SQL. A score of 100 means the query passed all validation checks. Review any query with a score below 80 carefully before executing it.</p><p><strong>Verify:</strong> Execute the generated SQL directly in the Db2 web UI or via a Db2 client to confirm it returns the expected results. If the SQL is incorrect, improve your metadata descriptions or add SQL samples as context and re-test.</p><h3>Step 9: Build a FastAPI app</h3><p>Package the workflow into a small service so any user or downstream application can send a plain-English question and receive generated SQL — and optionally the query results.</p><h4>What the service does</h4><ol><li>Accepts a natural-language question via HTTP POST</li><li>Calls the Text2SQL API with the question</li><li>Returns the generated SQL (and optionally executes it against Db2 and returns results)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fDHAeoPwMkGypG3ijW1G7A.png" /></figure><p><a href="https://github.com/hitalukder/data-engineering/tree/main">Github repository</a> for the application and data.</p><ul><li>GitHub: <a href="https://github.com/hitalukder/data-engineering/tree/main/data/structured/academic">Sample data files for data ingestion.</a></li><li>GitHub: <a href="https://github.com/hitalukder/data-engineering/tree/main/watsonx.data/text-to-sql">Application code with text-to-SQL instructions.</a></li></ul><h3>Conclusion</h3><p>In this tutorial, we connected Db2 to watsonx.data Intelligence, enriched metadata to add business context, and tested Text2SQL with natural-language questions. The key takeaway is that accurate metadata, meaningful categories, and reviewed relationships directly improve how well Text2SQL can translate questions into SQL.</p><p>As you move from demo to production, treat metadata enrichment as an iterative process. The more complete and accurate your business terms, descriptions, and relationships are, the more reliable your generated SQL will be.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf03eef1451b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/ibm-watsonx-data-intelligence-metadata-enrichment-text-to-sql-bf03eef1451b">IBM watsonx.data Intelligence: Metadata Enrichment &amp; Text-to-SQL</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PostgreSQL Operator Architecture & Backup and Recovery]]></title>
            <link>https://medium.com/@himadricuet/crunchy-postgresql-operator-architecture-backup-and-recovery-fe7ef4f20af9?source=rss-b09579f13f8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe7ef4f20af9</guid>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[ibm-cloud]]></category>
            <dc:creator><![CDATA[Himadri Talukder]]></dc:creator>
            <pubDate>Fri, 02 Jan 2026 17:38:14 GMT</pubDate>
            <atom:updated>2026-01-04T22:26:42.565Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>PostgreSQL architecture on an OpenShift cluster using the Crunchy Database Operator, with an explanation of how backup and recovery work.</em></h4><p><a href="https://medium.com/p/46aaa70b4f09">In my previous blog</a>, I explained how to install the PostgreSQL Operator and provision a PostgreSQL cluster on an OpenShift cluster.</p><p>In this blog, I will go a step further and explain how the Crunchy Database Operator manages the database and the architecture behind it. I will walk through the major components and explain their roles and functionality.</p><p>Before you begin, please read the blog <a href="https://medium.com/p/46aaa70b4f09">How to Provision PostgreSQL on an OpenShift Cluster</a> first.</p><h3>Architecture Overview</h3><p>At a high level, the Crunchy PostgreSQL Operator deploys <strong>separate, specialized components</strong>, each with a single responsibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*YvKfd8hqorHA3Mi7itcBew.png" /><figcaption>High level Component Diagram</figcaption></figure><h4>1. pgBouncer</h4><p><strong>Type:</strong> Deployment<br><strong>Deployment name:</strong> &lt;cluster name&gt;-pgbouncer<br><strong>Service name:</strong> &lt;cluster name&gt;-pgbouncer<br><strong>Pod name:</strong> &lt;cluster name&gt;-pgbouncer-xxxx<br><strong>Responsibility</strong>:</p><ul><li><strong>Connection pooling</strong>: Manages database connections efficiently</li><li>Reduces overhead of creating new connections</li><li>Routes queries to the correct PostgreSQL instance</li><li>Acts as a proxy between applications and PostgreSQL</li></ul><p><strong>Why It’s Important</strong>:</p><ul><li>PostgreSQL has a limited number of connections (max_connections: 100)</li><li>Without pooling, you could run out of connections quickly</li><li>pgBouncer can handle thousands of client connections using only a few PostgreSQL connections</li></ul><p><strong>Your applications should connect to</strong>: &lt;cluster name&gt;-pgbouncer service (not directly to PostgreSQL)</p><p><strong>Configuration from your YAML</strong>:</p><pre>proxy:<br>  pgBouncer:<br>    replicas: 1<br>    port: 5432</pre><h4><strong>PostgreSQL Instance Pods</strong></h4><p><strong>Type</strong>: StatefulSet<br><strong>StatefulSet name:</strong> &lt;cluster name&gt;-instance1-&lt;random-string&gt;<br><strong>Pod name:</strong> &lt;cluster name&gt;-instance1-&lt;random-string&gt;-&lt;0-n&gt;<br><strong>Responsibility</strong>:</p><ul><li>Runs the actual PostgreSQL database server</li><li>Handles all database queries (read/write operations)</li><li>Stores the data on persistent volumes</li><li>Uses Patroni for high availability</li></ul><p><strong>Containers in the Pod</strong>:</p><ul><li><strong>database</strong>: Main PostgreSQL container</li><li><strong>pgbackrest</strong>: database backup</li><li><strong>replication-cert-copy</strong>: Copies TLS certificates for replication</li><li><strong>exporter: </strong>for monitoring</li></ul><p><strong>Key Role</strong>:</p><ul><li>Each pod gets its own persistent volume for data</li><li>Patroni manages failover and leader election</li><li>If you set replicas: 3, you get 1 primary + 2 replicas</li></ul><p><strong>Configuration from your YAML</strong>:</p><pre>instances:<br>  - name: instance1<br>    replicas: 1  # Number of PostgreSQL instances<br>    dataVolumeClaimSpec:<br>      storage: 50Gi  # Database data storage</pre><h4><strong>pgBackRest Repo Host</strong></h4><p><strong>Type</strong>: StatefulSet<br><strong>StatefulSet name:</strong> &lt;cluster name&gt;-repo-host<br><strong>Pod name:</strong> &lt;cluster name&gt;-repo-host-&lt;0-n&gt;<br><strong>Responsibility</strong>:</p><ul><li><strong>Backup management</strong>: Stores and manages all backups</li><li>Handles backup scheduling (full, differential, incremental)</li><li>Manages WAL (Write-Ahead Log) archiving</li><li>Performs restore operations when needed</li></ul><p><strong>Containers in the Pod</strong>:</p><ul><li><strong>pgbackrest</strong>: Main backup container</li><li><strong>pgbackrest-config</strong>: container for configuration</li></ul><p><strong>Configuration from your YAML</strong>:</p><pre>backups:<br>  pgbackrest:<br>    repos:<br>      - name: repo1<br>        volume:<br>          storage: 100Gi  # Backup storage</pre><p><strong>Key Roles</strong>:</p><ul><li>Runs scheduled backups automatically</li><li>Stores backups on persistent volume</li><li>Can restore to point-in-time</li></ul><h4><strong>PostgreSQL Exporter</strong></h4><p><strong>Type</strong>: pod (part of monitoring) <br>you will see a container inside the backup and postgresql pod<br><strong>Responsibility</strong>:</p><ul><li>Exposes PostgreSQL metrics for Prometheus</li><li>Monitors database health, performance, queries</li><li>Provides observability into your database</li></ul><p><strong>Configuration from your YAML</strong>:</p><pre>monitoring:<br>    pgmonitor:<br>      exporter:<br>        image: &#39;registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:ubi8-5.6.1-0&#39;<br>        resources:<br>          limits:<br>            cpu: 100m<br>            memory: 128Mi<br>          requests:<br>            cpu: 50m<br>            memory: 64Mi</pre><h4>Kubernetes Services</h4><ol><li><strong>&lt;cluster name&gt;-pgbouncer (ClusterIP)</strong></li></ol><ul><li>Port: 5432</li><li>Purpose: Main entry point for applications</li><li>Routes to pgBouncer pods</li></ul><p><strong>2. &lt;cluster name&gt;-primary (ClusterIP)</strong></p><ul><li>Port: 5432</li><li>Purpose: Direct access to primary PostgreSQL instance</li><li>Used for admin tasks, not recommended for apps</li></ul><p><strong>3. &lt;cluster name&gt;-replicas (ClusterIP)</strong></p><ul><li>Port: 5432</li><li>Purpose: Read-only access to replica instances</li><li>Used for read-heavy workloads</li></ul><p><strong>4. &lt;cluster name&gt;-pods (Headless Service)</strong></p><ul><li>Purpose: Internal communication between pods</li><li>Used by Patroni for cluster management</li></ul><pre>export cluster_name=&lt;Cluster Name&gt;<br>export INSTANCE_POD=$(oc get pods -l postgres-operator.crunchydata.com/instance-set=instance1 -o jsonpath=&#39;{.items[0].metadata.name}&#39;)<br></pre><pre>oc get deployment -l postgres-operator.crunchydata.com/cluster=$cluster_name</pre><pre>oc get pods -l postgres-operator.crunchydata.com/cluster=$cluster_name</pre><pre>oc get svc -l postgres-operator.crunchydata.com/cluster=$cluster_name</pre><h3>Understanding backup strategy</h3><p>pgBackRest supports <strong>three complementary backup types</strong>.</p><h4>1. Full Backup (F)</h4><p><strong>What it does</strong><br>Backs up the <em>entire database</em>, regardless of previous backups.</p><p><strong>Characteristics</strong></p><ul><li>Independent restore</li><li>Largest size: Takes a large storage</li><li>Slowest backup: Takes long time to complete</li><li>Safest: If you lose all other backups, you can still restore from a full backup</li></ul><p><strong>Example</strong></p><ul><li>Database size: 100 GB</li><li>Backup size: ~30–50 GB (compressed)</li><li>Time: ~20 minutes</li></ul><p><strong>When to use</strong></p><ul><li>Weekly baseline</li><li>Before upgrades</li><li>After major schema changes</li></ul><h4>2. Differential Backup (D)</h4><p><strong>What it does</strong><br>Backs up everything changed <strong>since the last full backup</strong>.</p><p><strong>What it does</strong>: Backs up only the data that has <strong>changed since the last FULL backup</strong>.</p><p><strong>Characteristics</strong>:</p><ul><li><strong>References</strong>: Always looks back to the most recent full backup</li><li><strong>Medium size</strong>: Grows larger as time passes since the last full backup</li><li><strong>Medium speed</strong>: Faster than full, slower than incremental</li><li><strong>Requires</strong>: The full backup + this differential to restore</li></ul><pre>Sunday:    Full backup (100 GB database) → 30 GB backup file<br>Monday:    Differential (5 GB changed since Sunday) → 2 GB backup file<br>Tuesday:   Differential (10 GB changed since Sunday) → 4 GB backup file<br>Wednesday: Differential (15 GB changed since Sunday) → 6 GB backup file</pre><p>Notice how each differential grows because it includes ALL changes since Sunday’s full backup.</p><p><strong>To restore from Wednesday</strong>: You need Sunday’s full + Wednesday’s differential</p><h4>3. Incremental Backup (I)</h4><p><strong>What it does</strong><br>Backs up only the data that has <strong>changed since the last backup of ANY type</strong> (full, differential, or incremental).</p><p><strong>Characteristics</strong>:</p><ul><li><strong>Chain-dependent</strong>: Creates a chain of backups</li><li><strong>Smallest size</strong>: Only captures changes since the very last backup</li><li><strong>Fastest</strong>: Quickest to complete</li></ul><p><strong>Restore requirement</strong></p><ul><li>Full backup + <em>all incrementals in the chain</em></li></ul><p><strong>Risk</strong></p><ul><li>One corrupt incremental breaks the chain</li></ul><pre>Sunday 2am:    Full backup (100 GB) → 30 GB backup file<br>Sunday 6am:    Incremental (1 GB changed since 2am) → 0.3 GB backup file<br>Sunday 10am:   Incremental (1 GB changed since 6am) → 0.3 GB backup file<br>Sunday 2pm:    Incremental (1 GB changed since 10am) → 0.3 GB backup file</pre><p>Each incremental only includes changes since the previous backup (whether full, diff, or incr).</p><p><strong>To restore from Sunday 2pm</strong>: You need the full backup + all three incremental backups</p><p>In the pervious blog we have the below backup schedule</p><pre>schedules:<br>  full: &quot;0 2 * * 0&quot;           # Sunday at 2 AM<br>  differential: &quot;0 2 * * 1-6&quot; # Mon-Sat at 2 AM<br>  incremental: &quot;0 */4 * * *&quot;  # Every 4 hours</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZwUg1fYFRHXXmAFFYBLH4A.png" /><figcaption>Backup schedule</figcaption></figure><h4>How backup actually flow</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ky3iTwKkkYADUZV_NRxbQQ.png" /><figcaption>Backup flow</figcaption></figure><h4>Checking backup status</h4><pre>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  pgbackrest info --stanza=d</pre><p>Shows:</p><ul><li>Backup type (F/D/I)</li><li>Size</li><li>Timestamps</li><li>Reference chains</li></ul><h3>Storage Location</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dac6c13fc5cae2596a04a2159f67a87d/href">https://medium.com/media/dac6c13fc5cae2596a04a2159f67a87d/href</a></iframe><p>All backups and archives are stored in the <strong>repo-host pod</strong> at these locations:</p><pre># Main backup repository root<br>/pgbackrest/repo1/<br><br># Directory structure:<br>/pgbackrest/repo1/<br>├── archive/              # WAL archive files<br>│   └── db/              # Stanza name<br>│       ├── 16-1/        # PostgreSQL version-timeline<br>│       │   ├── 000000010000000000000001-abc123.gz<br>│       │   ├── 000000010000000000000002-def456.gz<br>│       │   └── ...      # Compressed WAL files<br>│       └── backup.history/<br>├── backup/              # Backup files<br>│   └── db/              # Stanza name<br>│       ├── 20251222-020000F/    # Full backup (F suffix)<br>│       │   ├── backup.manifest<br>│       │   ├── backup.info<br>│       │   └── pg_data/         # Actual database files<br>│       ├── 20251223-020000D/    # Differential backup (D suffix)<br>│       └── 20251223-060000I/    # Incremental backup (I suffix)<br>└── backup.info           # Backup metadata</pre><p>Check space</p><pre># Check pvc<br>oc get pvc -l postgres-operator.crunchydata.com/cluster=${cluster_name},postgres-operator.crunchydata.com/pgbackrest-repo=repo1<br><br># Total space used<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  df -h /pgbackrest/repo1<br><br># Breakdown by directory<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  du -h --max-depth=2 /pgbackrest/repo1/ | sort -h<br><br># Count backup files<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- bash -c \<br>  &#39;cd /pgbackrest/repo1/backup/db/ | ls -l | wc -l&#39;<br><br># Count archive files<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- bash -c \<br>  &#39;cd /pgbackrest/repo1/archive/db/ | ls - | wc -l&#39;</pre><h3>Restore Strategies</h3><h4>In-Place Restore</h4><p>Replaces existing data</p><pre>oc patch postgrescluster $cluster_name --type merge \<br>  -p &#39;{&quot;spec&quot;:{&quot;backups&quot;:{&quot;pgbackrest&quot;:{&quot;restore&quot;:{&quot;enabled&quot;:true,&quot;repoName&quot;:&quot;repo1&quot;}}}}}&#39;</pre><p>Operator will:</p><ul><li>Stop pods</li><li>Recreate PVCs</li><li>Restore data</li></ul><h4>Restore to a New Cluster</h4><p>Safe, non-destructive verification.</p><pre>spec:<br>  dataSource:<br>    postgresCluster:<br>      name: &lt;Cluster name&gt; # The name of the original cluster<br>      namespace: &lt;namespace name&gt;<br>      repoName: repo1</pre><p>This is the recommended way if you want to verify data without destroying your live database.</p><ol><li>Create a <strong>new</strong> YAML file (e.g., &lt;cluster name&gt;-recovery.yaml).</li><li>Set a new name: &lt;cluster name&gt;-restore.</li><li>Add the dataSource section to point to your existing cluster:</li></ol><p>Use this for:</p><ul><li>Testing restores</li><li>Audits</li><li>Disaster recovery drills</li></ul><h4>Point-in-Time Recovery (PITR)</h4><p>Restore to an exact timestamp:</p><pre>pgbackrest restore \<br>  --stanza=db \<br>  --type=time \<br>  --target=&quot;2025-12-22 10:00:00&quot;</pre><p>pgBackRest automatically selects:</p><ul><li>Correct full</li><li>Differential (if available)</li><li>Required incremental</li><li>WAL files</li></ul><h3>Common Failure Modes and Fixes</h3><p>Even with automation, backup systems can fail. Below are the most common issues seen with pgBackRest in Kubernetes environments.</p><h4>1. no stanza db exists</h4><p><strong>Cause</strong></p><ul><li>Repo-host pod recreated</li><li>Misconfigured volume mount</li><li>Stanza was never created</li></ul><p><strong>Check</strong></p><pre># Verify stanza (operator might auto-create it)<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  pgbackrest info<br><br># Check the configuration<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  cat /etc/pgbackrest/conf.d/pgbackrest_repo.conf<br><br># Check mount points<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  df -h | grep pgbackrest</pre><p><strong>Fix</strong></p><pre># Create stanza<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  pgbackrest stanza-create --stanza=db --log-level-console=info</pre><p><strong>Verify:</strong></p><pre># Run test full backup<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  pgbackrest backup --stanza=db --type=full --log-level-console=info<br><br># Test incremental backup<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  pgbackrest backup --stanza=db --type=incr<br><br># Test differential backup<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  pgbackrest backup --stanza=db --type=diff<br><br># Verify backup<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  pgbackrest info --stanza=db</pre><h4>2. Repo Host Out of Disk Space</h4><p><strong>Symptoms</strong></p><ul><li>Backup failures</li><li>WAL archive failures</li></ul><p><strong>Fix</strong></p><pre>oc exec ${cluster_name}-repo-host-0 -c pgbackrest - df -h /pgbackrest/repo1</pre><p>Mitigation:</p><ul><li>Increase backup PVC size. Do not increase the PVC directly, increase from cluster yaml configuration</li><li>Reduce retention</li><li>Take fewer incremental</li></ul><h4>Some useful command for debug</h4><pre># Total space used<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  df -h /pgbackrest/repo1<br><br># Breakdown by directory<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  du -h --max-depth=2 /pgbackrest/repo1/ | sort -h<br><br># Check backup status using pgbackrest<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  pgbackrest info --stanza=db<br><br># Check backup info in JSON format<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  pgbackrest info --stanza=db --output=json<br><br># List all backups<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  pgbackrest info --stanza=db --output=text<br><br># Check archive status<br>oc exec $INSTANCE_POD -c database -- \<br>  psql -U postgres -c &quot;SELECT archived_count, failed_count, last_archived_wal, last_archived_time FROM pg_stat_archiver;&quot;<br><br># Verify backup integrity<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  pgbackrest verify --stanza=db<br><br># Check pgBackRest logs<br>oc logs ${cluster_name}-repo-host-0 -c pgbackrest --tail=100<br><br># Check PostgreSQL logs<br>oc logs $INSTANCE_POD -c database --tail=100<br><br>############################################<br># If You Need to Backup/Export the Data.   #<br>############################################<br><br># Create a tar of all backups<br>oc exec ${cluster_name}-repo-host-0 -c pgbackrest -- \<br>  tar -czf /tmp/backups.tar.gz -C /pgbackrest/repo1 backup/<br><br># Copy to your local machine<br>oc cp ${cluster_name}-repo-host-0:/tmp/backups.tar.gz ./backups.tar.gz -c pgbackrest<br></pre><h4>Tips</h4><ul><li>Always connect applications via <strong>pgBouncer</strong></li><li>Keep <strong>at least two full backups</strong></li><li>Monitor incremental chain length</li><li>Regularly test <strong>restore-to-new-cluster</strong></li><li>Take manual full backups before risky changes</li></ul><h3>Key Takeaways:</h3><ul><li><strong>Declarative Recovery:</strong> Manage backups (Full, Differential, Incremental) using the same GitOps principles as your application code.</li><li><strong>Tested Reliability:</strong> Move beyond “hope-based” recovery by using automated restore workflows to validate your data in new clusters.</li><li><strong>Platform Integration:</strong> Leveraging PostgreSQL 16 on OpenShift ensures your database is a first-class citizen of your cloud-native stack.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe7ef4f20af9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Processing Millions of Records on IBM watsonx]]></title>
            <link>https://medium.com/ibm-data-ai/processing-millions-of-records-on-ibm-watsonx-db890b636281?source=rss-b09579f13f8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/db890b636281</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[llm-inference]]></category>
            <category><![CDATA[watsonx]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Himadri Talukder]]></dc:creator>
            <pubDate>Thu, 18 Dec 2025 01:18:54 GMT</pubDate>
            <atom:updated>2025-12-18T01:18:54.421Z</atom:updated>
            <content:encoded><![CDATA[<h4>How we achieved 3× throughput processing millions of LLM API calls by switching from multithreading to async I/O on IBM watsonx</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*SnFx1Mv1WDpXzwk1hYNuNg.jpeg" /></figure><p>When we set out to process over a <strong>million LLM API calls per day</strong>, we didn’t expect our system to hang after running perfectly for 24 hours.</p><p>It started as a simple batch job, just call the API, collect the responses, and move on. But when real-world constraints kicked in <strong>rate limits, long inference times, and CPU spikes</strong> things got interesting.</p><p>This is the story of how we scaled our inference workload from <strong>frustration to 3× faster performance</strong>, using a combination of <strong>Celery workers</strong>, <strong>RabbitMQ</strong>, and <strong>asynchronous I/O</strong> in Python.</p><h3>1. The Challenge</h3><p>We had a tough goal: <strong>process one million LLM API calls per day</strong> each taking 10–20 seconds and stay within a strict <strong>30 requests per second (RPS)</strong> limit.</p><p>Each record required <strong>three dependent API calls</strong>, so we were effectively handling <strong>3 million calls</strong> per dataset. The system needed to be:</p><ul><li>Scalable</li><li>Fault-tolerant</li><li>Efficient for large-scale batch processing</li></ul><p>It sounded simple on paper but in reality, it tested everything from architecture design to async optimization.</p><h3>2. Understanding the Workload</h3><p>Before diving in, it’s important to understand the type of data flow we were dealing with:</p><ul><li><strong>Batch processing</strong>: All records are available upfront (our use case).</li><li><strong>Streaming</strong>: Records arrive incrementally (not our use case).</li></ul><p>Since we had all data ready, we could fully parallelize the workload but rate limits and API latency made this challenging.</p><h3>3. Designing for Scale</h3><p>From the outset, it was clear that this required a <strong>distributed architecture</strong>. The idea:</p><ul><li>Multiple <strong>worker nodes</strong> pulling tasks from a message queue.</li><li>Each worker handles concurrent API calls using threads or async I/O.</li></ul><h4>Key Design Considerations</h4><ul><li><strong>Parallel Processing</strong> Use multiprocessing or async I/O to speed up execution.</li><li><strong>Retry &amp; Failure Handling</strong> Implement exponential backoff for API failures.</li><li><strong>Queue-Based Load Distribution</strong> Use RabbitMQ, Kafka, or Redis to distribute workloads.</li><li><strong>Horizontal Scaling </strong>Run multiple workers across machines or containers.</li></ul><h3>4. Evaluating Architectural Approaches</h3><h4>Option 1: Multithreading / Async I/O</h4><ul><li>Ideal for I/O-bound workloads like network requests.</li></ul><h4>Option 2: Celery with Redis or RabbitMQ</h4><p>Simple, reliable distributed task queue.</p><ul><li>Easy to scale horizontally</li><li>Built-in retry and fault tolerance</li><li>Works seamlessly in containers and the cloud</li></ul><h4>Option 3: Stream Processing (Kafka + Flink/Spark)</h4><ul><li>Powerful but heavy. Suited for real-time, event-driven systems.</li></ul><h4>Option 4: Ray Framework</h4><ul><li>Great for distributed computation (CPU/GPU).</li><li>Offers actor model, async I/O, and dataset APIs.</li></ul><h4>Option 5: MapReduce / Apache Spark</h4><ul><li>Best for large-scale, in-memory batch jobs.</li></ul><h4>Choosing the Right Tool</h4><ul><li><strong>Heavy API calls / I/O-bound workloads →</strong> Use <strong>Celery + Redis</strong></li><li><strong>CPU/GPU-intensive parallel tasks →</strong> Use <strong>Ray</strong></li><li><strong>Stream and batch hybrid with high throughput →</strong> Use <strong>Kafka + Spark/Flink</strong></li></ul><blockquote>Our use case was heavily <strong>I/O-bound</strong> and required <strong>developer-friendly scalability</strong>, so we went with <strong>Celery + RabbitMQ</strong>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*HirPu1kO_N8KL5sbVXlAvg.png" /></figure><h3>5. Initial Implementation</h3><p>We started simple:</p><ul><li><strong>2 Celery worker processes</strong></li><li><strong>Batch size:</strong> 1,000 records per batch</li><li><strong>64 threads per process</strong> = 128 total threads running in parallel</li><li><strong>Tech stack:</strong> ibm_watsonx_ai library for LLM API calls</li></ul><h4>Test Setup</h4><p>Before production, we ran progressive load tests:</p><ul><li>1K records → 3K API calls</li><li>10K records → 30K API calls</li><li>100K records → 300K API calls</li></ul><p>The problems began at scale. Let me walk you through the three critical bottlenecks we discovered and solved.</p><h3>6. Problem 1: Hitting the API Rate Limit</h3><p>At 10,000 records, we received <strong>HTTP 429 errors for too many request</strong>s.</p><p>Our 128-thread setup was sending ~128 RPS far above the 30 RPS limit. This created a <strong>producer-consumer mismatch</strong>: threads were producing requests faster than the API could handle.</p><p>Think of it like a restaurant kitchen: our 128 threads were like 128 chefs trying to send orders to a single cook (the API) who could only handle 30 orders per second. Orders piled up and got rejected.</p><h4>Solution 1: Controlled Concurrency</h4><p>We took a hybrid approach:</p><ul><li><strong>Reducing threads</strong> to 64</li><li><strong>Adding short delays</strong> between requests</li><li><strong>Implementing a rate limiter</strong> to ensure steady request flow</li></ul><p>This stabilized the system no more rate-limit errors but total <strong>runtime is unsatisfactory.</strong></p><p><strong>Insight:</strong> Throttling fixed reliability but slowed throughput. We needed smarter concurrency. The question is, even though we are using 64 threads, are we actually hitting the desired rate limit? As the API has a latency of 10–20 seconds, there is still room to improve.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/706bad399f05c1d347fa354aa49a674b/href">https://medium.com/media/706bad399f05c1d347fa354aa49a674b/href</a></iframe><h3>7. Problem 2: Celery Workers Hanging</h3><p>During our production run of 1 million records, <strong>Celery workers stopped responding after ~24 hours</strong>. They stopped pulling new tasks, and the connection to RabbitMQ was broken.</p><ul><li>100% CPU usage</li><li>Worker stuck in Kombu/AMQP serialization loops</li></ul><pre>Thread 806548 (active+gil): &quot;MainThread&quot; dumps (amqp/serialization.py:274) serializeproperties (amqp/serialization.py:558)<br>dumps (amqp/serialization.py:274)<br>serialize_properties (amqp/serialization.py:558)<br>write_frame (amqp/method_framing.py:122)<br>send_method (amqp/abstract_channel.py:70)<br>basic_publish (amqp/channel.py:1797)<br>_publish (kombu/messaging.py:214)<br>_ensured (kombu/connection.py:556)<br>publish (kombu/messaging.py:190)<br>publish_reply (kombu/pidbox.py:277)<br>reply (kombu/pidbox.py:147)<br>dispatch (kombu/pidbox.py:110)<br>...<br>drain_events (amqp/connection.py:526)</pre><p>The worker was <strong>stuck in the Kombu/AMQP stack</strong> while trying to publish a response back to the broker.</p><p>After investigation, we discovered <strong>Celery’s default broker settings</strong> were causing infinite retries and backoff loops during RabbitMQ disconnections. The worker wasn’t crashed, it was trapped in an infinite reconnection loop, silently hanging without timing out.</p><h4><strong>Default behaviour:</strong></h4><ul><li>Infinite retry loop on connection failures</li><li>Exponential backoff with no timeout</li><li>Workers appeared “stuck” but were actually retrying forever</li></ul><h4>Solution 2: Broker Resilience Tweaks</h4><p>We explicitly tuned the broker transport options:</p><pre># -----------------------------<br># Broker Transport Options (Enhanced Resilience)<br># -----------------------------<br>broker_transport_options = {<br>    &#39;max_retries&#39;: 3,                    # ← Stop infinite retries!<br>    &#39;interval_start&#39;: 0.1,               # Retry starts at 100ms<br>    &#39;interval_step&#39;: 0.5,<br>    &#39;interval_max&#39;: 5.0,                 # Max backoff = 5s<br>    &#39;socket_timeout&#39;: 30,                # ← Prevents hanging<br>    &#39;socket_connect_timeout&#39;: 30,        # Explicit connect timeout<br>    &#39;heartbeat&#39;: 60,                     # ← AMQP heartbeat for stability<br>    &#39;visibility_timeout&#39;: 3600,          # For long-running tasks<br>    &#39;fanout_prefix&#39;: True,               # Better routing performance<br>    &#39;fanout_patterns&#39;: True,<br>}<br><br># -----------------------------<br># Performance Tweaks<br># -----------------------------<br>task_acks_late = True                   # Acknowledge only after completion<br>worker_send_task_events = True          # Enable monitoring<br>task_send_sent_event = True </pre><p>These small tweaks prevented infinite retry loops and improved stability.</p><p><strong>Insight:</strong> Default Celery configs aren’t production-safe, always set connection timeouts and retry caps.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3cd6dc883d4c3e3de1d242d0f9c11201/href">https://medium.com/media/3cd6dc883d4c3e3de1d242d0f9c11201/href</a></iframe><h3>8. Problem 3: Unexpectedly Slow Performance</h3><p>Even after stability fixes, 1M records took <strong>6.2 days</strong> instead of the expected <strong>5.4 days</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*xoMW--Bvm6VqyMb3anARfw.png" /></figure><p>After analyzing logs, we found excessive retries:</p><ul><li>Implemented retry logic (up to 5 attempts) for:<br>- Context length exceeded errors<br>- Rate limit errors (429)<br>- Gateway/network errors (502, 503, 504)</li><li>Each retry had a 5–15 second random delay.</li><li>This resulted in <strong>~25% additional API calls</strong> (~750K extra requests)</li></ul><h4>The Real Problem: Not Fast Enough for Scale</h4><p>Our client’s goal was <strong>1 million records per day</strong>. If successful, they wanted to scale to <strong>50–100 million records</strong>.</p><p>At 6.2 days per million records, this was impossible.</p><h4>First Idea</h4><p><strong>Create multiple IBM watsonx instances</strong>, each with 30 RPS, and distribute load across more workers.</p><ul><li>We weren’t even fully utilizing the existing 30 RPS limit</li><li>Temporary patch, not a long-term solution.</li></ul><p>We can consider this option later when we make sure that we are utilizing the full 30 RPS.</p><p>We also noticed something else: <strong>multithreading wasn’t scaling well.</strong></p><h4>Why 64 Threads Didn’t Help Much</h4><ul><li><strong>Blocking Calls:</strong> Each requests.get() waits for a response CPU idles during this 10–20 second wait</li><li><strong>Context Switching:</strong> Managing 64 threads adds overhead. OS constantly switches between 64 active threads</li><li><strong>Python’s GIL:</strong> Only one thread executes Python bytecode at a time. Even though threads release the GIL during I/O, managing it adds overhead</li><li><strong>Idle Waiting:</strong> Most threads were waiting, not computing.</li><li><strong>Limited Parallelism: </strong>64 threads can start requests together, but each still blocks until completion. When one finishes, another starts throughput doesn’t scale linearly</li></ul><pre>──────────────────────────────────────────────<br> BEFORE: Multithreading (Blocking)<br>──────────────────────────────────────────────<br>[Thread 1] → Call API → (waits)<br>[Thread 2] → Call API → (waits)<br>[Thread 3] → Call API → (waits)<br>→ CPU idle most of the time</pre><blockquote>In short, your 64 threads spent most of their time either <strong>waiting to be scheduled by the OS</strong> or <strong>waiting for the network response</strong> (the 10–20 second wait). So even with 64 threads, many threads were still waiting for network responses instead of running in parallel. By nature threads are <strong>blocking</strong> call, means it will wait until it finishes.</blockquote><p>It became clear: <strong>we needed Async I/O.</strong></p><h3>9. Solution 3: Switching to Async I/O</h3><p>The ibm_watsonx_ai library provides an asynchronous chat method called achat():</p><pre>response = await generation_model.achat(<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}]<br>)</pre><h4>First Attempt Failed</h4><p>We ran a load test with:</p><ul><li>1,000 records</li><li>600 concurrent calls</li><li>Semaphore to control concurrency</li></ul><p><strong>Result:</strong> Runtime <strong>increased</strong> to 24 minutes! 😱</p><p>The culprit? After investigation, we found that the ibm_watsonx_ai library <strong>wasn&#39;t using the httpx async client by default</strong>. Review the watsonx client library documentation <a href="https://ibm.github.io/watsonx-ai-python-sdk/v1.4.0/httpx_configuration.html">here</a>.</p><h4>Fixing the Async Setup</h4><p>We had to:</p><ol><li>Initialize the httpx async client manually.</li><li>Pass it to the model constructor.</li><li>Prevent model re-initialization on every call</li></ol><pre>_worker_local = threading.local()<br><br>@worker_process_init.connect<br>def init_worker_process(**kwargs):<br>    &quot;&quot;&quot;Initialize resources once per worker process.&quot;&quot;&quot;<br>    logger.info(f&quot;Initializing worker process {os.getpid()}&quot;)<br>    <br>    # Configure httpx for optimal async performance<br>    timeout = httpx.Timeout(<br>        connect=10.0, <br>        read=600.0,      # Long timeout for LLM responses<br>        write=10.0, <br>        pool=10.0<br>    )<br>    limits = httpx.Limits(<br>        max_connections=200,           # ← High concurrency support<br>        max_keepalive_connections=50   # ← Reuse connections<br>    )<br>    <br>    # Initialize async httpx client<br>    _worker_local.async_httpx_client = httpx.AsyncClient(<br>        timeout=timeout, <br>        limits=limits<br>    )<br>    <br>    # Initialize API client with async support<br>    _worker_local.api_client = APIClient(<br>        credentials, <br>        async_httpx_client=_worker_local.async_httpx_client<br>    )<br>    <br>    # Add rate limiter to respect 30 RPS<br>    _worker_local.rate_limiter = RateLimiter(rate=30, per=1)<br>    <br>    # Pre-initialize model ONCE (not per call!)<br>    model_id = os.getenv(<br>        &quot;WX_MODEL_ID&quot;, <br>        &quot;meta-llama/llama-3-3-70b-instruct&quot;<br>    )<br>    project_id = os.environ[&quot;WATSONX_PROJECT_ID&quot;]<br>    <br>    _worker_local.llm_model = ModelInference(<br>        model_id=model_id,<br>        api_client=_worker_local.api_client,<br>        project_id=project_id<br>    )</pre><p>We also encountered errors like:</p><pre>RuntimeError: Event loop is closed</pre><p>This happened because Celery reuses worker processes, and asyncio.run() creates a new event loop each time.</p><p><strong>Fix:</strong><br> We initialized the loop once per worker and reused it safely within each task execution.</p><h4>Key Optimizations</h4><ol><li><strong>Async httpx client</strong> with high connection limits (200 concurrent)</li><li><strong>Model initialized once</strong> per worker (not per API call)</li><li><strong>Rate limiter</strong> to prevent 429 errors</li><li><strong>Semaphore</strong> to manage in-flight requests.</li></ol><h4>The Result</h4><p>After these optimizations, throughput improved by 3<strong>×</strong>.</p><ul><li>Stable execution</li><li>No rate-limit errors</li><li>Efficient resource usage</li><li>Smooth observability via <strong>Flower</strong>, <strong>Prometheus</strong>, and <strong>Grafana</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e4f415842b3bdbf66ebb04f196aa431/href">https://medium.com/media/6e4f415842b3bdbf66ebb04f196aa431/href</a></iframe><h4>Comparison</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*h-2m4LW8ozveR3j_WSlSGA.png" /></figure><h3>10. Why Async I/O Works So Well</h3><p>Async I/O uses <strong>non-blocking sockets</strong> and <strong>event-driven scheduling</strong>.</p><ul><li>Linux uses epoll</li><li>macOS uses kqueue</li><li>Windows uses IOCP</li></ul><p>These OS mechanisms <strong>notify your app when I/O is ready</strong>, so you don’t waste time waiting.</p><pre>──────────────────────────────────────────────<br> AFTER: Async I/O (Non-Blocking)<br>──────────────────────────────────────────────<br>[Single Event Loop]<br> ├── Await API 1<br> ├── Await API 2<br> ├── Await API 3<br> └── Await API 4<br>→ No waiting — all requests overlap efficiently</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*7B9WNclVRqGN0aM9-J1SWw.png" /><figcaption>Asynchronous call</figcaption></figure><h4>What Happens Behind the Scenes</h4><p>When you call:</p><pre>await httpx.get(&quot;https://example.com&quot;)</pre><p>Here’s the step-by-step process:</p><ol><li><strong>Socket becomes non-blocking</strong>send() and recv() return immediately</li><li><strong>Event loop registers with OS </strong>“Tell me when this socket is ready”</li><li><strong>Coroutine yields </strong>Pauses and gives control back to event loop</li><li><strong>Event loop polls OS</strong> “Which sockets have data?” (via epoll_wait())</li><li><strong>OS notifies when ready</strong> “Socket #7 has data!”</li><li><strong>Event loop resumes coroutine</strong> Continues from where it paused</li><li><strong>Meanwhile </strong>Other coroutines (sockets #8, #9, etc.) are running</li></ol><pre>──────────────────────────────────────────────<br> AFTER: Async I/O (Non-Blocking)<br>──────────────────────────────────────────────<br>[Single Event Loop]<br> ├── Await API 1<br> ├── Await API 2<br> ├── Await API 3<br> └── Await API 4<br>→ No waiting — all requests overlap efficiently</pre><h4>How Many Requests Can One Thread Handle?</h4><p><strong>Potentially thousands</strong>, because:</p><ul><li>The thread never blocks on I/O</li><li>Operations happen asynchronously in the kernel.</li><li>Only limits are system memory and OS settings (ulimit -n)</li></ul><p>This is why frameworks like <strong>FastAPI</strong>, <strong>aiohttp</strong>, and <strong>httpx</strong> can handle massive concurrency without creating hundreds of threads.</p><h3>11. Final Lessons Learned</h3><pre>| Challenge       | Key Learning                        |<br>| --------------- | ----------------------------------- |<br>| Rate limits     | Use a distributed rate limiter      |<br>| Celery hangs    | Always set broker timeouts          |<br>| Slow throughput | Use async I/O for I/O-bound tasks   |<br>| API overhead    | Reuse clients/models within workers |<br>| Monitoring      | Always include observability tools  |</pre><h3>12. Key Takeaways</h3><ul><li><strong>Async I/O beats threads</strong> for I/O-bound LLM workloads.</li><li><strong>Pre-initialize models and clients</strong> to cut overhead.</li><li><strong>Rate limiting</strong> is essential for API stability.</li><li><strong>Celery + Async I/O</strong> is an ideal combo for batch LLM inference.</li><li><strong>3× performance gain</strong> achieved with the same infrastructure.</li></ul><h3>Conclusion</h3><p>Scaling LLM inference isn’t just about more compute it’s about smarter concurrency.</p><p>If you’re building large-scale LLM pipelines, try combining <strong>Celery + Async I/O</strong> with IBM watsonx. It’s a powerful, developer-friendly approach that balances performance and stability.</p><h3>Reference:</h3><ul><li><a href="https://ibm.github.io/watsonx-ai-python-sdk/v1.4.0/index.html">https://ibm.github.io/watsonx-ai-python-sdk/v1.4.0/index.html</a></li><li><a href="https://docs.python.org/3/howto/a-conceptual-overview-of-asyncio.html">https://docs.python.org/3/howto/a-conceptual-overview-of-asyncio.html</a></li><li><a href="https://docs.celeryq.dev/en/v5.5.3/getting-started/introduction.html">https://docs.celeryq.dev/en/v5.5.3/getting-started/introduction.html</a></li><li><a href="https://cloud.ibm.com/apidocs/watsonx-ai">https://cloud.ibm.com/apidocs/watsonx-ai</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db890b636281" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/processing-millions-of-records-on-ibm-watsonx-db890b636281">Processing Millions of Records on IBM watsonx</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Provision PostgreSQL on an OpenShift Cluster]]></title>
            <link>https://medium.com/@himadricuet/how-to-provision-postgresql-on-an-openshift-cluster-46aaa70b4f09?source=rss-b09579f13f8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/46aaa70b4f09</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[openshift-operators]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Himadri Talukder]]></dc:creator>
            <pubDate>Mon, 17 Nov 2025 16:56:51 GMT</pubDate>
            <atom:updated>2026-01-02T17:39:57.665Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>A clear, step-by-step guide for beginners and practitioners</em></h4><p>Provisioning PostgreSQL in OpenShift is simple once your OperatorHub is healthy and the Crunchy Data PostgreSQL Operator is installed. In this guide, I’ll walk through everything — from checking OperatorHub to deploying your first Postgres cluster.</p><p>Let’s get started.</p><h3>Step 1 — Make Sure OperatorHub is Working</h3><p>Before installing PostgreSQL, OpenShift must have a functioning <strong>OperatorHub</strong>.<br> If OperatorHub is enabled, your OpenShift console will show all the operators you can install.</p><p>If you instead see this error:</p><blockquote><strong><em>No OperatorHub items found. Check that the OperatorHub is running and that you have created a valid CatalogSource.</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yfloHd4aYfmdd3bk4_MX1g.png" /></figure><p>…it means OperatorHub isn’t configured correctly.</p><p>Check if OperatorHub exists</p><p>Run:</p><pre>oc get operatorhub cluster</pre><p>if you see:</p><pre>NAME       AGE<br>cluster    ...</pre><p>OperatorHub is enabled. ✔️</p><p>If it’s missing, recreate it:</p><pre>cat &lt;&lt;EOF | oc apply -f -<br>apiVersion: config.openshift.io/v1<br>kind: OperatorHub<br>metadata:<br>  name: cluster<br>spec:<br>  disableAllDefaultSources: false<br>EOF</pre><p>This brings back all Red Hat default sources.</p><h3>Step 2 — Check Your CatalogSources</h3><p>CatalogSources tell OperatorHub where to fetch operator metadata.</p><p>Run:</p><pre>oc get catalogsources -n openshift-marketplace</pre><p>You should see something like:</p><pre>NAME                          DISPLAY               TYPE   PUBLISHER   AGE<br>redhat-operators              Red Hat Operators     grpc   Red Hat     3d<br>certified-operators           Certified Operators   grpc   Red Hat     3d<br>community-operators           Community Operators   grpc   Red Hat     3d<br>redhat-marketplace            Red Hat Marketplace   grpc   Red Hat     3d</pre><p>If they’re missing, OperatorHub can’t display anything.</p><h3>Step 3 — Restore Missing CatalogSources</h3><p>Option A — Restore individually:</p><pre>oc apply -f https://raw.githubusercontent.com/openshift/cluster-samples-operator/master/assets/operatorhub/defaults/catalogsource-redhat-operators.yaml<br>oc apply -f https://raw.githubusercontent.com/openshift/cluster-samples-operator/master/assets/operatorhub/defaults/catalogsource-certified-operators.yaml<br>oc apply -f https://raw.githubusercontent.com/openshift/cluster-samples-operator/master/assets/operatorhub/defaults/catalogsource-community-operators.yaml</pre><p>Option B — Restore all defaults:</p><pre>oc patch operatorhub cluster --type merge --patch &#39;{&quot;spec&quot;:{&quot;disableAllDefaultSources&quot;:false}}&#39;</pre><h3>Step 4 — Verify Marketplace Pods</h3><p>Run:</p><pre>oc get pods -n openshift-marketplace</pre><p>You should see pods like:</p><ul><li>marketplace-operator-xxxxx</li><li>redhat-operators-xxxxx</li><li>certified-operators-xxxxx</li></ul><p>If these exist, OperatorHub is good to go.</p><p>Or you can check in the OpenShift web console and see the page like below from the Operators-&gt;OperatorHub menu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*5pRGv0d0dU6Xh5b3ut6WoQ.png" /></figure><h3>🐘 Now Let’s Install PostgreSQL (Crunchy Data Operator)</h3><p>There are two ways to install Crunchy Data Postgres Operator (PGO):</p><h3>Option 1: Install Using the OpenShift Web Console (Easy)</h3><ol><li><strong>Open the OpenShift Console</strong></li><li>Go to <strong>Operators → OperatorHub</strong></li><li>Search for <strong>crunchy</strong></li><li>Click <strong>Crunchy Postgres for Kubernetes</strong> (the certified one)</li><li>Click <strong>Install</strong></li><li>Choose:<br>- <strong>All namespaces</strong> OR <strong>Specific namespace <br>- </strong>Select <strong>channel</strong> and <strong>approval strategy</strong></li><li>Click <strong>Install</strong></li></ol><p>Verify the operator:</p><pre>oc get csv -n openshift-operators | grep postgres<br>oc get pods -n openshift-operators | grep pgo</pre><p>You’re now ready to create PostgreSQL clusters using the operator!</p><h3>Option 2: Install Using CLI (More DevOps-friendly)</h3><p>This approach installs PGO using YAML definitions.</p><h4>Step 1 — Create a Namespace</h4><p>Recommended namespace: postgres-operator</p><pre>oc new-project postgres-operator</pre><h4>Step 2 — Create OperatorGroup</h4><p>Create operatorgroup.yaml:</p><pre>apiVersion: operators.coreos.com/v1<br>kind: OperatorGroup<br>metadata:<br>  name: postgres-operator-group<br>  namespace: postgres-operator<br>spec:<br>  targetNamespaces:<br>    - postgres-operator</pre><p>Apply it:</p><pre>oc apply -f operatorgroup.yaml</pre><h4>Step 3 — Create Subscription</h4><p>Create subscription.yaml:</p><pre>apiVersion: operators.coreos.com/v1alpha1<br>kind: Subscription<br>metadata:<br>  name: postgres-operator<br>  namespace: postgres-operator<br>spec:<br>  channel: stable<br>  name: crunchy-postgres-operator<br>  source: redhat-operators<br>  sourceNamespace: openshift-marketplace</pre><p>Apply it:</p><pre>oc apply -f subscription.yaml</pre><h3>Step 4 — Verify Installation</h3><pre>oc get csv -n postgres-operator</pre><p>Look for:</p><pre>crunchy-postgres-operator.v5.x.x   Succeeded</pre><p><strong>PGO is now installed!</strong></p><h3>Deploy Your First PostgreSQL Cluster</h3><p>Here’s a simple working example and run in the command line:</p><pre>oc project postgres-operator</pre><pre>cat &lt;&lt;EOF | oc apply -f -<br>apiVersion: postgres-operator.crunchydata.com/v1beta1<br>kind: PostgresCluster<br>metadata:<br>  name: hippo-pg<br>  namespace: postgres-operator<br>  finalizers:<br>    - postgres-operator.crunchydata.com/finalizer<br>  labels:<br>    app: hippo-pg<br>spec:<br>  port: 5432<br>  users:<br>    - databases:<br>        - postgres<br>      name: postgres<br>      options: SUPERUSER<br>      password:<br>        type: AlphaNumeric<br>    - databases:<br>        - postgres<br>      name: admin<br>      options: SUPERUSER<br>      password:<br>        type: AlphaNumeric<br>  proxy:<br>    pgBouncer:<br>      port: 5432<br>      replicas: 3<br>      service:<br>        type: ClusterIP<br>  monitoring:<br>    pgmonitor:<br>      exporter:<br>        image: &#39;registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:ubi8-5.6.1-0&#39;<br>        resources:<br>          limits:<br>            cpu: 100m<br>            memory: 128Mi<br>          requests:<br>            cpu: 50m<br>            memory: 64Mi<br>  backups:<br>    pgbackrest:<br>      global:<br>        repo1-retention-full: &#39;2&#39;<br>        repo1-retention-diff: &#39;3&#39;<br>        archive-push-queue-max: 4GiB<br>        repo1-retention-archive-type: &#39;diff&#39;<br>        repo1-retention-archive: &#39;2&#39;<br>      repos:<br>        - name: repo1<br>          schedules:<br>              full: &quot;0 2 * * 0&quot;           # Full backup every Sunday at 2 AM<br>              differential: &quot;0 2 * * 1-6&quot; # Differential backup Mon-Sat at 2 AM<br>              incremental: &quot;0 */8 * * *&quot;  # Incremental backup every 8 hours<br>          volume:<br>            volumeClaimSpec:<br>              accessModes:<br>                - ReadWriteOnce<br>              resources:<br>                requests:<br>                  storage: 100Gi<br>  service:<br>    type: ClusterIP<br>  patroni:<br>    dynamicConfiguration:<br>      postgresql:<br>        parameters:<br>          max_connections: 100<br>          shared_buffers: 256MB<br>      synchronous_mode: false<br>    leaderLeaseDurationSeconds: 30<br>    port: 8008<br>    switchover:<br>      enabled: true<br>      type: Switchover<br>    syncPeriodSeconds: 10<br>  instances:<br>    - dataVolumeClaimSpec:<br>        accessModes:<br>          - ReadWriteOnce<br>        resources:<br>          requests:<br>            storage: 50Gi<br>      name: instance1<br>      replicas: 3<br>      resources:<br>        limits:<br>          cpu: 1<br>          memory: 1Gi<br>        requests:<br>          cpu: 500m<br>          memory: 512Mi<br>  postgresVersion: 16<br>EOF</pre><h4>Lets create the pgAdmin database browser</h4><p>Create a secret for pgAdmin password</p><pre>export PGADMIN_PASSWORD=&#39;mypgadminpassword&#39;</pre><p>Create a secret for pgAdmin password</p><pre>oc create secret generic pgadmin-password-secret \<br>  --from-literal=pgadmin-admin-password=$PGADMIN_PASSWORD</pre><p>Run the below command to create pgAdmin instance</p><pre>cat &lt;&lt;EOF | oc apply -f -<br>apiVersion: postgres-operator.crunchydata.com/v1beta1<br>kind: PGAdmin<br>metadata:<br>  name: hippo-pgadmin<br>  namespace: postgres-operator<br>spec:<br>  dataVolumeClaimSpec:<br>    accessModes:<br>      - ReadWriteOnce<br>    resources:<br>      requests:<br>        storage: 1Gi<br>  serverGroups:<br>    - name: Crunchy Postgres for Kubernetes<br>      postgresClusterSelector: {}<br>  serviceName: hippo-pgadmin<br>  users:<br>    - passwordRef:<br>        key: pgadmin-admin-password<br>        name: pgadmin-password-secret<br>      role: Administrator<br>      username: admin@example.com<br>EOF</pre><p>Create a route for the pgAdmin database explorer.</p><pre>cat &lt;&lt;EOF | oc apply -f -<br>kind: Route<br>apiVersion: route.openshift.io/v1<br>metadata:<br>  name: route-pgadmin<br>  namespace: postgres-operator<br>spec:<br>  to:<br>    kind: Service<br>    name: hippo-pgadmin<br>    weight: 100<br>  port:<br>    targetPort: pgadmin-port<br>  tls:<br>    termination: edge<br>    insecureEdgeTerminationPolicy: Redirect<br>  wildcardPolicy: None<br>EOF</pre><p>Ge the pgAdmin url to login to the database explorer.</p><pre>oc get routes route-pgadmin -o jsonpath=&#39;https://{.spec.host}&#39; </pre><ul><li>username: admin@example.com</li><li>password: $PGADMIN_PASSWORD</li></ul><p>Then create a database connection after login to the pgAdmin explorer.</p><h3>Final Thoughts</h3><p>That’s it — you now have:</p><ul><li>A working OperatorHub</li><li>Crunchy Data PostgreSQL Operator installed</li><li>Your first Postgres cluster running in OpenShift</li></ul><p>Adjust all the configuration and settings based on your requirements. This setup gives you production-ready PostgreSQL with backups, high availability, monitoring, and seamless upgrades — all managed by Kubernetes.</p><blockquote><strong><em>In the next post of this series, we’ll take a closer look at PostgreSQL architecture using the PostgreSQL Operator and walk through how backup and recovery work in practice. Check out the next article using the link below.</em></strong></blockquote><p><a href="https://medium.com/@himadricuet/crunchy-postgresql-operator-architecture-backup-and-recovery-fe7ef4f20af9">Crunchy PostgreSQL Operator Architecture &amp; Backup and Recovery</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46aaa70b4f09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Production-Ready Observability for vLLM]]></title>
            <link>https://medium.com/ibm-data-ai/building-production-ready-observability-for-vllm-a2f4924d3949?source=rss-b09579f13f8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2f4924d3949</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Himadri Talukder]]></dc:creator>
            <pubDate>Thu, 26 Jun 2025 15:19:50 GMT</pubDate>
            <atom:updated>2025-07-09T15:28:00.228Z</atom:updated>
            <content:encoded><![CDATA[<h4>Monitor, trace, and visualize vLLM using OpenTelemetry, Prometheus, Grafana, and Jaeger for robust, scalable, and LLM operations.</h4><p>Picture this: You’ve just deployed a shiny new Large Language Model using vLLM, generating responses faster than you ever imagined. But then, during peak traffic, something goes wrong. Responses slow to a crawl, costs spiral out of control, and you’re left scrambling to figure out what happened.</p><p>Sound familiar? You’re not alone. The difference between a successful LLM deployment and a costly disaster is <strong>observability</strong>.</p><p>Think of observability as your LLM’s personal health monitor. Just like you wouldn’t run a marathon without tracking your heart rate, you shouldn’t run an LLM in production without monitoring its vital signs.</p><h3>Why Observability Matters for LLMs</h3><p>LLMs introduce unique challenges in observability that go beyond traditional software monitoring. Let’s break these down:</p><h4>1. Token Usage and Cost</h4><p>Every token generated has a cost. Monitoring token usage helps in cost optimization and capacity planning.</p><h4>2. Latency and Throughput</h4><p>Fast response times are crucial for user experience. Observing metrics like <strong>latency per token/request</strong> and <strong>throughput (tokens/second)</strong> ensures smooth operations.</p><h4>3. GPU Utilization</h4><p>LLMs are resource-intensive. Efficient GPU utilization is key to maximizing hardware investment.</p><h4>4. Model Performance</h4><p>Operational metrics can hint at token generation errors or other issues, guiding improvements.</p><h4>5. Distributed Tracing</h4><p>Complex architectures — like Retrieval-Augmented Generation (RAG) or multi-agent pipelines — require end-to-end tracing to quickly identify bottlenecks and failure points.</p><h3>Building Your LLM Mission Control Center</h3><p>Let’s combine a powerful open-source stack that works in harmony</p><ul><li><strong>RAG application:</strong> A FastAPI app simulating a RAG pipeline, instrumented for tracing and metrics.</li><li><strong>vLLM</strong>: <strong>Your fast inference engine — the star of the show.</strong> The latest version of vLLM now supports OpenTelemetry and exposes a metrics endpoint that Prometheus can scrape. <a href="https://github.com/vllm-project/vllm">Learn more about vLLM here.</a></li><li><strong>OpenTelemetry</strong>: The universal translator for telemetry data. OpenTelemetry (OTel) is an open-source observability framework that provides a standardized way to collect, generate, and export telemetry data (metrics, logs, and traces) from cloud-native applications and infrastructure.</li><li><strong>Jaeger</strong>: Your request, detective (traces every step). Uses OTel to show tracing.</li><li><strong>Prometheus</strong>: The data collector that never sleeps</li><li><strong>Grafana</strong>: Your beautiful, customizable dashboard</li></ul><p>Here’s a high-level overview of how these components interact:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C4_zcWTKkNFEL0Ij6b7F8w.png" /><figcaption>vLLM observability</figcaption></figure><h3>Setting Up Your Monitoring Stack (The Easy Way)</h3><p>Don’t worry — you don’t have to set everything up manually. We chose Docker Compose to handle the heavy lifting and make it easy for you to try out the stack.</p><h4>Prerequisites</h4><p>Before diving in, ensure you have:</p><ul><li><strong>Docker or Podman</strong> is installed and running</li><li>A VM with a GPU</li><li><strong>Basic familiarity</strong> with vLLM, Prometheus, and Grafana concepts</li></ul><h4>Project Structure</h4><p>First, let’s organize our files: A complete code can be found in the GitHub repo <a href="https://github.com/hitalukder/mlops/tree/main/vLLM/observability">here</a></p><pre>.<br>├── docker-compose.yml<br>├── prometheus.yml<br>├── rag-app/<br>│   ├── Dockerfile<br>│   └── app.py<br>├── vllm-server/<br>│   └── Dockerfile<br>└── grafana/<br>    ├── provisioning/<br>    │   ├── datasources/<br>    │   │   └── prometheus.yml<br>    │   └── dashboards/<br>    │       └── dashboard.yml<br>    └── dashboards/<br>        ├── vllm-dashboard.json<br>        └── rag-dashboard.json</pre><h4>Step 1: The Docker Compose Configuration</h4><p>Here’s your complete docker-compose.yml that orchestrates the entire monitoring symphony:</p><pre>services:<br>  ## Mocked RAG application (embedding + vector search mocked)<br>  rag-app:<br>    build:<br>      context: rag-app  # Builds the image using Dockerfile from the rag-app directory<br>    container_name: rag-app<br>    ports:<br>      - &quot;8002:8002&quot;  # FastAPI main API endpoint<br>      - &quot;8001:8001&quot;  # Prometheus metrics endpoint<br>    environment:<br>      - OTEL_EXPORTER_OTLP_ENDPOINT=grpc://jaeger:4317  # Send traces to Jaeger<br>    depends_on:<br>      - jaeger<br>      - prometheus<br><br>  ## vLLM server for LLM inference<br>  vllm-server:<br>    build: vllm-server  # Builds the image from the vllm-server directory<br>    container_name: vllm-server<br>    ports:<br>      - &quot;8000:8000&quot;  # vLLM inference API port<br>    environment:<br>      - MODEL_NAME=facebook/opt-125m  # Name of the model to serve<br>      - OTEL_SERVICE_NAME=vllm-server  # Service name used in tracing<br>      - NVIDIA_VISIBLE_DEVICES=all  # Make all GPUs visible to the container<br>      - NVIDIA_DRIVER_CAPABILITIES=compute,utility  # Required for GPU compute workloads<br>      - OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://jaeger:4317  # Send traces to Jaeger<br>      - OTEL_EXPORTER_OTLP_TRACES_INSECURE=true  # Allow insecure (non-TLS) trace export<br>      - VLLM_LOGGING_LEVEL=DEBUG  # Set logging level<br>    command: vllm serve facebook/opt-125m --otlp-traces-endpoint=grpc://jaeger:4317  # Start vLLM server<br>    depends_on:<br>      - jaeger<br>    deploy:<br>      resources:<br>        reservations:<br>          devices:<br>            - driver: nvidia  # Use the NVIDIA runtime<br>              count: all      # Use all available GPUs<br>              capabilities: [gpu]  # Request GPU capability<br><br>  ## Jaeger: distributed tracing backend<br>  jaeger:<br>    image: jaegertracing/all-in-one:1.57  # Jaeger all-in-one image<br>    container_name: jaeger<br>    ports:<br>      - &quot;16686:16686&quot;   # Jaeger UI<br>      - &quot;4317:4317&quot;     # OTLP gRPC endpoint<br>      - &quot;4318:4318&quot;     # OTLP HTTP endpoint<br>      - &quot;6831:6831/udp&quot; # Jaeger agent UDP port<br>      - &quot;6832:6832/udp&quot;<br>      - &quot;14250:14250&quot;<br>      - &quot;14268:14268&quot;<br>      - &quot;14269:14269&quot;<br>      - &quot;5778:5778&quot;<br>      - &quot;9411:9411&quot;     # Zipkin compatible port<br><br>  ## Prometheus: metrics collector<br>  prometheus:<br>    image: prom/prometheus<br>    container_name: prometheus<br>    volumes:<br>      - ./prometheus.yml:/etc/prometheus/prometheus.yml  # Mount custom config<br>    ports:<br>      - &quot;9090:9090&quot;  # Prometheus UI and API<br><br>  ## Grafana: metrics dashboard and visualization<br>  grafana:<br>    image: grafana/grafana<br>    container_name: grafana<br>    ports:<br>      - &quot;3000:3000&quot;  # Grafana dashboard UI<br>    environment:<br>      - GF_SECURITY_ADMIN_USER=admin  # Default admin username<br>      - GF_SECURITY_ADMIN_PASSWORD=admin  # Default admin password<br>    volumes:<br>      - grafana-storage:/var/lib/grafana  # Persist Grafana data<br>      - ./grafana/dashboards:/var/lib/grafana/dashboards  # Mount custom dashboards<br>      - ./grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards  # Auto-provision dashboards<br>      - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources  # Auto-provision data sources<br>    depends_on:<br>      - prometheus<br>      - jaeger<br><br>volumes:<br>  grafana-storage:  # Named volume for persistent Grafana data</pre><p>This setup is more than just monitoring — it’s an RAG application with observability baked in:</p><p><strong>🔍 RAG Application Features:</strong></p><ul><li>Exposes both API endpoints (port 8002) and metrics (port 8001)</li><li>Sends traces to Jaeger for end-to-end visibility</li><li>Simulates a real-world RAG pipeline</li></ul><p><strong>⚡ vLLM Configuration Highlights:</strong></p><ul><li>Uses the lightweight facebook/opt-125m model (perfect for testing)</li><li>GPU support with NVIDIA runtime</li><li>Debug logging is enabled for troubleshooting</li><li>Direct OTLP tracing to Jaeger</li></ul><p><strong>📊 Monitoring Integration:</strong></p><ul><li>Jaeger collects traces from both the RAG app and vLLM by OTLP</li><li>Prometheus scrapes metrics from multiple endpoints using the /metrics endpoints exposed by the RAG app and vLLM.</li><li>Grafana provides unified dashboards with persistent storage</li><li>Jaeger UI visualizes the end-to-end tracing.</li></ul><h4>Step 3: Configure Prometheus Scraping</h4><p>Your prometheus.yml should target both services:</p><pre>global:<br>  scrape_interval: 5s<br><br>scrape_configs:<br>  - job_name: &#39;rag-app&#39;<br>    static_configs:<br>      - targets: [&#39;rag-app:8001&#39;]<br><br>  - job_name: &#39;vllm-server&#39;<br>    static_configs:<br>      - targets: [&#39;vllm-server:8000&#39;]</pre><p>Prometheus will collect metrics every 5 seconds from these endpointshttp://vllm-server:8000/metrics and http://rag-app:8001/metrics</p><h4>Step 4: Grafana Provisioning</h4><p>Grafana can be pre-configured to automatically connect to Prometheus and load dashboards. This involves:</p><ul><li><strong>datasources/datasource.yml:</strong> Defines Prometheus as a data source, pointing to http://prometheus:9090 within the Docker network.</li></ul><pre># Save this as ./grafana/provisioning/datasources/prometheus.yml<br>apiVersion: 1<br><br>datasources:<br>  - name: prometheus<br>    type: prometheus<br>    access: proxy<br>    url: http://prometheus:9090<br>    isDefault: true<br>    editable: true</pre><ul><li><strong>dashboards/dashboard.yml:</strong> Instructs Grafana to load dashboard definitions from .json files within the mounted dashboards directory. I have provided two predefined dashboards in the <a href="https://github.com/hitalukder/mlops/tree/main/vLLM/observability/grafana/dashboards">GitHub repo</a>, one for the vLLM server and the other for the RAG application.</li></ul><h3>Step 4: Launch Your Monitoring Empire</h3><p>Build the docker file</p><pre>docker compose build</pre><p>Run the entire stack with:</p><pre>docker compose up -d</pre><p>Within minutes, you’ll have:</p><ul><li><strong>RAG App</strong>: <a href="http://localhost:8002/">http://localhost:8002</a> (your application)</li><li><strong>vLLM Server</strong>: <a href="http://localhost:8000/">http://localhost:8000</a> (model serving)</li><li><strong>Jaeger UI</strong>: <a href="http://localhost:16686/">http://localhost:16686</a> (trace visualization)</li><li><strong>Prometheus</strong>: <a href="http://localhost:9090/">http://localhost:9090</a> (metrics storage)</li><li><strong>Grafana</strong>: <a href="http://localhost:3000/">http://localhost:3000</a> (dashboards — admin/admin)</li></ul><h3>Expected Results</h3><p><strong>Successful vLLM startup logs:</strong></p><pre>docker compose ps<br>NAME          IMAGE                            COMMAND                  SERVICE       CREATED          STATUS          PORTS<br>grafana       grafana/grafana                  &quot;/run.sh&quot;                grafana       25 seconds ago   Up 24 seconds   0.0.0.0:3000-&gt;3000/tcp, :::3000-&gt;3000/tcp<br>jaeger        jaegertracing/all-in-one:1.57    &quot;/go/bin/all-in-one-…&quot;   jaeger        25 seconds ago   Up 24 seconds   0.0.0.0:4317-4318-&gt;4317-4318/tcp, :::4317-4318-&gt;4317-4318/tcp, 0.0.0.0:5778-&gt;5778/tcp, :::5778-&gt;5778/tcp, 0.0.0.0:9411-&gt;9411/tcp, :::9411-&gt;9411/tcp, 0.0.0.0:14250-&gt;14250/tcp, :::14250-&gt;14250/tcp, 0.0.0.0:14268-14269-&gt;14268-14269/tcp, :::14268-14269-&gt;14268-14269/tcp, 0.0.0.0:16686-&gt;16686/tcp, :::16686-&gt;16686/tcp, 5775/udp, 0.0.0.0:6831-6832-&gt;6831-6832/udp, :::6831-6832-&gt;6831-6832/udp<br>prometheus    prom/prometheus                  &quot;/bin/prometheus --c…&quot;   prometheus    25 seconds ago   Up 24 seconds   0.0.0.0:9090-&gt;9090/tcp, :::9090-&gt;9090/tcp<br>rag-app       vllm_opentelemetry-rag-app       &quot;uvicorn rag_app:app…&quot;   rag-app       25 seconds ago   Up 24 seconds   0.0.0.0:8001-8002-&gt;8001-8002/tcp, :::8001-8002-&gt;8001-8002/tcp<br>vllm-server   vllm_opentelemetry-vllm-server   &quot;vllm serve facebook…&quot;   vllm-server   25 seconds ago   Up 24 seconds   0.0.0.0:8000-&gt;8000/tcp, :::8000-&gt;8000/tcp</pre><p><strong>Prometheus targets should show:</strong></p><ul><li>rag-app:8001 - UP</li><li>vllm-server:8000 - UP</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GwYP0cFie8ZWnx2_lfmojA.png" /><figcaption>Target server status</figcaption></figure><p><strong>Jaeger tracing display:</strong></p><p>This sample trace shows the end-to-end flow from the RAG app to embedding, vector database, LLM generation, and finally the vLLM server. Each service and its components are clearly shown as spans.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LPsBHfdQC7TU_oaZa42iTA.png" /><figcaption>Jaeger tracing</figcaption></figure><p><strong>Grafana should display:</strong></p><ul><li>Pre-configured dashboards for vLLM and RAG metrics</li><li>Real-time metrics updating. The vLLM dashboard should show all the following metrics. You can configure your customized panel.<br>- Request latency<br>- Token Throughput<br>- Time Per Output Token Latency<br>- Time To First Token Latency<br>- Request Prompt Length<br>- Request Generation Length<br>- Requests Prefill and Decode Time<br>- Max Generation Token in Sequence Group</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BPtg5zcY-M-mvQA8pHO55w.png" /><figcaption>Two dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L2cghQTOCZKfIa81qLMkcw.png" /><figcaption>vLLM dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HlSupnP8bMvI4fEYYOvADA.png" /><figcaption>vLLM dashboard</figcaption></figure><h3>Best Practices for Production</h3><ul><li><strong>Persistent Storage:</strong> Mount volumes for Prometheus and Grafana to preserve metrics and dashboard configs across restarts.</li><li><strong>Security:</strong> Change default passwords, secure network access, and consider HTTPS for all UIs.</li><li><strong>Scalability:</strong> For large deployments, explore distributed Prometheus (Thanos, Mimir) and scalable Jaeger backends (Elasticsearch, Cassandra). Use an OpenTelemetry Collector for robust ingestion.</li><li><strong>Alerting:</strong> Set alerts in Grafana to detect high latency, error spikes, or low throughput and notify your team proactively.</li><li><strong>Centralized Logging:</strong> Complement metrics and tracing with logging solutions like ELK or Loki for deeper debugging.</li><li><strong>Advanced Observability:</strong> Integrate modern LLM-focused tools such as Langsmith, Helicone, Instana, or traceloop to enhance your AI observability.</li></ul><h4>Source code:</h4><p><a href="https://github.com/hitalukder/mlops/tree/main/vLLM/observability">https://github.com/hitalukder/mlops/tree/main/vLLM/observability</a></p><h4>Resources</h4><ul><li><a href="https://github.com/vllm-project/vllm">vLLM GitHub</a></li><li><a href="https://opentelemetry.io/">OpenTelemetry</a></li><li><a href="https://www.jaegertracing.io/">Jaeger</a></li><li><a href="https://prometheus.io/">Prometheus</a></li><li><a href="https://grafana.com/">Grafana</a></li></ul><h3>Conclusion</h3><p>Observability transforms how we manage and optimize LLM deployments. By integrating vLLM with OpenTelemetry, Jaeger, Prometheus, and Grafana, you gain actionable insights that help diagnose issues and ensure smooth, cost-effective, and reliable operations at scale.</p><p>This observability stack does <strong>not</strong> detect challenges like model bias, hallucinations, harmful content, or personally identifiable information (PII). For those, consider governance tools such as IBM’s <a href="https://www.ibm.com/products/watsonx-governance">watsonx.governance.</a></p><p>Whether you’re an MLOps engineer or AI enthusiast, building a strong observability foundation is your first step to mastering LLM operations. Dive in, experiment, and unlock the full potential of vLLM!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2f4924d3949" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/building-production-ready-observability-for-vllm-a2f4924d3949">Building Production-Ready Observability for vLLM</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Watsonx Discovery Elasticsearch Vector Database Deployment on OpenShift with Helm]]></title>
            <link>https://medium.com/towards-generative-ai/automating-watsonx-discovery-elasticsearch-vector-database-deployment-on-openshift-with-helm-97df93112977?source=rss-b09579f13f8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/97df93112977</guid>
            <category><![CDATA[watsonx-discovery]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[helm]]></category>
            <dc:creator><![CDATA[Himadri Talukder]]></dc:creator>
            <pubDate>Thu, 20 Mar 2025 18:36:42 GMT</pubDate>
            <atom:updated>2025-03-21T14:08:07.735Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn how to leverage Elasticsearch’s vector search capabilities and streamline deployment with Helm for efficient AI applications and Elasticsearch sizing guide.</h4><h3>Introduction</h3><p>If you’re building an AI-powered search, RAG application, or recommendation system, vector similarity search is essential. Elasticsearch is a powerful open-source vector database that combines traditional text and vector search, improving accuracy and relevance. Setting up a production-ready Elasticsearch cluster is complex, requiring the configuration of multiple components and making installation challenging.</p><p>That’s where Helm comes in. This guide simplifies the process using Helm, a package manager for Kubernetes that can be used to automate Elasticsearch deployment. We’ll walk through provisioning an Elasticsearch vector database using Helm on an OpenShift cluster.</p><h3>Prerequisites</h3><p>Before you begin, ensure you have:</p><ul><li>An <strong>OpenShift Cluster</strong> with admin access</li><li><strong>OpenShift CLI (oc)</strong> installed (<a href="https://www.ibm.com/docs/en/guardium-insights/3.2.x?topic=azure-install-openshift-command-line-interface-cli">Download Here</a>)</li><li><strong>Helm</strong> installed on your local machine (<a href="https://helm.sh/docs/intro/install/">Installation</a>)</li></ul><h3>Why Elasticsearch for Vector Databases?</h3><p>Elasticsearch has evolved beyond traditional text search to support vector search, making it a powerful choice for AI and machine learning workloads. Here’s why it stands out:</p><ul><li><strong>Hybrid Search</strong>: Combines text and vector search for more accurate and relevant results. This means you get both precise keyword matching and semantic understanding.</li><li><strong>Fast Similarity Matching</strong>: Supports k-NN (approximate nearest neighbour) search for high-speed vector similarity lookups.</li><li><strong>Built-in Semantic Search</strong>: Includes Elastic Learned Sparse Encoder (ELSER) for advanced semantic search without needing custom embedding models.</li><li><strong>Custom Embeddings</strong>: You can use built-in models or deploy your own (e.g., Hugging Face or proprietary models) to generate embeddings and store them in Elasticsearch.</li><li><strong>Scalability</strong>: Its distributed architecture ensures efficient scaling across multiple nodes.</li><li><strong>High-Speed Indexing</strong>: Uses ingest pipelines and parallel bulk indexing to handle large-scale data efficiently.</li><li><strong>Easy Management</strong>: Managed via ECK operators on Kubernetes, automating scaling, upgrades, and resource management.</li><li><strong>Open-Source &amp; Flexible</strong>: It can be deployed in the cloud, on-premise, or hybrid environments, with customization options to fit your needs.</li><li><strong>Rich Ecosystem</strong>: Integrates with other Elasticsearch tools, providing a complete platform for search and analytics.</li></ul><p>However, it’s important to consider that specialized vector databases might offer advantages in specific areas like raw speed and handling extremely high-dimensional vectors. However, for many use cases, especially those requiring a balance of text and semantic search, Elasticsearch’s vector database presents a compelling solution.</p><h3>Choosing the Right Elasticsearch Nodes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*MFx02YZUFDq0zGSgZ-ItMQ.png" /></figure><blockquote>You can generate embeddings either outside or within the Elasticsearch cluster. If you choose to create embeddings inside the cluster, you will need an ML node. Your cluster must have at least one ML node to use machine learning features. The choice depends on your use case.</blockquote><p>Elasticsearch clusters consist of different node types:</p><ul><li><strong>Master Nodes</strong>: Manage cluster operations. They handle cluster state management, metadata handling, and index management.</li><li><strong>Data Nodes</strong>: Store and retrieve data</li><li><strong>Ingest Nodes</strong>: Preprocess data before indexing</li><li><strong>ML Nodes</strong> (if using ELSER ): Perform machine learning tasks</li></ul><h3>Cluster sizing:</h3><p>Have you ever felt like you’re playing a guessing game with your Elasticsearch cluster size? You’re not alone! Getting the perfect balance between performance and cost can be tricky. It&#39;s too small, and your searches are slow. Too big, and you’re wasting resources. Let’s walk through how to size your cluster like a pro, step by step.</p><p><strong>How Many Shards Do You <em>Really</em> Need?</strong></p><p>Shards are the heart of your Elasticsearch data, but how many is enough? Here’s a simple formula to get you started:</p><ul><li><strong><em>Number of Shards = (Source Data + Room to Grow) * (1 + Indexing Overhead) / Desired Shard Size</em></strong></li></ul><p><strong>Example:</strong></p><p>Let’s say you have 150GB of data and expect it to double (100% growth).</p><ul><li><strong><em>(150GB + 150GB) x 1.10 / 25GB = 13.2 (round up to 14 shards)</em></strong></li></ul><p><strong>Pro Tip:</strong> Remember, you can’t easily change the number of primary shards later, so plan ahead!</p><p><strong>How Much Storage Do You Need?</strong></p><p>Now, let’s figure out the minimum storage you’ll need.</p><ul><li><strong><em>Your Data x (1 + Number of Replicas) x 1.45</em></strong></li></ul><p><strong>Example (Continuing from above):</strong></p><ul><li>200 GB x (1 + 1) x 1.45 = 580 GB (approximately)</li></ul><p><strong>How Many Servers (Nodes) and What Kind?</strong></p><p>Think of nodes as the workhorses of your Elasticsearch cluster. Here’s how to size them:</p><ul><li><strong>For servers handling many searches and updates,</strong> <em>Use 2 CPU cores and 8GB of memory for every 100GB of storage.</em></li></ul><p><strong>Don’t Forget the JVM Heap Size!</strong></p><p>This is crucial. Set your Elasticsearch node’s memory (JVM heap size) to no more than half (max 32GB) of your server’s memory. Too much, and you’ll run into performance issues.</p><p><strong>Putting It All Together</strong></p><p>Let’s build a minimum setup with 4 data nodes, 1 replica per shard, 3 master nodes, and 4 ML nodes for high-performance vector database indexing and retrieval.</p><h4>Data Nodes:</h4><ul><li><strong>Total Storage:</strong> 580 GB (from our example)</li><li><strong>Compute &amp; Memory:</strong> 580 GB * ( 2 vCPU and 8 GB memory)/ 100 GB ≈ 12 vCPU and 46 GB of memory.</li><li><strong>Each Node:</strong> 3 CPUs, 12 GB memory, 150 GB storage.</li><li><strong>JVM Heap:</strong> With 12GB memory, set the JVM heap to 6 GB (remember the 50% rule).</li></ul><h4>Master Nodes:</h4><ul><li><strong>Compute &amp; Memory:</strong> For production environments, each node should have a minimum of 2-4 vCPU cores and 8 GB of memory.</li><li><strong>Total Storage: </strong>A starting point of 50GB to 100GB of storage is generally sufficient for most master node deployments.</li><li><strong>Each Node:</strong> 2-4 CPUs, 8GB memory, 50–100 GB storage.</li><li><strong>JVM Heap:</strong> With 8GB memory, set the JVM heap to 4GB (remember the 50% rule).</li></ul><h4>ML Nodes:</h4><blockquote>Please note that you will not need any ML node if you generate a vector outside the Elasticsearch cluster.</blockquote><ul><li><strong>4 ML Nodes</strong> (for embeddings): 16 CPUs, 32GB RAM per node</li><li>Use 16 threads per allocation, with 16 allocations to accommodate 16 simultaneous document embeddings. With 4 nodes you will get a 4*16 = 64 simultaneous document embedding.</li><li><strong>Total Storage: </strong>A starting point of 50GB to 100GB is generally sufficient.</li></ul><p><strong>Checklist for Sizing Your Cluster:</strong></p><ol><li>Calculate the number of shards.</li><li>Determine the minimum storage required.</li><li>Figure out the number of nodes and their configuration (CPU, memory, storage).</li><li>Set the JVM heap size correctly.</li><li>For production Elasticsearch clusters, it’s best practice to have dedicated master nodes and dedicated data nodes.</li><li>Combining roles can simplify setup and reduce resource requirements for small-scale development or testing environments.</li><li>Monitor your cluster’s performance and adjust as needed.</li><li>For a multi-zone setup, using the <strong>cx2.32x64 node flavour</strong> is recommended to accommodate all Elasticsearch nodes with different configurations.</li><li>To ensure high availability, <strong>use node affinity</strong> to distribute Elasticsearch nodes across specific machines.</li></ol><h3>Installing the ECK Operator on OpenShift</h3><p>Elastic Cloud on Kubernetes (ECK) uses the Kubernetes Operator pattern to make it easier to deploy and manage Elasticsearch, Kibana, APM Server, Enterprise Search, Beats, Elastic Agent, Elastic Maps Server, and Logstash on Kubernetes. For more information, follow the <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-overview.html">link</a>.</p><p><strong>Apply the ECK Operator Manifest:</strong></p><blockquote>If you already installed the ECK operator, please ignore the step below</blockquote><pre>oc create -f https://download.elastic.co/downloads/eck/2.16.1/crds.yaml<br>oc apply -f https://download.elastic.co/downloads/eck/2.16.1/operator.yaml</pre><h4>Applying for an Enterprise Trial License</h4><blockquote>If you have already applied for the license, please ignore the step below</blockquote><pre>oc apply -f - &lt;&lt;EOF<br>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: eck-trial-license<br>  namespace: openshift-operators<br>  labels:<br>    license.k8s.elastic.co/type: enterprise_trial<br>  annotations:<br>    elastic.co/eula: accepted<br>EOF</pre><p>Verify the Operator is Running:</p><pre>oc get pods -n elastic-system</pre><p>Once the operator is running, proceed to deploy Elasticsearch.</p><h3>Deploying Elasticsearch Vector Database with Helm</h3><ol><li>Clone the Helm Chart Repository</li></ol><pre>git clone git@github.com:hitalukder/elasticsearch.git<br>cd elasticsearch/elasticsearch-helm</pre><p>2. Open the repository in a code editor and modify the values file based on your use case. I have provided sample Helm values files for Elasticsearch deployment:</p><ul><li>A basic version with one of each node type: <a href="https://github.com/hitalukder/elasticsearch/blob/main/elasticsearch-helm/values.yaml">values.yaml</a></li><li>A production-like version, as discussed in the previous section: <a href="https://github.com/hitalukder/elasticsearch/blob/main/elasticsearch-helm/prod-values.yaml">prod-values.yaml</a></li></ul><p>Please follow the readme file and the comments in the values file to understand all the parameters and their usage.</p><p>3. Deploy Elasticsearch</p><p>Create a new project to deploy an Elasticsearch cluster.</p><pre>oc new-project elastic-vector-db</pre><p>Basic deployment</p><pre>helm install vector-db .</pre><p>Production like deployment</p><pre>helm install vector-db -f prod-values.yaml</pre><p>If you want to update values, after editing the values file, use the below command to update the Elasticsearch helm release</p><pre>helm upgrade vector-db .</pre><p>4. Verify deployment</p><p>Execute the below command to verify the installation.</p><pre>oc get Elasticsearch</pre><p>If successful, the health status should be green and phase Ready.</p><h4>Check Kibana status</h4><pre>oc get Kibana</pre><p>The status should be green and phase Ready.</p><h4>Accessing the Elasticsearch Cluster</h4><p>Set the cluster name in a variable</p><pre>export CLUSTER_NAME=&lt;your cluster name provided in values.yaml&gt;</pre><p>Retrieve credentials:</p><pre>echo $(oc get secret ${CLUSTER_NAME}-es-elastic-user -o go-template=&#39;{{.data.elastic | base64decode}}&#39;)</pre><p>Find the URL for Kibana and Elasticsearch:</p><pre>oc get route</pre><pre>export ES_USER=elastic<br>export ES_PASSWORD=$(oc get secret ${CLUSTER_NAME}-es-elastic-user -o go-template=&#39;{{.data.elastic | base64decode}}&#39;)<br>export ES_URL=$(oc get routes route-${CLUSTER_NAME} -o jsonpath=&#39;https://{.spec.host}&#39;)</pre><p>Verify Elasticsearch installation.</p><pre>curl -u &quot;${ES_USER}:${ES_PASSWORD}&quot; -k &quot;${ES_URL}&quot;</pre><p>If everything works well, the output should be like below</p><pre>{<br>  &quot;name&quot; : &quot;es-sample-es-ml-0&quot;,<br>  &quot;cluster_name&quot; : &quot;es-sample&quot;,<br>  &quot;cluster_uuid&quot; : &quot;CbcJtY3URqqWf4N_V0ykHQ&quot;,<br>  &quot;version&quot; : {<br>    &quot;number&quot; : &quot;8.12.0&quot;,<br>    &quot;build_flavor&quot; : &quot;default&quot;,<br>    &quot;build_type&quot; : &quot;docker&quot;,<br>    &quot;build_hash&quot; : &quot;1665f706fd9354802c02146c1e6b5c0fbcddfbc9&quot;,<br>    &quot;build_date&quot; : &quot;2024-01-11T10:05:27.953830042Z&quot;,<br>    &quot;build_snapshot&quot; : false,<br>    &quot;lucene_version&quot; : &quot;9.9.1&quot;,<br>    &quot;minimum_wire_compatibility_version&quot; : &quot;7.17.0&quot;,<br>    &quot;minimum_index_compatibility_version&quot; : &quot;7.0.0&quot;<br>  },<br>  &quot;tagline&quot; : &quot;You Know, for Search&quot;<br>}</pre><p>To get the kibana url, execute the below command</p><pre>oc get routes route-&lt;kibana.name in the .yaml file&gt; -o jsonpath=&#39;https://{.spec.host}&#39;</pre><h4>Installing the ELSER Model (Only if you have ML node)</h4><blockquote>Skip the steps if you don’t have ML nodes</blockquote><ol><li>Log into <strong>Kibana</strong></li><li>Navigate to <strong>Analytics → Machine Learning → Train Models</strong></li><li>Click the <strong>Download</strong> button</li><li>Once downloaded, click <strong>Play</strong> to deploy</li></ol><h4><strong>Clean up</strong></h4><pre>helm delete vector-db</pre><h4>What’s Next?</h4><p>In the next blog, we’ll explore indexing documents using <strong>Sparse and Dense Embedding</strong> for efficient retrieval.</p><p>The source code of this blog can be found <a href="https://github.com/hitalukder/elasticsearch/tree/main/elasticsearch-helm">here in the GitHub repo</a>.</p><h3>Conclusion</h3><p>Using Helm, we’ve simplified Elasticsearch deployment for vector search. Helm provides structured configurations, automates management, and ensures scalability for production environments. Stay tuned for more insights on optimizing AI-powered search systems!</p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more content related to the latest in AI advancement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97df93112977" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/automating-watsonx-discovery-elasticsearch-vector-database-deployment-on-openshift-with-helm-97df93112977">Automating Watsonx Discovery Elasticsearch Vector Database Deployment on OpenShift with Helm</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy an application on the IBM Code Engine]]></title>
            <link>https://medium.com/towards-generative-ai/deploy-an-application-on-the-ibm-code-engine-2edc0d8f71b7?source=rss-b09579f13f8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2edc0d8f71b7</guid>
            <category><![CDATA[watsonx-ai]]></category>
            <category><![CDATA[ibm-cloud]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[code-engine]]></category>
            <category><![CDATA[rag-pipeline]]></category>
            <dc:creator><![CDATA[Himadri Talukder]]></dc:creator>
            <pubDate>Tue, 21 Jan 2025 13:20:55 GMT</pubDate>
            <atom:updated>2025-04-22T21:35:33.456Z</atom:updated>
            <content:encoded><![CDATA[<h4>Deploying a Containerized Application: A Step-by-Step Guide to IBM Cloud Code Engine</h4><p>IBM Cloud Code Engine is a fully managed, serverless platform that simplifies deploying and running containerized applications.1 Here are some key points:</p><ul><li><strong>Fully Managed:</strong> No need to worry about the underlying infrastructure, allowing you to focus on your application.</li><li><strong>Serverless:</strong> Automatically scales your application based on demand, optimizing resource usage.</li><li><strong>Containerized Workloads:</strong> Supports various applications, including web apps, microservices, and batch jobs.</li><li><strong>Fast Deployment:</strong> Quickly deploys your applications with minimal setup and configuration.</li><li><strong>Built on Kubernetes and Knative: </strong>Leverages open-source technologies for flexibility and scalability.</li><li><strong>Integrated with IBM Cloud Services: </strong>Seamlessly integrates with other IBM Cloud services for a comprehensive solution.</li></ul><p>This guide walks you through deploying your application to IBM Cloud Code Engine.</p><p><strong>Prerequisites:</strong></p><ul><li>An IBM Cloud account</li><li>An IBM Cloud CLI installation</li></ul><p><a href="https://github.com/hitalukder/fastapi-scaffold/tree/main">Here</a> is a git repo sample RAG application built with the FastAPI framework using watsonx.ai. You can use it as a reference for this step-by-step guide to deploying an application on IBM Code Engine.</p><h3>Step — 1. Set Up IBM Cloud CLI</h3><h4><strong>Install IBM Cloud CLI</strong></h4><p>If you’ve already installed the CLI, you can skip the following command.</p><pre>   curl -fsSL https://clis.cloud.ibm.com/install/linux | sh</pre><p>ref: <a href="https://github.com/IBM-Cloud/ibm-cloud-cli-release/releases/">https://github.com/IBM-Cloud/ibm-cloud-cli-release/releases/</a></p><p><strong>Note:</strong> Adjust the command for your operating system if necessary.</p><h4>Install the code engine plugins</h4><pre>ibmcloud plugin install code-engine</pre><h4>Install the container registry plugins</h4><pre>ibmcloud plugin install container-registry</pre><h4>Login to IBM Cloud</h4><p>You’ll be prompted for your email and password.</p><pre>ibmcloud login</pre><p>Federated Login:</p><pre>ibmcloud login --sso</pre><h4>Setup resource group</h4><p>If you want to use your resource group, use the command below. Otherwise, it will use the Default resource group.</p><pre>ibmcloud target -g &lt;RESOURCE_GROUP&gt;</pre><h3>Step — 2. Set Up IBM Container Registry (ICR)</h3><h4><strong>Target a region</strong></h4><pre>ibmcloud cr region-set &lt;region&gt;</pre><p>Replace `&lt;region&gt;` with your preferred region, e.g., `us-south`.</p><p>You can find the list of region in here <a href="https://cloud.ibm.com/docs/overview?topic=overview-locations">https://cloud.ibm.com/docs/overview?topic=overview-locations</a></p><h4><strong>Create a namespace</strong></h4><p>Set the namespace in a variable and Replace `&lt;namespace&gt;` with a unique name for your project.</p><pre>export NAMESPACE=&lt;namespace&gt;<br>ibmcloud cr namespace-add $NAMESPACE</pre><h4>Build and Push Your Container Image to ICR</h4><p>These instructions assume you’re using podman. Replace commands with docker if docker is your preferred tool.</p><h4><strong>Log in to Container Registry</strong></h4><pre>ibmcloud cr login</pre><p>After executing the command output will be like below.</p><pre>ibmcloud cr login<br>Logging &#39;podman&#39; in to &#39;us.icr.io&#39;...<br>Logged in to &#39;us.icr.io&#39;.</pre><p>In that case, I logged in to us.icr.io, with ‘<strong>us</strong>’ being my region. In your case, it might be different.</p><p>Region-Specific Endpoints for IBM Container Registry</p><ul><li><strong>US South</strong>: us.icr.io</li><li><strong>US East</strong>: us-east.icr.io</li><li><strong>United Kingdom</strong>: uk.icr.io</li><li><strong>Germany</strong>: de.icr.io</li><li><strong>Tokyo</strong>: jp.icr.io</li><li><strong>Sydney</strong>: au.icr.io</li><li><strong>Toronto</strong>: ca.icr.io</li><li><strong>São Paulo</strong>: br.icr.io</li></ul><p>Replace `&lt;region&gt;`, `&lt;namespace&gt;`, `&lt;image-name&gt;`, and `&lt;tag&gt;` as needed.</p><p>Set Environment Variables (Replace with your details):</p><pre>export REGION=&lt;your-region&gt;<br>export IMAGE_NAME=&lt;your-image-name&gt;<br>export IMAGE_TAG=&lt;your-image-tag&gt;</pre><h4><strong>Build the image locally</strong></h4><pre>podman build -t $REGION.icr.io/$NAMESPACE/$IMAGE_NAME:$IMAGE_TAG .</pre><h4><strong>Push the image to ICR</strong></h4><pre>podman push $REGION.icr.io/$NAMESPACE/$IMAGE_NAME:$IMAGE_TAG</pre><h3>Step — 3. Deploy the containerized application</h3><h4>Create a Code Engine Project</h4><p>Assign the project name to an environment variable so that we can reuse it later. Replace ‘&lt;project-name&gt;’ with your preferred name.</p><pre>export CE_PROJECT_NAME=&lt;project-name&gt;<br>ibmcloud ce project create --name $CE_PROJECT_NAME<br>ibmcloud ce project select --name $CE_PROJECT_NAME</pre><p>If you already have a project, you can use the below command to select it as a default project in this session.</p><pre>ibmcloud ce project select --name $CE_PROJECT_NAME</pre><h4>Create the Registry Access Secret</h4><p>The API key is required to authenticate with the container registry. We will create a secret using the API key to pull the image from the ICR.</p><pre>ibmcloud iam api-key-create my-api-key -d &quot;API key for Code Engine&quot; --file apiKey.json</pre><p>Replace <em>&lt;your-ibm-cloud-api-key&gt;</em> with the API key from the previous step.</p><pre>export API_KEY=&lt;your-ibm-cloud-api-key&gt;<br>ibmcloud ce registry create \<br>    --name my-ce-registry-secret \<br>    --server $REGION.icr.io \<br>    --username iamapikey \<br>    --password $API_KEY</pre><h4><strong>Prepare a </strong><strong>.env File (optional)</strong></h4><p>If your application requires environment variables, you can create a secret and set it as an environment variable with key-value pairs.</p><p>Create a .env file (e.g., my-env.env) to store environment variables for your application. Make sure values are not wrapped in quotes and remove comments to avoid including them in the secret.</p><p>example env file</p><pre>   ENV_VAR_1=value1<br>   ENV_VAR_2=value2<br>  </pre><p>Execute the below command to create a secret to set environment variable</p><pre>ibmcloud ce secret create --name my-env-secret --from-env-file my-env.env</pre><h4>Deploy the application</h4><p>Before deploying the application lets set the service name in a variable. Please choose a service name for your application</p><pre>export SERVICE_NAME=&lt;service-name&gt;</pre><p>Execute the below command to create the application service.</p><p>Replace &lt;port&gt; , cpu , memory , min-scale , max-scale as needed</p><pre>ibmcloud ce application create \<br>        --name $SERVICE_NAME \<br>        --image private.$REGION.icr.io/$NAMESPACE/$IMAGE_NAME:$IMAGE_TAG \<br>        --cpu 1 --memory 2G --port &lt;port&gt;\<br>        --env-from-secret my-env-secret \<br>        --min-scale 0\<br>        --max-scale 3 \<br>        --registry-secret my-ce-registry-secret</pre><h4><strong>Verify the deployment</strong></h4><pre>ibmcloud ce application get --name $SERVICE_NAME</pre><h3>Step — 4. <strong>Access the Deployed Service</strong></h3><h4><strong>Get the service URL</strong></h4><pre>ibmcloud ce application get --name $SERVICE_NAME --output url</pre><p>Use `curl` or a browser to access the URL</p><h3>Step — 5. <strong>Update the Service if Needed</strong></h3><p>Update the image or other settings</p><pre>ibmcloud ce application update \<br>       --name &lt;service-name&gt; \<br>       --image private.&lt;region&gt;.icr.io/&lt;namespace&gt;/&lt;image-name&gt;:&lt;new-tag&gt;</pre><p>Restart the service after updates</p><pre>ibmcloud ce application restart --name &lt;service-name&gt;</pre><p>Set the ` — max-scale` and ` — min-scale` to zero to effectively scale down your application</p><pre>ibmcloud ce application update \<br>    --name &lt;service-name&gt; \<br>    --min-scale 0 \<br>    --max-scale 0</pre><p>` — min-scale 0`: Allows the application to scale down to zero replicas.</p><p>` — max-scale 0`: Ensures no replicas are running.</p><h3>Step — 6. Monitoring &amp; debugging</h3><p>You can check the status of all applications to confirm that it has been scaled down or deleted:</p><pre>ibmcloud ce application list</pre><p>This will display a list of all applications and their statuses.</p><p>To access the logs</p><pre>ibmcloud ce application logs --name $SERVICE_NAME</pre><h3>Step — 7. Clean up</h3><p>Command to Delete an Application</p><p>You will be prompted to confirm. To skip the confirmation, add the `-f` flag</p><pre>ibmcloud ce application delete --name &lt;service-name&gt; -f</pre><p>If you delete the application, consider cleaning up associated resources like secrets, registry credentials, and environment variables if they are no longer needed. Use the following commands</p><ul><li>Delete a secret</li></ul><pre>ibmcloud ce secret delete --name &lt;secret-name&gt; -f</pre><ul><li>Delete a registry secret</li></ul><pre>ibmcloud ce registry delete --name &lt;registry-secret-name&gt; -f</pre><h3>Conclusion</h3><p>Deploying your containerized application to IBM Code Engine is a straightforward process that simplifies scaling and management. By leveraging Code Engine’s serverless capabilities, you can focus on building your application without the complexities of infrastructure.</p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more content related to the latest in AI advancement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2edc0d8f71b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/deploy-an-application-on-the-ibm-code-engine-2edc0d8f71b7">Deploy an application on the IBM Code Engine</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Document Indexing with SAP HANA Vector Database]]></title>
            <link>https://medium.com/towards-generative-ai/document-indexing-with-sap-hana-vector-database-71e6d3d7f1fa?source=rss-b09579f13f8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/71e6d3d7f1fa</guid>
            <category><![CDATA[rag-pipeline]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[sap-hana]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Himadri Talukder]]></dc:creator>
            <pubDate>Tue, 21 Jan 2025 13:19:58 GMT</pubDate>
            <atom:updated>2025-01-22T14:39:04.580Z</atom:updated>
            <content:encoded><![CDATA[<h4>A simple Guide to Indexing Documents in SAP HANA Vector Database for Semantic Search Capabilities</h4><h3><strong>Introduction</strong></h3><p>The integration of vector indexing and retrieval-augmented generation (RAG) has revolutionized the way we approach semantic search and AI-generated content. By combining tools like SAP HANA Vector Database with frameworks such as LangChain, IBM watsonx.ai, Generative AI Hub SDK, and state-of-the-art embedding models, developers can build scalable and highly efficient systems for indexing and searching documents.</p><p>This blog is a hands-on guide to help you to get SAP HANA’s vector database for semantic search by processing and indexing documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GoV_V4xBPxCC3IbwvxXV1w.png" /><figcaption>Architecture diagram RAG Pipeline</figcaption></figure><h3><strong>Why Use Vector Indexing in SAP HANA?</strong></h3><p>SAP HANA’s vector database provides a robust solution for handling high-dimensional data, making it a great choice for use cases like:</p><ul><li><strong>Semantic Search:</strong> Retrieve content based on meaning rather than keywords.</li><li><strong>Recommendation Systems:</strong> Match user preferences with relevant results.</li><li><strong>Clustering and Anomaly Detection:</strong> Identify patterns in data or detect outliers.</li><li><strong>Retrieval-Augmented Generation (RAG):</strong> Enhance the relevance of AI-generated responses by retrieving pertinent context.</li></ul><p>With the <strong>REAL_VECTOR</strong> data type and seamless SQL integration, SAP HANA simplifies managing enterprise data and vector embeddings in a unified architecture. It integrates seamlessly with SQL for CRUD and Query operations on vectors to perform semantic search.</p><h3>Key Components</h3><ul><li><strong>Documents for Indexing:</strong> A collection of files (PDF, DOCX, HTML).</li><li><strong>Embedding Model:</strong> E.g., sentence-transformers/all-MiniLM-L6-v2, for vectorizing content.</li><li><strong>SAP HANA Cloud Instance:</strong> To store and retrieve indexed vectors.</li></ul><h3>Step 1: Document Parsing</h3><p>Parsing documents is the first step toward extracting meaningful content. Below is a Python function for parsing PDFs using Apache Tika.</p><p>Loading pdf from s3 bucket.</p><pre>def parse_pdfs_from_s3_buffer(bucket_name, prefix=&quot;&quot;):<br>    &quot;&quot;&quot;<br>    Parses PDF files from S3 bucket using Tika.<br>    <br>    Args:<br>        bucket_name (str): S3 bucket name.<br>        prefix (str): S3 prefix filter.<br><br>    Returns:<br>        List of tuples with file key, content, and metadata.<br>    &quot;&quot;&quot;<br>    try:<br>        s3 = boto3.client(&#39;s3&#39;)<br>        paginator = s3.get_paginator(&#39;list_objects_v2&#39;)<br>        page_iterator = paginator.paginate(Bucket=bucket_name, Prefix=prefix)<br>        all_pdf_data = []<br><br>        for page in page_iterator:<br>            if &#39;Contents&#39; in page:<br>                for obj in page[&#39;Contents&#39;]:<br>                    if obj[&#39;Key&#39;].endswith(&quot;.pdf&quot;):<br>                        try:<br>                            response = s3.get_object(Bucket=bucket_name, Key=obj[&#39;Key&#39;])<br>                            file_content = response[&#39;Body&#39;].read()<br>                            xml_data = parser.from_buffer(file_content, xmlContent=True)<br>                            all_pdf_data.append((obj[&#39;Key&#39;], xml_data[&#39;content&#39;], xml_data[&#39;metadata&#39;]))<br>                        except Exception as e:<br>                            print(f&quot;Error processing {obj[&#39;Key&#39;]}: {e}&quot;)<br><br>        return all_pdf_data<br>    except Exception as e:<br>        print(f&quot;An error occurred: {e}&quot;)<br>        return []</pre><pre>def extract_pdf_metadata_page(file_content, metadata, file_name):<br>    &quot;&quot;&quot;<br>    Extracts metadata and content of a PDF file.<br>    &quot;&quot;&quot;<br>    soup = BeautifulSoup(file_content, &quot;html.parser&quot;)<br>    pages = soup.find_all(&quot;div&quot;, class_=&quot;page&quot;)<br>    total_pages = len(pages)<br><br>    metadata_fields = {<br>        &quot;total_pages&quot;: metadata.get(&#39;meta:page-count&#39;, total_pages),<br>        &quot;title&quot;: metadata.get(&#39;pdf:docinfo:title&#39;, &quot;&quot;),<br>        &quot;keywords&quot;: metadata.get(&#39;pdf:docinfo:keywords&#39;, &quot;&quot;),<br>        &quot;create_date&quot;: metadata.get(&#39;xmp:CreateDate&#39;, &quot;&quot;),<br>        &quot;modify_date&quot;: metadata.get(&#39;xmp:ModifyDate&#39;, &quot;&quot;)<br>    }<br><br>    doc_list = []<br>    for page_num, page in enumerate(pages, start=1):<br>        page_content = pre_process_text(str(page))<br>        doc = {<br>            &quot;file_name&quot;: file_name,<br>            &quot;total_pages&quot;: metadata_fields[&quot;total_pages&quot;],<br>            &quot;keywords&quot;: metadata_fields[&quot;keywords&quot;],<br>            &quot;create_date&quot;: metadata_fields[&quot;create_date&quot;],<br>            &quot;modify_date&quot;: metadata_fields[&quot;modify_date&quot;],<br>            &quot;page_content&quot;: page_content,<br>            &quot;page_number&quot;: page_num,<br>            &quot;content_length&quot;: len(page_content)<br>        }<br>        doc_list.append(doc)<br><br>    return doc_list</pre><p>Here’s a breakdown of the code:</p><h4><strong>Extracting Metadata:</strong></h4><ul><li>The file_content is parsed using BeautifulSoup to enable easy extraction of HTML elements.</li><li>metadata_fields stores the metadata of the document.</li><li>doc_list contains contents and the corresponding metadata of the document.</li></ul><h3>Step 2: Preprocessing Documents</h3><p>Preprocessing ensures that the extracted data is ready for embedding.</p><pre>def preprocess_documents(documents):<br>    &quot;&quot;&quot;<br>    Prepares documents by separating content and metadata.<br>    &quot;&quot;&quot;<br>    content = []<br>    metadata = []<br><br>    for doc in documents:<br>        metadata.append({<br>            &quot;title&quot;: doc.get(&quot;file_name&quot;, &quot;&quot;),<br>            &quot;document_url&quot;: doc.get(&quot;file_name&quot;, &quot;&quot;),<br>            &quot;page_number&quot;: doc.get(&quot;page_number&quot;, &quot;&quot;)<br>        })<br>        content.append(f&quot;Document Content: {doc.get(&#39;page_content&#39;, &#39;&#39;)}&quot;)<br><br>    return content, metadata</pre><h3>Step 3: Chunking for Efficient Processing</h3><p><strong>Chunking Text for Efficient Processing</strong>:</p><ul><li>When dealing with large texts (e.g., documents, books, or articles), many NLP models have token limits. Breaking text into chunks ensures that each piece is within these limits.</li><li>Overlapping chunks improve the context for models that work with segmented text.</li></ul><pre># Parameters<br>parameters = {<br>    &quot;ingestion_chunk_size&quot;: 256,<br>    &quot;ingestion_chunk_overlap&quot;: 128<br>}<br><br># Text splitter initialization<br>text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(<br>    chunk_size=parameters[&quot;ingestion_chunk_size&quot;],<br>    chunk_overlap=parameters[&quot;ingestion_chunk_overlap&quot;],<br>    disallowed_special=()<br>)</pre><h3>Step 4: Vector Embedding</h3><p>Using Sentence Transformers, you can map text to high-dimensional vectors for tasks such as clustering and semantic search. Below is an example of a Sentence Transformer embedding model sentence-transformers/all-MiniLM-L6-V2. Alternatively, you can use your preferred embedding model, such as text-embedding-ada-002.</p><pre>from sentence_transformers import SentenceTransformer<br><br>model = SentenceTransformer(&#39;sentence-transformers/all-MiniLM-L6-v2&#39;)<br><br># Generate embeddings for each chunk<br>document_to_index = []<br>for idx, doc in enumerate(split_docs):<br>    try:<br>        embed = model.encode(doc[&#39;page_content&#39;]).tolist() ## converting text to vector embedding<br>        document_to_index.append([<br>            idx + 1,<br>            doc[&#39;title&#39;],<br>            doc[&#39;page_number&#39;],<br>            doc[&#39;document_url&#39;],<br>            doc[&#39;page_content&#39;],<br>            embed<br>        ])<br>    except Exception as e:<br>        print(f&quot;Error processing document {idx + 1}: {e}&quot;)</pre><h3>Step 5: Storing in SAP HANA</h3><p>Establishing a Secure Connection</p><pre># Establish a secure connection to an SAP HANA database using hdbcli <br>import hdbcli<br>from hdbcli import dbapi<br><br>cc = dbapi.connect(<br>    address=HANA_DB_HOST,<br>    port=&#39;443&#39;,<br>    user=HANA_DB_USER,<br>    password=HANA_DB_PASSWORD,<br>    encrypt=True<br>    )</pre><h4>Create a database table</h4><p><strong>Table name: </strong><em>TBL_VECTOR_STORE</em></p><ul><li><em>ID </em><strong><em>BIGINT</em></strong><em>,</em></li><li><em>TITLE </em><strong><em>NVARCHAR</em></strong><em>(100),</em></li><li><em>PAGE_NUMBER </em><strong><em>NVARCHAR</em></strong><em>(3),</em></li><li><em>URL </em><strong><em>NVARCHAR</em></strong><em>(100),</em></li><li><em>TEXT </em><strong><em>NCLOB</em></strong><em>,</em></li><li><em>VECTOR_STR </em><strong><em>REAL_VECTOR</em></strong></li></ul><p>The SAP HANA Cloud database has the built-in vector data type REAL_VECTOR. To learn about the REAL_VECTOR datatype please follow the <a href="https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-vector-engine-guide/real-vector-data-type">link here</a></p><pre># Create a table<br>cursor = cc.cursor()<br>sql_command = &#39;&#39;&#39;<br>CREATE TABLE TBL_VECTOR_STORE (<br>    ID BIGINT,<br>    TITLE NVARCHAR(100),<br>    PAGE_NUMBER NVARCHAR(5),<br>    URL NVARCHAR(100),<br>    TEXT NCLOB,<br>    VECTOR_STR REAL_VECTOR<br>)<br>&#39;&#39;&#39;<br>cursor.execute(sql_command)<br>cursor.close()</pre><h4>Inserting data</h4><pre># Inserting data into the specified table using a prepared SQL statement with real vector conversion.<br>cursor = conn.cursor()<br>sql_insert = &#39;&#39;&#39;<br>    INSERT INTO TBL_VECTOR_STORE <br>    (ID, TITLE, PAGE_NUMBER, URL, TEXT, VECTOR_STR) <br>    VALUES (?, ?, ?, ?, ?, TO_REAL_VECTOR(?))<br>&#39;&#39;&#39;<br>cursor.executemany(sql_insert, document_to_index)<br>conn.commit()<br>cursor.close()</pre><h3>Step 6: Performing Vector Search</h3><p>Retrieve similar documents using vector similarity metrics like COSINE_SIMILARITY.</p><pre>def run_vector_search(query, cursor, table_name, metric=&quot;COSINE_SIMILARITY&quot;, k=4):<br>    &quot;&quot;&quot;<br>    Performs vector search on indexed documents.<br>    &quot;&quot;&quot;<br>    try:<br>        query_vector = get_embedding(query)<br>        if not query_vector:<br>            raise ValueError(&quot;Failed to generate query embedding.&quot;)<br><br>        sort_order = &quot;DESC&quot; if metric != &quot;L2DISTANCE&quot; else &quot;ASC&quot;<br>        sql_query = f&quot;&quot;&quot;<br>        SELECT TOP {k} ID, PAGE_NUMBER, TEXT<br>        FROM {table_name}<br>        ORDER BY {metric}(VECTOR_STR, TO_REAL_VECTOR(&#39;{query_vector}&#39;)) {sort_order}<br>        &quot;&quot;&quot;<br>        cursor.execute(sql_query)<br>        return cursor.fetchall()<br>    except Exception as e:<br>        print(f&quot;Error during vector search: {e}&quot;)<br>        return []</pre><pre>context = run_vector_search(&quot;Overview of the Granite Pre-Training Dataset?&quot;, &#39;COSINE_SIMILARITY&#39;, 4)</pre><p>Returns the top k results, including fields like &quot;ID&quot;, &quot;PAGE_NUMBER&quot;, and &quot;TEXT&quot; This function is doing semantic search, allowing retrieval of the most relevant content from a vectorized database based on a text query and similarity metric.</p><p>A complete project showing document indexing from a COS bucket, including pre-processing, post-processing, ingestion into the SAP HANA database and containerized project, is available in the <a href="https://github.com/ibm-ecosystem-engineering/SAP-watsonx-integration/tree/main/5.%20SAP%20Generative%20AI%20Hub%20and%20HANA%20Cloud%20Vector%20Engine/5.1%20Q%26A%20RAG/code/docs-indexing">GitHub repository here.</a></p><h3><strong>Conclusion</strong></h3><p>SAP HANA Vector Database is a game-changer for building advanced AI applications. With its seamless integration of vector and relational data, robust SQL support, and native handling of vector embeddings, you can efficiently implement solutions like semantic search and RAG.</p><p>Discover the complete implementation in our <a href="https://github.com/ibm-ecosystem-engineering/SAP-watsonx-integration/tree/main/5.%20SAP%20Generative%20AI%20Hub%20and%20HANA%20Cloud%20Vector%20Engine/5.1%20Q%26A%20RAG/code/docs-indexing">GitHub repository</a>.</p><p>Start exploring the power of vector indexing with SAP HANA today!</p><p>To learn how to deploy an IBM Granite model, <a href="https://medium.com/towards-generative-ai/deploying-ibm-granite-model-on-sap-generative-ai-hub-befd4670a8d2">click here</a>. For details on implementing a RAG pipeline using the SAP Generative AI Hub with the IBM Granite model, <a href="https://medium.com/towards-generative-ai/deploying-a-rag-pipeline-to-sap-ai-core-with-the-ibm-granite-model-1e3178892b93">click here.</a></p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more content related to the latest in AI advancement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71e6d3d7f1fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/document-indexing-with-sap-hana-vector-database-71e6d3d7f1fa">Document Indexing with SAP HANA Vector Database</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building  a RAG Pipeline to SAP AI Core with the IBM Granite Model]]></title>
            <link>https://medium.com/towards-generative-ai/deploying-a-rag-pipeline-to-sap-ai-core-with-the-ibm-granite-model-1e3178892b93?source=rss-b09579f13f8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e3178892b93</guid>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[rag-pipeline]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[granite]]></category>
            <category><![CDATA[sap-ai-core]]></category>
            <dc:creator><![CDATA[Himadri Talukder]]></dc:creator>
            <pubDate>Tue, 21 Jan 2025 13:19:12 GMT</pubDate>
            <atom:updated>2025-02-03T17:12:35.871Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a RAG Pipeline to SAP AI Core with the IBM Granite Model</h3><h4>Serve Scalable and trustworthy AI Solutions with SAP AI Core, SAP HANA as VectorDB, and the IBM Granite Model using a serving template.</h4><p>SAP AI Core, a critical <a href="https://www.sap.com/products/technology-platform.html">SAP Business Technology Platform (SAP BTP)</a> component, provides a robust, cloud-agnostic framework for managing AI assets. It streamlines the lifecycle of AI models and pipelines, empowering developers to scale their AI workflows easily. With seamless integration into SAP solutions and support for the <a href="https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/generative-ai-hub-in-sap-ai-core">Generative AI Hub</a>, SAP AI Core bridges the gap between cutting-edge AI and enterprise needs.</p><p>This guide walks you through the steps to deploy a Retrieval-Augmented Generation (RAG) pipeline to SAP AI Core using a serving template. Serving templates allow you to manage the deployment of a model/application at the main tenant level. They define how models/applications are deployed and are used to deploy one or more models on a server that handles inference requests. These templates are stored in your git repository for versioning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GoV_V4xBPxCC3IbwvxXV1w.png" /><figcaption>RAG Architecture</figcaption></figure><p>Follow these clear steps to get your AI solution up and running.</p><h3>What You’ll Need</h3><p>Before starting, ensure you have the following:</p><ul><li><strong>SAP BTP Enterprise Account</strong></li><li><strong>SAP AI Core Extended Service Plan</strong></li><li><strong>SAP AI Launchpad Standard Service Plan</strong></li><li><strong>Personal Access Token (PAT)</strong> for Docker registry</li><li><strong>GitHub Repository and PAT</strong> for repository access</li></ul><h3>Steps to Deploy a RAG Pipeline</h3><ol><li>Prepare Input Parameters and Credentials</li><li>Build the RAG Pipeline Application Container</li><li>Configure the Serving Template</li><li>Connect to SAP AI Core</li><li>Onboard the GitHub Repository</li><li>Register a Docker Registry Secret</li><li>Create an Application</li><li>Define a Configuration</li><li>Deploy the Pipeline</li><li>Test the Deployment</li></ol><h3>1. Prepare Input Parameters and Credentials</h3><p>Gather configuration details for the RAG pipeline, including your <strong>SAP AI Core service key</strong>, <strong>GitHub details</strong>, and <strong>Docker credentials</strong>.</p><p>Download the aicore_service_key.json file from your SAP AI Core service instance. Follow the steps here to retrieve the file.</p><ul><li><strong>Log in to BTP Cockpit: </strong>Access your BTP Cockpit by logging in with your SAP credentials.</li><li><strong>Navigate to Account Explorer:</strong> In the left-hand menu of BTP Cockpit, click on <strong>Account Explorer</strong>.</li><li><strong>Select Your Subaccount: </strong>Under the <strong>Subaccount</strong> tab, choose the subaccount where you want to deploy the model.</li><li><strong>Access the Service Instance: </strong>Click on the link to the service instance associated with your subaccount.</li><li><strong>Download the Credentials: </strong>Click on the <strong>Credentials</strong> key to download the necessary credentials. Your credential file should look like this:</li></ul><pre>{<br>    &quot;clientid&quot;: &quot;sb-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx!xxxxxxx|aicore!xxxx&quot;,<br>    &quot;clientsecret&quot;: &quot;xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx$***&quot;,<br>    &quot;url&quot;: &quot;https://***.authentication.***.hana.ondemand.com&quot;,<br>    &quot;identityzone&quot;: &quot;***&quot;,<br>    &quot;identityzoneid&quot;: &quot;xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&quot;,<br>    &quot;appname&quot;: &quot;xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx!xxxxxxx|aicore!xxxx&quot;,<br>    &quot;serviceurls&quot;: {<br>        &quot;AI_API_URL&quot;: &quot;https://api.ai.***.hana.ondemand.com&quot;<br>    }<br>}</pre><ul><li><strong>GitHub Info</strong></li></ul><pre>{<br>    &quot;name&quot;: &quot;my-github-secret&quot;,<br>    &quot;url&quot;: &quot;https://github.com/hitalukder/myrepo&quot;,<br>    &quot;username&quot;: &quot;hitalukder&quot;,<br>    &quot;password&quot;: &quot;ghp_***&quot;<br>}</pre><ul><li><strong>Docker Registry Secret</strong></li></ul><pre>{<br>    &quot;name&quot;: &quot;my-dockerhub-secret&quot;,<br>    &quot;data&quot;: {<br>        &quot;.dockerconfigjson&quot;: &quot;{\&quot;auths\&quot;:{\&quot;https://index.docker.io\&quot;:{\&quot;kevinxhuang\&quot;:\&quot;\&quot;,\&quot;password\&quot;:\&quot;dckr_pat_***\&quot;}}}&quot;<br>    }<br>}</pre><h3>2. Build the RAG Pipeline Application Container</h3><p>The RAG application is a Python fastapi project integrating advanced components to deliver accurate responses using a Large Language Model (LLM). The complete source code and step-by-step instructions are described <a href="https://github.com/ibm-ecosystem-engineering/SAP-watsonx-integration/blob/8137780442e19cd0ebecbdaf68c5b0463f63fe32/5.%20SAP%20Generative%20AI%20Hub%20and%20HANA%20Cloud%20Vector%20Engine/5.1%20Q&amp;A%20RAG/code/ai-core-rag-pipeline"><strong><em>here</em></strong></a>:</p><ul><li>Build a docker container</li><li>Push it to a docker registry</li></ul><p>Take note of the docker image URL. You’ll need it for preparing the serving template in the next step.</p><h3>3. Configure the Serving Template</h3><p>In SAP AI Core, serving templates are executables with required metadata attributes. Model deployment is managed using Kubernetes Custom Resource Definitions (CRD) provided by KServe, which are defined in a YAML file specifying input parameters and artifacts.</p><p>For this example, we have created a Git repository containing a ready-to-use serving template. You can find the reference YAML and repository <a href="https://github.com/ibm-ecosystem-engineering/SAP-watsonx-integration/blob/8137780442e19cd0ebecbdaf68c5b0463f63fe32/5.%20SAP%20Generative%20AI%20Hub%20and%20HANA%20Cloud%20Vector%20Engine/5.1%20Q&amp;A%20RAG/code/ai-core-serving-template"><strong>here</strong></a>.</p><p><strong>NOTE:</strong></p><ul><li>You should create a separate Git repository for versioning the serving template.</li><li>You can specify a resource plan for the deployment with ai.sap.com/resourcePlan. Otherwise, default starter resource plan will apply, which should be enough to run the RAG pipeline. Find out more about the available resource plans in <a href="https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/choose-resource-plan-train">SAP Help Portal</a>.</li><li>You can set the auto scaling of the server with the parameters: minReplicas and maxReplicas.</li><li>You can use a Docker registry secret in AI Core via imagePullSecrets to access a container image in a private container registry. If you are using an image from docker.io, the Docker registry secret must points to the URL https://index.docker.io. The name of the secret should match what&#39;s set in the docker_registry_secret.json file.</li><li>You must set the name of the container to kfserving-container or kserve-container.</li><li>metadata.name = executable_id</li><li>metadata.name.annotations.scenarios.ai.sap.com/id = scenario_id</li></ul><pre>apiVersion: ai.sap.com/v1alpha1<br>kind: ServingTemplate<br>metadata:<br>  name: rag-pipeline-20241118<br>  annotations:<br>    scenarios.ai.sap.com/description: &quot;RAG Pipeline&quot;<br>    scenarios.ai.sap.com/name: &quot;rag-pipeline&quot;<br>    executables.ai.sap.com/description: &quot;RAG Pipeline: retriever, building prompt, generation&quot;<br>    executables.ai.sap.com/name: &quot;rag-pipeline-executable&quot;<br>  labels:<br>    scenarios.ai.sap.com/id: &quot;rag-pipeline&quot;<br>    ai.sap.com/version: &quot;1.0.0&quot;<br>spec:<br>  inputs:<br>    parameters:<br>      - name: myResourcePlan<br>        type: string<br>        default: Starter<br>      - name: AICORE_AUTH_URL<br>        type: string<br>      - name: AICORE_CLIENT_ID<br>        type: string<br>      - name: AICORE_CLIENT_SECRET<br>        type: string<br>      - name: AICORE_BASE_URL<br>        type: string<br>      - name: AICORE_RESOURCE_GROUP<br>        type: string<br>      - name: ORC_API_URL<br>        type: string<br>      - name: HANA_DB_HOST<br>        type: string<br>      - name: HANA_DB_USER<br>        type: string<br>      - name: HANA_DB_PASSWORD<br>        type: string<br>      - name: HANA_DB_TABLE_NAME<br>        type: string<br>  template:<br>    apiVersion: &quot;serving.kserve.io/v1beta1&quot;<br>    metadata:<br>      annotations: |<br>        autoscaling.knative.dev/metric: concurrency<br>        autoscaling.knative.dev/target: 1<br>        autoscaling.knative.dev/targetBurstCapacity: 0<br>      labels: |<br>        ai.sap.com/resourcePlan: &quot;{{inputs.parameters.myResourcePlan}}&quot;<br>    spec: |<br>      predictor:<br>        imagePullSecrets:<br>          - name: my-dockerhub-secret<br>        minReplicas: 1<br>        maxReplicas: 5<br>        containers:<br>        - name: kserve-container<br>          image: docker.io/kevinxhuang/rag-pipeline:latest<br>          command: [&quot;python3&quot;, &quot;main.py&quot;]<br>          ports:<br>            - containerPort: 3001<br>              protocol: TCP<br>          env:<br>            - name: AICORE_AUTH_URL<br>              value: &quot;{{inputs.parameters.AICORE_AUTH_URL}}&quot;<br>            - name: AICORE_CLIENT_ID<br>              value: &quot;{{inputs.parameters.AICORE_CLIENT_ID}}&quot;<br>            - name: AICORE_CLIENT_SECRET<br>              value: &quot;{{inputs.parameters.AICORE_CLIENT_SECRET}}&quot;<br>            - name: AICORE_BASE_URL<br>              value: &quot;{{inputs.parameters.AICORE_BASE_URL}}&quot;<br>            - name: AICORE_RESOURCE_GROUP<br>              value: &quot;{{inputs.parameters.AICORE_RESOURCE_GROUP}}&quot;<br>            - name: ORC_API_URL<br>              value: &quot;{{inputs.parameters.ORC_API_URL}}&quot;<br>            - name: HANA_DB_HOST<br>              value: &quot;{{inputs.parameters.HANA_DB_HOST}}&quot;<br>            - name: HANA_DB_USER<br>              value: &quot;{{inputs.parameters.HANA_DB_USER}}&quot;<br>            - name: HANA_DB_PASSWORD<br>              value: &quot;{{inputs.parameters.HANA_DB_PASSWORD}}&quot;<br>            - name: HANA_DB_TABLE_NAME<br>              value: &quot;{{inputs.parameters.HANA_DB_TABLE_NAME}}&quot;</pre><h3>4. Connect to SAP AI Core</h3><p>Authenticate and connect using the AI Core SDK:</p><pre>from ai_core_sdk.ai_core_v2_client import AICoreV2Client<br><br># Create an AI API client instance<br>ai_core_client = AICoreV2Client(<br>    base_url = aicore_service_key[&quot;serviceurls&quot;][&quot;AI_API_URL&quot;] + &quot;/v2&quot;, # The present AI API version is 2<br>    auth_url=  aicore_service_key[&quot;url&quot;] + &quot;/oauth/token&quot;,<br>    client_id = aicore_service_key[&#39;clientid&#39;],<br>    client_secret = aicore_service_key[&#39;clientsecret&#39;]<br>)</pre><h3>5. Onboard the GitHub Repository</h3><p>Add your serving template to version control by onboarding the GitHub repository:</p><pre>ai_core_client.repositories.create(<br>    name = github_info[&quot;name&quot;],<br>    url = github_info[&quot;url&quot;],<br>    username = github_info[&quot;username&quot;],<br>    password = github_info[&quot;password&quot;]<br>)</pre><h3>6. Register a Docker Registry Secret</h3><p>The credentials for your Docker registries are managed using secrets. The following code cell sends the credentials through a POST request. The encoded data is then stored as a secret on SAP AI Core.</p><pre>response = ai_core_client.docker_registry_secrets.create(<br>    name = docker_registry_secret[&quot;name&quot;],<br>    data = docker_registry_secret[&quot;data&quot;]<br>)<br><br>print(response.__dict__)</pre><h3>7. Create AI Core Application</h3><p>Define the AI Core application linked to the GitHub repository:</p><pre>response = ai_core_client.applications.create(<br>    application_name = rag_pipeline_params[&quot;application_name&quot;],<br>    repository_url = github_info[&quot;url&quot;],<br>    path = rag_pipeline_params[&quot;application_path&quot;],<br>    revision = &quot;HEAD&quot;<br>)</pre><h3>8. Define a Configuration</h3><p>Here are the important pieces of your configuration:</p><ul><li>The scenario_id should contain the same value as in your executable.</li><li>The executable_id is the name key of your executable.</li></ul><pre># Replace the artifact_id field value with your own ID, then execute the code.<br>from ai_core_sdk.models import InputArtifactBinding, ParameterBinding<br><br>response = ai_core_client.configuration.create(<br>    name = rag_pipeline_params[&quot;configuration_name&quot;],<br>    resource_group = &quot;default&quot;,<br>    scenario_id = rag_pipeline_params[&quot;scenario_id&quot;],<br>    executable_id = rag_pipeline_params[&quot;executable_id&quot;],<br>    parameter_bindings = [<br>        ParameterBinding(key=&quot;AICORE_AUTH_URL&quot;, value=rag_pipeline_params[&quot;aicore_auth_url&quot;]),<br>        ParameterBinding(key=&quot;AICORE_CLIENT_ID&quot;, value=rag_pipeline_params[&quot;aicore_client_id&quot;]),<br>        ParameterBinding(key=&quot;AICORE_CLIENT_SECRET&quot;, value=rag_pipeline_params[&quot;aicore_client_secret&quot;]),<br>        ParameterBinding(key=&quot;AICORE_BASE_URL&quot;, value=rag_pipeline_params[&quot;aicore_ai_api_url&quot;] + &quot;/v2&quot;),<br>        ParameterBinding(key=&quot;AICORE_RESOURCE_GROUP&quot;, value=rag_pipeline_params[&quot;aicore_resource_group&quot;]),<br>        ParameterBinding(key=&quot;ORC_API_URL&quot;, value=rag_pipeline_params[&quot;orc_api_endpoint&quot;]),<br>        ParameterBinding(key=&quot;HANA_DB_HOST&quot;, value=rag_pipeline_params[&quot;hana_db_host&quot;]),<br>        ParameterBinding(key=&quot;HANA_DB_USER&quot;, value=rag_pipeline_params[&quot;hana_db_user&quot;]),<br>        ParameterBinding(key=&quot;HANA_DB_PASSWORD&quot;, value=rag_pipeline_params[&quot;hana_db_password&quot;]),<br>        ParameterBinding(key=&quot;HANA_DB_TABLE_NAME&quot;, value=rag_pipeline_params[&quot;hana_db_table_name&quot;])<br>    ]<br>)<br><br>print(response.__dict__)<br>configuration_id=response.__dict__[&#39;id&#39;]</pre><h3>9. Deploy the Pipeline</h3><p>Execute the code below to create a Deployment from the Configuration:</p><pre>response = ai_core_client.deployment.create(<br>    resource_group = &quot;default&quot;,<br>    configuration_id = configuration_id<br>)<br><br>print(response.__dict__)<br>deployment_id=response.__dict__[&#39;id&#39;]</pre><h3>10. Test the RAG Pipeline</h3><pre>import json<br>import requests<br><br>data = {<br>    &quot;query&quot;: &quot;what is ibm cloud?&quot;<br>}<br><br>endpoint = f&quot;{deployment_url}/v2/generate&quot;<br>headers = {&quot;Content-Type&quot;: &quot;application/json&quot;,<br>           &quot;Authorization&quot;: ai_core_client.rest_client.get_token(),<br>           &quot;AI-Resource-Group&quot;: &quot;default&quot;}<br>response = requests.post(endpoint, headers=headers, data=json.dumps(data))<br><br>print(response.text)</pre><p>A complete notebook with examples is available in the <a href="https://github.com/ibm-ecosystem-engineering/SAP-watsonx-integration/blob/main/5.%20SAP%20Generative%20AI%20Hub%20and%20HANA%20Cloud%20Vector%20Engine/5.1%20Q%26A%20RAG/notebooks/deploy-rag-pipeline-on-aicore.ipynb">GitHub repository.</a></p><p>Congratulations! You have successfully deployed a RAG pipeline on SAP AI Core, which leverages the IBM Granite model deployed in Generative AI Hub and the grounding data stored in SAP HANA Cloud database. This guide demonstrated the Generative AI capabilities of SAP BTP in managing and scaling complex AI workflows that seamlessly integrate with your SAP systems.</p><p>To learn how to deploy an IBM Granite model, <a href="https://medium.com/towards-generative-ai/deploying-ibm-granite-model-on-sap-generative-ai-hub-befd4670a8d2">click here</a>. For document indexing on an SAP HANA instance, <a href="https://medium.com/towards-generative-ai/document-indexing-with-sap-hana-vector-database-71e6d3d7f1fa">click here</a>.</p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more content related to the latest in AI advancement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e3178892b93" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/deploying-a-rag-pipeline-to-sap-ai-core-with-the-ibm-granite-model-1e3178892b93">Building  a RAG Pipeline to SAP AI Core with the IBM Granite Model</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>