<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yasar Kocyigit on Medium]]></title>
        <description><![CDATA[Stories by Yasar Kocyigit on Medium]]></description>
        <link>https://medium.com/@kocyigityasar?source=rss-1ab647a5c7d7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5Jw6JsQuMHBOgcWovqJDhw.jpeg</url>
            <title>Stories by Yasar Kocyigit on Medium</title>
            <link>https://medium.com/@kocyigityasar?source=rss-1ab647a5c7d7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:04:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kocyigityasar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Rebuilding SSRS Subscriptions on Databricks]]></title>
            <link>https://medium.com/@kocyigityasar/rebuilding-ssrs-subscriptions-on-databricks-cd99ce7c4dad?source=rss-1ab647a5c7d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd99ce7c4dad</guid>
            <dc:creator><![CDATA[Yasar Kocyigit]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 17:35:42 GMT</pubDate>
            <atom:updated>2026-11-01T17:35:42.506Z</atom:updated>
            <content:encoded><![CDATA[<h4>We moved our SSRS reports to Power BI paginated reports. The subscriptions moved to a Databricks App on Unity Catalog.</h4><p>When we migrated our SSRS reports to Power BI paginated reports, the reports themselves got most of the attention. They were converted, published and compared against the originals.</p><p>But the first question business users ask after a migration like this usually isn’t about the reports. It’s: <em>“Where’s my Monday email?”</em></p><p>In SSRS, a report was rarely just something people opened. It was something that arrived in their inbox at 07:00 as a PDF or an Excel file, filtered to their region. Subscriptions are the part people actually depend on, and they’re easy to leave out of a migration plan.</p><p>This post walks through how I rebuilt them. Our data already lives in Databricks Unity Catalog, so the subscription layer now runs as a Databricks App, right next to the data.</p><h3>What SSRS subscriptions were actually doing</h3><p>Before building anything, I listed what the old subscriptions did. It came down to four jobs:</p><ol><li><strong>Standard subscriptions.</strong> A fixed list of recipients and a fixed set of parameter values, on a schedule.</li><li><strong>Data-driven subscriptions.</strong> A query decides who gets the report, with which parameters and in which format. One subscription can produce hundreds of personalised emails.</li><li><strong>Every export format.</strong> PDF, Excel, Word, PowerPoint, CSV, XML, MHTML and images.</li><li><strong>Email delivery.</strong> The file arrives as an attachment, with a subject and a body.</li></ol><p>Behind the scenes, SSRS ran subscriptions as SQL Server Agent jobs and handed the output to its email delivery extension. When the report server is switched off, all of that goes with it. All four jobs still had to work.</p><h3>Why Databricks</h3><p>Power BI Service has its own subscriptions. Our requirements pointed somewhere else:</p><ul><li><strong>The recipient logic belongs with the data.</strong> Data-driven recipient lists come from tables we already govern in Unity Catalog. I didn’t want to copy them into another system.</li><li><strong>Subscriptions should be data too.</strong> If the definitions are Delta tables, they’re queryable, auditable and versioned like everything else in the lakehouse.</li><li><strong>One place for self-service, approvals and monitoring.</strong> Report owners needed to manage their own subscriptions without filing tickets, with an approval step before anything is sent, and with a clear view of what failed.</li></ul><p>Databricks Apps made that straightforward. The app runs inside the workspace, next to Unity Catalog, so there’s no extra server to host and no data to move.</p><h3>The architecture: one app, three stages</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5-3H_bxtCjGbwaMaUC38fQ.png" /></figure><p>The delivery flow. Subscriptions and recipient data live in Unity Catalog, and every run goes through three stages: resolve, export and deliver.</p><p>The app has four parts: a UI where owners manage their subscriptions, a scheduler, a fan-out step that resolves recipients, and an export engine that talks to Power BI. Delivery is handed to an Azure Logic App.</p><p>Every run goes through the same three stages:</p><ol><li><strong>Resolve:</strong> work out who gets what.</li><li><strong>Export:</strong> render the paginated report in the requested format.</li><li><strong>Deliver:</strong> email it as an attachment.</li></ol><h3>Stage 1: Resolve, standard or data-driven</h3><p>A scheduler inside the app wakes up on each subscription’s cron schedule and picks up the subscriptions that are due. Only approved subscriptions are eligible; more on that below.</p><p>From there, the two subscription types split:</p><ul><li>A <strong>standard</strong> subscription already contains everything: recipients, parameter values and format. It becomes exactly one delivery.</li><li>A <strong>data-driven</strong> subscription carries a query. At send time, the app runs that query against Unity Catalog, and every row becomes its own delivery, with its own recipients, parameters and format.</li></ul><p>A recipient query looks like this (table and column names simplified):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g1oZGsfQj95gd4fofQgSpw.png" /></figure><pre>SELECT email,<br>       region           AS p_region,<br>       preferred_format AS format<br>FROM   main.finance.ar_report_recipients<br>WHERE  is_active = true</pre><p>In the UI, the owner maps the columns to delivery fields: email to the recipient, p_region to the report&#39;s Region parameter, and format to the output format, with a default when it&#39;s empty. Because the query runs at send time, the list is always current. When someone joins a regional team, they start getting the report without anyone touching the subscription.</p><p>Creating a data-driven subscription. The recipient query runs against Unity Catalog, and every row becomes one delivery.</p><p>The subscription definitions are themselves a Delta table. A simplified version:</p><pre>CREATE TABLE main.subscriptions.definitions (<br>  subscription_id   STRING,<br>  report_id         STRING,<br>  workspace_id      STRING,<br>  subscription_type STRING,              -- standard | data_driven<br>  output_format     STRING,              -- PDF, XLSX, ...<br>  schedule_cron     STRING,<br>  recipients        ARRAY&lt;STRING&gt;,       -- standard<br>  parameters        MAP&lt;STRING,STRING&gt;,  -- standard<br>  recipient_query   STRING,              -- data-driven<br>  status            STRING,              -- draft | pending | approved<br>  owner             STRING,<br>  updated_at        TIMESTAMP<br>)</pre><h3>Stage 2: Export, through Power BI’s Export To File API</h3><p>Every delivery is rendered by the <strong>Export To File</strong> endpoint of the Power BI REST API. It’s asynchronous and works in three steps: start an export job, poll it until it finishes, then download the file.</p><p>A simplified version of the export call:</p><pre>import time<br>import requests</pre><pre>API = &quot;https://api.powerbi.com/v1.0/myorg&quot;</pre><pre>def export_paginated(token, group_id, report_id, fmt, params):<br>    h = {&quot;Authorization&quot;: f&quot;Bearer {token}&quot;}<br>    report = f&quot;{API}/groups/{group_id}/reports/{report_id}&quot;</pre><pre>    # 1. Start the export job<br>    values = [{&quot;name&quot;: k, &quot;value&quot;: v} for k, v in params.items()]<br>    body = {<br>        &quot;format&quot;: fmt,  # PDF, XLSX, DOCX, PPTX, CSV, XML, ...<br>        &quot;paginatedReportConfiguration&quot;: {&quot;parameterValues&quot;: values},<br>    }<br>    job = requests.post(f&quot;{report}/ExportTo&quot;, headers=h, json=body)<br>    job.raise_for_status()<br>    export_id = job.json()[&quot;id&quot;]</pre><pre>    # 2. Poll until the job reaches a terminal state<br>    while True:<br>        r = requests.get(f&quot;{report}/exports/{export_id}&quot;, headers=h)<br>        r.raise_for_status()<br>        status = r.json()[&quot;status&quot;]<br>        if status in (&quot;Succeeded&quot;, &quot;Failed&quot;):<br>            break<br>        time.sleep(int(r.headers.get(&quot;Retry-After&quot;, 5)))</pre><pre>    if status == &quot;Failed&quot;:<br>        raise RuntimeError(f&quot;Export {export_id} failed&quot;)</pre><pre>    # 3. Download the rendered file<br>    f = requests.get(f&quot;{report}/exports/{export_id}/file&quot;, headers=h)<br>    f.raise_for_status()<br>    return f.content</pre><p>This one API covers every format the old subscriptions used: PDF (including accessible PDF), Excel, Word, PowerPoint, CSV, XML, MHTML, and images as TIFF, PNG, JPEG, BMP, GIF or EMF.</p><p>A few things worth knowing before you rely on it, all from Microsoft’s documentation:</p><ul><li>The report has to sit in a workspace on <strong>Premium, Embedded or Fabric capacity</strong>. Premium Per User allows only one export request every five minutes, which rules it out for subscriptions.</li><li>The API allows at most <strong>250 concurrent paginated render requests</strong>. Beyond that you get <strong>429 Too Many Requests</strong>, so spread the load and retry with backoff.</li><li>Respect the Retry-After header while polling. A single export can run for at most <strong>60 minutes</strong>.</li></ul><h3>Stage 3: Deliver, with Logic Apps and Outlook</h3><p>Once the file is downloaded, the app posts it to an Azure Logic App with an HTTP trigger. A trimmed-down example of the payload:</p><pre>{<br>  &quot;to&quot;: &quot;ar-emea@example.com&quot;,<br>  &quot;subject&quot;: &quot;AR aging · EMEA&quot;,<br>  &quot;body&quot;: &quot;&lt;p&gt;Your AR aging report is attached.&lt;/p&gt;&quot;,<br>  &quot;fileName&quot;: &quot;AR_Aging_EMEA.pdf&quot;,<br>  &quot;contentBytes&quot;: &quot;&lt;base64-encoded file&gt;&quot;<br>}</pre><p>The workflow itself is short: <strong>When an HTTP request is received</strong>, then <strong>Send an email (V2)</strong> from the Office 365 Outlook connector, with the attachment’s name and content taken from the payload. To the recipient, it looks exactly like the SSRS email they used to get.</p><p>Keeping delivery in a Logic App has a useful side effect. The app never holds mail credentials, and adding another channel later, such as a Teams message or a SharePoint folder, is a change to the workflow rather than to the app.</p><h3>Self-service, with an approval gate</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5-3H_bxtCjGbwaMaUC38fQ.png" /></figure><p>Approval and monitoring. Nothing reaches the scheduler until it’s approved, and every run feeds the monitoring view.</p><p>In SSRS, data-driven subscriptions needed someone with elevated permissions on the report server. Here, report owners create their own subscriptions in the app, standard or data-driven, without filing a ticket.</p><p>Self-service doesn’t mean no control. A new subscription starts as an approval request, and nothing is sent until an approver signs off. Only then does it land in the table the scheduler reads.</p><p>This matters most for data-driven subscriptions. A query that returns the wrong rows can email the wrong report to the wrong people, and a review before the first run is cheap insurance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MBuEDqZdn9PU65BbvKd4kQ.png" /></figure><p>The approval queue. The approver sees the report, the recipients, the query and the schedule before anything is scheduled.</p><h3>Monitoring: failures you can see</h3><p>In many SSRS setups, a failed subscription is easy to miss. The status sits on a page nobody checks, and the first alert is a user asking where their report went.</p><p>The app has a monitoring dashboard for exactly that: which subscriptions failed and why. The reasons are the kind you’d expect from this stack: the capacity throttling exports, a report parameter without a value, an export that timed out, an attachment too large for a mailbox. Grouped together, the pattern is obvious, and the owner can act on it the same morning.</p><p>The monitoring view: failed deliveries per day, the reasons behind them and the runs that still need attention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0T3FbhPWjYUUfOPrS6lESQ.png" /></figure><h3>What I’d tell you before you build this</h3><ul><li><strong>Treat subscriptions as data.</strong> With the definitions in Delta, they’re easy to query, audit and report on. “Who receives the AR aging report?” becomes a SQL query.</li><li><strong>Design data-driven subscriptions as a fan-out.</strong> One row, one delivery. A bad row then fails on its own instead of taking the whole subscription down.</li><li><strong>Don’t schedule everything at 07:00.</strong> Export capacity is shared and throttled. Staggering schedules by a few minutes avoids a burst of 429s at the start of the day.</li><li><strong>Watch attachment sizes.</strong> A large Excel export can exceed mailbox limits. For big files, a link can work better than an attachment.</li><li><strong>Keep delivery decoupled.</strong> The app decides what to send; the Logic App decides how it gets there.</li></ul><h3>Wrapping up</h3><p>Migrating the reports was only half of retiring SSRS. The subscriptions were the half business users noticed.</p><p>Rebuilding them as a Databricks App kept all four jobs SSRS did: standard and data-driven subscriptions, every export format and email delivery. It also added two things our SSRS setup didn’t have: self-service with an approval gate, and a single place to see what failed.</p><p>If you’re moving SSRS onto a Databricks lakehouse, I’d love to hear how you’re handling subscriptions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd99ce7c4dad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Should Data Engineers Use Local LLMs?]]></title>
            <link>https://medium.com/@kocyigityasar/how-should-data-engineers-use-local-llms-eef7a32cbfd8?source=rss-1ab647a5c7d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/eef7a32cbfd8</guid>
            <dc:creator><![CDATA[Yasar Kocyigit]]></dc:creator>
            <pubDate>Mon, 20 Jul 2026 15:27:23 GMT</pubDate>
            <atom:updated>2026-07-20T15:27:23.313Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_N_4Dc1mYIm34HeGWdWlSg.png" /></figure><h3>Building a Fully Local ADF-to-Fabric Migration Agent with Hermes and NVIDIA DGX Spark</h3><p>Local LLMs are often discussed in terms of privacy, cost savings, and offline access.</p><p>Those benefits are real, but for Data Engineers, the more interesting question is:</p><p><strong>How can we turn a local LLM into a reliable engineering agent that understands our tools, standards, and migration processes?</strong></p><p>To explore this, I am working on a fully local ADF-to-Microsoft Fabric migration agent using:</p><ul><li>NVIDIA DGX Spark</li><li>Hermes Agent</li><li>Qwen3.6–35B-A3B</li><li>vLLM</li><li>Custom Hermes skills</li><li>Deterministic validation scripts</li><li>Git-based change tracking</li></ul><p>The goal is not to build another chatbot.</p><p>The goal is to build a domain-specific Data Engineering agent that can assess Azure Data Factory environments, understand pipeline dependencies, identify Fabric compatibility issues, generate migration artifacts, validate its output, and prepare changes for human review.</p><p>All inference runs locally, without relying on a paid cloud LLM API.</p><h3>Why Local LLMs Matter for Data Engineering</h3><p>Data Engineering environments often contain sensitive information:</p><ul><li>Connection details</li><li>Pipeline configurations</li><li>Table and column names</li><li>Business logic</li><li>Infrastructure metadata</li><li>Operational logs</li><li>Internal architecture patterns</li><li>Deployment scripts</li></ul><p>Sending this context to a cloud model may not always be acceptable.</p><p>A local LLM allows the agent to work directly with internal repositories, exported pipeline definitions, logs, and configuration files while keeping the data inside the local environment.</p><p>However, privacy alone is not enough.</p><p>A useful Data Engineering agent must also be:</p><ul><li>Accurate</li><li>Repeatable</li><li>Testable</li><li>Explainable</li><li>Controlled</li><li>Aware of engineering standards</li></ul><p>This is where Hermes skills and deterministic tools become important.</p><h3>The Core Architecture</h3><p>The architecture looks like this:</p><pre>Data Engineer<br>     │<br>     ▼<br>Hermes ADF Migration Agent<br>     │<br>     ├── Agent Memory<br>     ├── Project Rules<br>     ├── Migration Skills<br>     ├── Tool Orchestration<br>     └── Human Approval Gates<br>     │<br>     ▼<br>Local Migration Workspace<br>     │<br>     ├── Exported ADF JSON<br>     ├── Approved Migration Examples<br>     ├── Compatibility Rules<br>     ├── Git Repository<br>     └── Migration State Database<br>     │<br>     ▼<br>Deterministic Tools<br>     │<br>     ├── Python Conversion Scripts<br>     ├── JSON Schema Validation<br>     ├── Dependency Graph Generation<br>     ├── Azure CLI<br>     ├── ADF and Fabric APIs<br>     ├── pytest<br>     └── Git<br>     │<br>     ▼<br>vLLM<br>     │<br>     ▼<br>Qwen3.6-35B-A3B<br>     │<br>     ▼<br>NVIDIA DGX Spark</pre><p>The main design principle is simple:</p><p><strong>Use the LLM for reasoning and orchestration, and use deterministic tools for generation, validation, and deployment.</strong></p><p>This separation is critical.</p><p>The LLM should decide what needs to happen, which skill to use, which artifacts are affected, and what risks exist.</p><p>It should not be trusted to generate and deploy hundreds of lines of migration JSON without validation.</p><h3>What Hermes Does</h3><p>Hermes acts as the main orchestrator.</p><p>It receives a migration request, loads the relevant project rules and skills, inspects the migration workspace, calls tools, analyzes results, and continues until the task is complete.</p><p>A typical request may look like this:</p><pre>Assess this Azure Data Factory environment and prepare an ADF-to-Fabric migration plan.</pre><pre>Do not deploy anything.</pre><p>Hermes would then:</p><ol><li>Export or read ADF artifacts.</li><li>Build an inventory.</li><li>Extract pipeline dependencies.</li><li>Identify linked services, datasets, triggers, and child pipelines.</li><li>Check activity compatibility with Microsoft Fabric.</li><li>Assign complexity and confidence scores.</li><li>Classify each artifact.</li><li>Generate a migration report.</li><li>Propose migration waves.</li><li>Flag anything that requires manual review.</li></ol><p>Hermes is not being fine-tuned for this task.</p><p>Instead, it is being specialized through skills, project rules, approved examples, memory, and tools.</p><p>That makes the system easier to update and easier to audit than traditional model fine-tuning.</p><h3>Teaching Hermes Through Skills</h3><p>A Hermes skill is a reusable procedure that explains how a specific task should be performed.</p><p>Rather than creating one large ADF migration skill, the migration process should be split into smaller domain-specific skills.</p><p>For example:</p><pre>adf-inventory<br>adf-dependency-analysis<br>fabric-compatibility-assessment<br>pipeline-conversion<br>connection-mapping<br>validation<br>migration-reporting<br>deployment</pre><p>Each skill should define:</p><ul><li>When the skill should be used</li><li>Required inputs</li><li>Step-by-step procedures</li><li>Approved patterns</li><li>Validation checks</li><li>Safety restrictions</li><li>Completion criteria</li><li>Examples of correct output</li><li>Known unsupported scenarios</li></ul><p>An example skill could look like this:</p><pre># ADF-to-Fabric Pipeline Migration</pre><pre>## Use this skill when</pre><pre>An Azure Data Factory pipeline needs to be assessed or migrated to Microsoft Fabric Data Factory.</pre><pre>## Process</pre><pre>1. Read the source ADF pipeline JSON.<br>2. Extract activities, parameters, references, and dependencies.<br>3. Check every activity against the compatibility matrix.<br>4. Classify the pipeline as AUTO, ASSISTED, MANUAL, or BLOCKED.<br>5. Generate a canonical migration specification.<br>6. Convert the specification using deterministic scripts.<br>7. Run validation.<br>8. Produce a migration report.</pre><pre>## Safety</pre><pre>- Do not modify the source ADF environment.<br>- Do not invent unsupported mappings.<br>- Do not deploy without explicit approval.<br>- Flag uncertain conversions for manual review.</pre><pre>## Completion Criteria</pre><pre>- The generated artifact is valid JSON.<br>- All referenced connections exist.<br>- Unsupported activities are documented.<br>- Parameters are mapped.<br>- Validation results are attached.</pre><p>This gives Hermes procedural knowledge without modifying the underlying Qwen model.</p><h3>Memory, Skills, and Project Rules</h3><p>These three concepts should remain separate.</p><h3>Memory</h3><p>Memory stores stable facts about the environment.</p><p>Examples:</p><ul><li>Development workspace name</li><li>Azure subscription name</li><li>Repository locations</li><li>Naming conventions</li><li>Default deployment environment</li><li>Approval requirements</li></ul><h3>Skills</h3><p>Skills store procedures.</p><p>Examples:</p><ul><li>How to inventory an ADF factory</li><li>How to migrate a Copy activity</li><li>How to map linked services to Fabric connections</li><li>How to validate generated pipelines</li><li>How to produce a migration report</li></ul><h3>Project Rules</h3><p>A repository-level file such as AGENTS.md stores rules specific to that project.</p><p>For example:</p><pre>## Architecture Rules</pre><pre>- Use parameterized pipelines.<br>- Do not hard-code environment values.<br>- Use managed identity whenever supported.<br>- Preserve existing logging behavior.<br>- Use Lakehouse Bronze as the default landing destination.</pre><pre>## Safety Rules</pre><pre>- Source ADF is read-only.<br>- DEV deployment is allowed only after validation.<br>- QA requires approval.<br>- Production always requires explicit human approval.</pre><p>This structure prevents unrelated information from being mixed together and keeps the agent context clean.</p><h3>Why Deterministic Conversion Is Necessary</h3><p>A major risk with LLM-generated infrastructure is hallucinated configuration.</p><p>An LLM may:</p><ul><li>Invent unsupported properties</li><li>Use the wrong activity structure</li><li>Reference a missing connection</li><li>Produce invalid expressions</li><li>Misconfigure dependencies</li><li>Skip environment-specific settings</li><li>Generate syntactically correct but operationally incorrect JSON</li></ul><p>The safer approach is to let Hermes generate a canonical migration specification first.</p><p>For example:</p><pre>pipeline:<br>  name: pl_customer_ingestion</pre><pre>  source:<br>    type: rest<br>    connection: con_customer_api</pre><pre>  target:<br>    type: fabric_lakehouse<br>    workspace: ws_data_dev<br>    lakehouse: lh_bronze<br>    path: Files/customer</pre><pre>  activities:<br>    - type: copy<br>      retry_count: 3<br>      timeout_minutes: 60</pre><pre>  incremental:<br>    strategy: modified_timestamp<br>    watermark_column: modifiedDate</pre><pre>  migration:<br>    classification: AUTO<br>    confidence: 0.95<br>    manual_review_required: false</pre><p>A deterministic Python generator then converts this specification into the final Fabric artifact.</p><p>This provides several benefits:</p><ul><li>The output format is predictable.</li><li>Validation is easier.</li><li>Mappings can be unit-tested.</li><li>Changes are repeatable.</li><li>The LLM does not need to generate every low-level property directly.</li><li>The migration logic can be version-controlled.</li></ul><h3>Migration Classification</h3><p>Not every pipeline should be treated as fully automatable.</p><p>The agent should classify each artifact into one of four categories.</p><h3>AUTO</h3><p>The pipeline can be converted automatically using known and validated mappings.</p><h3>ASSISTED</h3><p>Most of the pipeline can be converted, but a human must review specific sections.</p><h3>MANUAL</h3><p>The pipeline requires significant redesign or unsupported functionality.</p><h3>BLOCKED</h3><p>The pipeline depends on a connector, activity, runtime, or feature that currently prevents migration.</p><p>An example assessment report might look like this:</p><pre>summary:<br>  pipelines: 120<br>  datasets: 76<br>  linked_services: 18<br>  triggers: 14</pre><pre>classification:<br>  auto: 64<br>  assisted: 32<br>  manual: 18<br>  blocked: 6</pre><pre>major_risks:<br>  - custom_dotnet_activity<br>  - unsupported_connector<br>  - self_hosted_integration_runtime<br>  - hardcoded_secrets<br>  - complex_mapping_data_flows</pre><p>This is more valuable than an agent that simply claims every pipeline can be migrated.</p><h3>Validation Is the Real Source of Accuracy</h3><p>The LLM does not make the system reliable by itself.</p><p>The validation loop does.</p><p>The target workflow should be:</p><pre>Analyze<br>→ Generate<br>→ Validate<br>→ Test<br>→ Inspect Errors<br>→ Fix<br>→ Revalidate<br>→ Show Git Diff<br>→ Request Approval</pre><p>Validation should happen at several levels.</p><h3>Static Validation</h3><ul><li>Valid JSON</li><li>Required properties</li><li>Naming standards</li><li>Dependency references</li><li>Parameter types</li><li>Connection references</li><li>No embedded credentials</li></ul><h3>Semantic Validation</h3><ul><li>Source and target mappings make sense</li><li>Activity behavior is preserved</li><li>Retry and timeout behavior is retained</li><li>Parameter values are correctly translated</li><li>Unsupported behavior is clearly flagged</li></ul><h3>Runtime Validation</h3><ul><li>Deploy to isolated DEV</li><li>Run with sample data</li><li>Validate execution status</li><li>Compare source and target row counts</li><li>Check schema</li><li>Check duplicate behavior</li><li>Validate incremental logic</li><li>Re-run to test idempotency</li></ul><h3>Business Reconciliation</h3><ul><li>Record counts</li><li>Business keys</li><li>Aggregated values</li><li>Null distribution</li><li>Duplicate detection</li><li>Watermark consistency</li><li>Expected output files or tables</li></ul><p>Accuracy should be measured by successful validation, not by how convincing the generated explanation sounds.</p><h3>Human Approval Must Remain Part of the Architecture</h3><p>A local agent should not automatically deploy migration artifacts to production.</p><p>The workflow should include explicit gates:</p><pre>Assessment<br>→ Human Scope Approval<br>→ Artifact Generation<br>→ Git Diff Review<br>→ DEV Deployment<br>→ Automated Testing<br>→ QA Approval<br>→ Business Validation<br>→ Production Approval</pre><p>Permissions should also be separated.</p><pre>Source ADF<br>Read-only</pre><pre>Fabric DEV<br>Write access allowed</pre><pre>Fabric QA<br>Approval required</pre><pre>Fabric Production<br>Explicit approval required</pre><p>The agent may prepare commands, plans, diffs, and deployment packages.</p><p>It should not execute destructive or production-level operations without approval.</p><h3>Why Qwen3.6–35B-A3B</h3><p>For this project, Qwen3.6–35B-A3B is used as the local model behind Hermes.</p><p>The important characteristics are:</p><ul><li>Strong coding ability</li><li>Tool-calling support</li><li>Long-context capability</li><li>Good balance between performance and model size</li><li>MoE architecture</li><li>Suitable for multi-step agent workflows</li><li>Practical for deployment on DGX Spark</li></ul><p>The model is served through vLLM as an OpenAI-compatible endpoint.</p><p>The flow is:</p><pre>Hermes<br>→ vLLM API<br>→ Qwen3.6-35B-A3B<br>→ DGX Spark</pre><p>vLLM is a better fit than a desktop-focused runtime when the environment needs:</p><ul><li>Concurrent requests</li><li>Multiple agents</li><li>Better KV-cache management</li><li>Prefix caching</li><li>Higher throughput</li><li>Long-running inference services</li><li>OpenAI-compatible API access</li></ul><h3>A Fully Local and Open Stack</h3><p>The project can be built using an entirely self-hosted stack.</p><pre>Hardware<br>NVIDIA DGX Spark</pre><pre>Agent<br>Hermes Agent</pre><pre>Model<br>Qwen3.6-35B-A3B</pre><pre>Inference<br>vLLM</pre><pre>Backend<br>Python and FastAPI</pre><pre>Frontend<br>React</pre><pre>State<br>SQLite or PostgreSQL</pre><pre>Version Control<br>Git</pre><pre>Validation<br>pytest<br>JSON Schema<br>Custom Python validators</pre><pre>Monitoring<br>Prometheus<br>Grafana<br>vLLM metrics</pre><p>There is no cloud LLM API cost.</p><p>Of course, local inference is not completely free because hardware, electricity, storage, and maintenance still have costs.</p><p>However, the architecture eliminates usage-based model API charges and keeps engineering data local.</p><h3>The Initial MVP</h3><p>The first version should remain intentionally narrow.</p><p>The MVP should support:</p><ul><li>Exported ADF pipeline JSON</li><li>Pipeline inventory</li><li>Dataset and linked-service inventory</li><li>Dependency analysis</li><li>Activity compatibility assessment</li><li>Pipeline classification</li><li>Basic Copy activity migration</li><li>Parameter mapping</li><li>Connection mapping</li><li>Deterministic artifact generation</li><li>Static validation</li><li>Git diff generation</li><li>Migration reporting</li></ul><p>The first version should not attempt:</p><ul><li>Complex Mapping Data Flow conversion</li><li>Custom .NET activity migration</li><li>Automatic production deployment</li><li>Secret migration</li><li>Fully autonomous infrastructure changes</li><li>Every possible ADF connector</li></ul><p>A smaller and measurable scope will produce a much more credible result.</p><h3>How the Agent Improves Over Time</h3><p>Hermes can gradually become more effective as new migration cases are completed.</p><p>For example:</p><ol><li>Hermes converts a pipeline.</li><li>Validation detects an incorrect watermark pattern.</li><li>The Data Engineer corrects the implementation.</li><li>The corrected pattern is added to the relevant skill.</li><li>A new regression test is created.</li><li>Future migrations reuse the improved procedure.</li></ol><p>The agent is therefore not learning through uncontrolled self-modification.</p><p>It is improving through:</p><ul><li>Reviewed skill updates</li><li>Approved examples</li><li>New validation rules</li><li>Expanded compatibility mappings</li><li>Regression tests</li><li>Version-controlled migration patterns</li></ul><p>Every improvement should be committed to Git and reviewed like normal engineering code.</p><h3>Measuring Whether It Works</h3><p>The project should be treated as an engineering experiment, not just a demo.</p><p>A useful benchmark dataset could contain:</p><pre>10 simple pipelines<br>10 parameterized pipelines<br>10 metadata-driven pipelines<br>10 pipelines with complex dependencies<br>5 partially supported pipelines<br>5 unsupported pipelines</pre><p>Several configurations could then be compared:</p><pre>A. Qwen with a basic prompt<br>B. Hermes with local Qwen<br>C. Hermes with migration skills<br>D. Hermes with skills and approved examples<br>E. Hermes with skills, examples, and deterministic validation</pre><p>Important metrics include:</p><ul><li>Successful migration percentage</li><li>Valid artifact percentage</li><li>Activity mapping accuracy</li><li>Parameter mapping accuracy</li><li>Unsupported-feature detection rate</li><li>Hallucinated property count</li><li>Manual correction time</li><li>Tool-call success rate</li><li>Total migration duration</li><li>Human review time</li><li>Regression-test pass rate</li></ul><p>The most important question is not:</p><p><strong>How many tokens per second does the model generate?</strong></p><p>The more important question is:</p><p><strong>How many pipelines can the agent migrate correctly with minimal human correction?</strong></p><h3>Broader Applications</h3><p>ADF-to-Fabric migration is only one example.</p><p>The same architecture can be applied to:</p><ul><li>SSIS-to-ADF migration</li><li>Synapse-to-Fabric migration</li><li>ADF-to-Databricks Workflows migration</li><li>Legacy SQL ETL modernization</li><li>Informatica migration</li><li>Airflow DAG modernization</li><li>Data-quality rule generation</li><li>Metadata-driven ingestion development</li><li>Infrastructure validation</li><li>Incident troubleshooting</li></ul><p>The reusable pattern is:</p><pre>Local LLM<br>+ Agent Orchestration<br>+ Domain Skills<br>+ Approved Examples<br>+ Deterministic Tools<br>+ Validation<br>+ Human Approval</pre><h3>Final Thought</h3><p>The best use of a local LLM for Data Engineering is not simply asking it to write SQL or generate pipeline JSON.</p><p>The real opportunity is to build specialized engineering agents that understand the environment, follow approved procedures, use tools, validate their own output, and remain under human control.</p><p>In this architecture:</p><ul><li>Qwen provides the general reasoning and coding capability.</li><li>Hermes manages the workflow and selects the right skills.</li><li>Skills capture migration knowledge.</li><li>Project rules define local standards.</li><li>Deterministic tools generate and validate artifacts.</li><li>Git provides traceability.</li><li>Human approvals protect critical environments.</li><li>DGX Spark keeps the entire process local.</li></ul><p>That is how a local LLM can move from being a chatbot to becoming a practical Data Engineering system.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eef7a32cbfd8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Benchmarked Google’s Gemma 4 on NVIDIA DGX Spark Here’s What 128GB of Unified Memory Can Do]]></title>
            <link>https://medium.com/@kocyigityasar/i-benchmarked-googles-gemma-4-on-nvidia-dgx-spark-here-s-what-128gb-of-unified-memory-can-do-14f4a6d28604?source=rss-1ab647a5c7d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/14f4a6d28604</guid>
            <dc:creator><![CDATA[Yasar Kocyigit]]></dc:creator>
            <pubDate>Fri, 03 Apr 2026 17:25:02 GMT</pubDate>
            <atom:updated>2026-04-03T17:25:02.153Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Running a 62GB open-source LLM at full precision on a desktop AI supercomputer, and discovering why quantization might actually be the smarter choice.</em></p><h3>The Day Gemma 4 Dropped</h3><p>On April 2, 2026, Google released Gemma 4 their most capable open-source LLM family to date. The headliner: a 31B-parameter dense model with a 256K context window, Apache 2.0 license, and benchmark numbers that rival closed-source models from just a year ago.</p><p>Within hours, I had it running locally on an NVIDIA DGX Spark no cloud, no API keys, no per-token billing. Just a desktop machine with 128GB of unified memory and a Blackwell GPU.</p><p>This article documents a systematic benchmark of Gemma 4 31B across two precision levels BF16 (full precision) and Q8_0 (8-bit quantization) on real coding and math reasoning tasks. The results challenged my assumptions about the precision vs. speed tradeoff.</p><h3>The Hardware: NVIDIA DGX Spark</h3><p>The DGX Spark is NVIDIA’s desktop AI supercomputer, and it’s a different beast from anything else on a desk:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*n2y9IwjrhnyqpcO7PyvEsA.png" /></figure><p>The 128GB unified memory is the key differentiator. Most consumer GPUs top out at 24GB VRAM, which means even a Q4-quantized 31B model requires offloading to system RAM. On the DGX Spark, the entire model — weights, KV cache, and context buffer — lives in one unified address space with full GPU bandwidth.</p><p>This is what makes running a 62GB BF16 model practical on a local machine.</p><h3>The Models Under Test</h3><p>I tested two quantization levels of the same model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6uQE5ZTLUuUHwdAUpnGZ8Q.png" /></figure><p><em>*BF16 uses 117GB at runtime because Ollama allocates the full 256K context window KV cache (~55GB) alongside the 62GB model weights.</em></p><p>Both models were served through Ollama 0.20.0 with default settings, accessed via the local API at localhost:11434.</p><h3>Benchmark Suite Design</h3><p>I built a custom benchmark suite targeting four dimensions:</p><h3>1. Speed Test</h3><p>Three generation tasks at different output lengths:</p><ul><li><strong>Short</strong>: 32 tokens (single sentence)</li><li><strong>Medium</strong>: 256 tokens (one paragraph)</li><li><strong>Long</strong>: 1,024 tokens (full code implementation)</li></ul><h3>2. HumanEval (Coding)</h3><p>20 function-completion problems from OpenAI’s HumanEval benchmark. Each problem provides a function signature with docstring; the model must generate a working implementation that passes unit tests. Problems include string manipulation, list operations, mathematical functions, and algorithmic challenges.</p><h3>3. GSM8K (Math Reasoning)</h3><p>20 grade-school math word problems requiring multi-step arithmetic reasoning. The model must show its work step-by-step and provide a final numerical answer.</p><h3>4. Quality &amp; Capability Tests</h3><p>5 targeted tests covering:</p><ul><li><strong>Turkish language understanding</strong> (multilingual capability)</li><li><strong>JSON generation</strong> (structured output)</li><li><strong>Code debugging</strong> (finding a bug in fibonacci implementation)</li><li><strong>Logical reasoning</strong> (syllogism evaluation)</li><li><strong>Instruction following</strong> (formatting constraints)</li></ul><p>All tests used temperature=0 for reproducibility. The full benchmark script is open-source in the repository.</p><h3>Results: The Numbers</h3><h3>Speed Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*AzoBhWDk6c953x4qF5v6Xg.png" /></figure><p><em>*The Q8 short test shows lower tok/s because with only 32 tokens, the prompt evaluation time dominates total elapsed time. For sustained generation, Q8 consistently delivers 6.4 tok/s.</em></p><p><strong>Key insight</strong>: BF16 at 62GB is compute-bound at 3.9 tok/s. Q8 at 34GB nearly doubles throughput. The memory bandwidth savings from halving the weight precision translates almost linearly to generation speed.</p><h3>HumanEval: Coding</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*GoOC2J0tYE-NS-3IOwLkSw.png" /></figure><p><strong>Wait is Q8 actually smarter than BF16?</strong></p><p>No. The BF16 run used a 512-token generation limit (my initial misconfiguration), while Q8 used 2,048 tokens. Gemma 4 uses a “thinking” mode by default — it generates &lt;think&gt;...&lt;/think&gt; reasoning tokens before the actual answer. With only 512 tokens, the model would spend ~400 tokens thinking and get cut off before producing the code.</p><p><strong>Evidence</strong>: Every BF16 failure took exactly 132.1 seconds (= 512 tokens / 3.9 tok/s), confirming the output was truncated, not wrong.</p><p>When properly configured, both BF16 and Q8 would achieve the same ~85% accuracy. The Q8 result is the real benchmark.</p><p><strong>Gemma 4 Q8 HumanEval Detailed Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*hZqjN-KbiMPBEayx2v88vg.png" /></figure><p>The 3 failures are genuine make_palindrome and how_many_times had code extraction issues (model wrapped code in explanatory text), and count_distinct_characters failed an assertion. These are not token-limit artifacts.</p><h3>GSM8K: Math Reasoning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*7i9ve8BmCvzqBfoNaRRoNA.png" /></figure><p>Same story BF16’s low score was entirely due to token truncation, not model capability.</p><p><strong>Gemma 4 Q8 GSM8K Detailed Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*gQTqLoiBl6u_BWTzmP9UEQ.png" /></figure><p>The 2 NO_ANSWER cases hit the 2,048 token limit (model was still “thinking”). The 1 WRONG answer (predicted 12 instead of 13 for a lemon tree break-even problem) is a genuine reasoning error the model likely made an off-by-one in the year calculation.</p><h3>Quality &amp; Capability Tests</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*VU0M5XwEa8n4sGO98Fm1rw.png" /></figure><p>Both variants scored perfectly. Some notable responses from Q8:</p><p><strong>Code Debugging</strong>: <em>“The recursive call uses n-3 instead of n-2, which does not follow the Fibonacci sequence formula.”</em> Identified the exact bug in one sentence.</p><p><strong>Logical Reasoning</strong>: <em>“No. The flowers that fade quickly are not necessarily roses.”</em> Correctly identified the syllogistic fallacy.</p><h3>The Real Comparison</h3><p>Here’s the cleaned-up comparison, accounting for the token-limit issue in BF16:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T14dC80_BXbRAISYRHvOoA.png" /></figure><p><em>*Estimated BF16 accuracy was artificially lowered by 512-token limit, but the model itself has the same capability.</em></p><p><em>**Similar total time because while Q8 generates faster, both spend time on prompt evaluation and thinking.</em></p><h3>Analysis: What I Learned</h3><h3>1. Thinking Mode is Expensive at Low Throughput</h3><p>Gemma 4’s built-in “thinking” mode (&lt;think&gt;...&lt;/think&gt;) is great for reasoning quality but costly when your hardware generates at single-digit tok/s. A simple math problem that needs 50 tokens of actual answer might generate 400+ tokens of chain-of-thought first.</p><p>At 6.4 tok/s (Q8), a 500-token thinking response takes ~78 seconds. At cloud speeds (100+ tok/s), the same response takes 5 seconds. This fundamentally changes how you configure token limits for local inference.</p><p><strong>Practical rule</strong>: For local BF16/Q8 inference with thinking models, set num_predict to at least 4x what you&#39;d expect the actual answer to be.</p><h3>2. Q8 is the Sweet Spot for DGX Spark</h3><p>With 128GB unified memory, you might assume “just run BF16 — we have the RAM.” But the data tells a different story:</p><ul><li><strong>Speed</strong>: Q8 is 64% faster (6.4 vs 3.9 tok/s)</li><li><strong>Quality</strong>: Identical across all benchmarks</li><li><strong>RAM headroom</strong>: Q8 leaves ~58GB free vs ~11GB for BF16</li></ul><p>That 58GB of headroom matters. You can run a second model simultaneously, keep a large RAG index in memory, or use the remaining RAM for preprocessing pipelines. BF16 leaves you with barely enough to run htop.</p><h3>3. 85% on HumanEval is Remarkable for Local Open-Source</h3><p>For context, here’s where Gemma 4 31B Q8 sits relative to other models:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*KTMmrH1HFiAh-0HChQvxRQ.png" /></figure><p><em>*Llama 3.1 70B requires ~140GB for BF16, doesn’t fit on DGX Spark without quantization.</em></p><p>Gemma 4 31B is delivering near-frontier coding capability from a box that sits on your desk. No API costs, no data leaving your network, no rate limits.</p><h3>4. The 256K Context Window is Real But Expensive</h3><p>Ollama allocated a 256K token context window by default, consuming ~55GB of RAM for the KV cache alone. For most practical use cases, you can reduce this significantly:</p><pre># Reduce context to 8K — saves ~50GB RAM<br>ollama run gemma4:31b-it-q8_0 --ctx-size 8192</pre><p>Unless you’re processing very long documents, a 8K-32K context window is sufficient and dramatically reduces memory pressure.</p><h3>How to Reproduce This</h3><h3>Prerequisites</h3><ul><li>NVIDIA DGX Spark (or any system with 128GB+ unified/GPU memory)</li><li>Ollama 0.20.0+ installed</li></ul><h3>Setup</h3><pre># Install the Q8 model (recommended)<br>ollama pull gemma4:31b-it-q8_0</pre><pre># Or the full BF16 if you want maximum precision<br>ollama pull gemma4:31b-it-bf16</pre><pre># Verify it&#39;s running<br>ollama list</pre><h3>Quick Test</h3><pre>ollama run gemma4:31b-it-q8_0 &quot;Write a Python function that checks if a string is a palindrome&quot;</pre><h3>Run the Full Benchmark</h3><pre>git clone &lt;repo-url&gt;<br>cd autoresearch-log-classifier<br>python3 -m venv bench_env<br>source bench_env/bin/activate<br>pip install human-eval requests</pre><pre># Run with Q8 (recommended)<br>BENCH_MODEL=gemma4:31b-it-q8_0 python3 gemma4_benchmark.py</pre><pre># Or with BF16<br>BENCH_MODEL=gemma4:31b-it-bf16 python3 gemma4_benchmark.py</pre><h3>Conclusion</h3><p>Running Gemma 4 31B locally on DGX Spark is not just feasible it’s practical. At Q8 precision, you get:</p><ul><li><strong>85% on HumanEval</strong> (near GPT-4 level coding)</li><li><strong>85% on GSM8K</strong> (strong math reasoning)</li><li><strong>100% on quality tests</strong> (multilingual, structured output, logic)</li><li><strong>6.4 tok/s</strong> sustained generation</li><li><strong>Zero API costs</strong>, complete data privacy</li></ul><p>The biggest surprise was that Q8 quantization introduces no measurable quality degradation on these benchmarks while delivering 64% better throughput and using 47GB less memory. For anyone with access to a DGX Spark, Q8 is unambiguously the right choice for the 31B model.</p><p>We’re at a point where a single desktop machine can run a model that would have been considered frontier-class 18 months ago. The gap between “what you can run locally” and “what the best cloud APIs offer” is shrinking fast and Gemma 4 on DGX Spark is proof.</p><p><strong>Hardware</strong>: NVIDIA DGX Spark, GB10 Blackwell, 128GB unified memory <strong>Model</strong>: Google Gemma 4 31B Dense (April 2, 2026), Apache 2.0 License <strong>Inference</strong>: Ollama 0.20.0 <strong>Benchmark date</strong>: April 3, 2026 <strong>Total benchmark time</strong>: 91 minutes (Q8), 93 minutes (BF16)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14f4a6d28604" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NVIDIA DGX Spark Nemotron 3 Nano 30B: 1M Context Window Benchmark]]></title>
            <link>https://medium.com/@kocyigityasar/nvidia-dgx-spark-nemotron-3-nano-30b-1m-context-window-benchmark-57b4d0809991?source=rss-1ab647a5c7d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/57b4d0809991</guid>
            <dc:creator><![CDATA[Yasar Kocyigit]]></dc:creator>
            <pubDate>Mon, 16 Mar 2026 02:14:58 GMT</pubDate>
            <atom:updated>2026-03-16T02:20:04.840Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong><em>Date:</em></strong><em> March 15, 2026</em></blockquote><blockquote><strong><em>Hardware:</em></strong><em> NVIDIA DGX Spark</em></blockquote><blockquote><strong><em>Model:</em></strong><em> NVIDIA Nemotron 3 Nano 30B (Q4_K_M GGUF)</em></blockquote><blockquote><strong><em>Inference Engine:</em></strong><em> llama.cpp (compiled from source with CUDA sm_121)</em></blockquote><blockquote><strong><em>Test:</em></strong><em> Needle-in-a-Haystack across 7K–700K tokens</em></blockquote><h3>Table of Contents</h3><ol><li>Executive Summary</li><li>Hardware Specifications</li><li>Model Architecture</li><li>The 1M Context Window Challenge</li><li>Infrastructure &amp; Setup</li><li>Needle-in-a-Haystack Test Methodology</li><li>Results</li><li>Performance Analysis</li><li>Ollama Baseline Benchmarks</li><li>Inference Engine Comparison</li><li>Failed Approaches &amp; Lessons Learned</li><li>Key Takeaways</li><li>Reproduction Guide</li></ol><h3>Executive Summary</h3><p>We successfully tested the <strong>NVIDIA Nemotron 3 Nano 30B</strong> model with a <strong>1M token context window</strong> on the <strong>NVIDIA DGX Spark</strong>, achieving <strong>perfect needle-in-a-haystack retrieval at 699,676 tokens</strong> — the equivalent of approximately <strong>5 full novels</strong> or <strong>2,800 pages of text</strong> in a single prompt.</p><p>The key breakthrough was using <strong>Q4_0 KV cache quantization</strong> via llama.cpp, which compressed the KV cache from <strong>96 GB (FP16) down to just 1.7 GB</strong> — a <strong>56x reduction</strong> — making 1M context feasible on a device with 128 GB unified memory.</p><h3>Headline Numbers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*W_WhlH4kDfQwqx4lSoGZOA.png" /></figure><h3>Hardware Specifications</h3><h3>NVIDIA DGX Spark</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*Rm-Y9cRj4-BJkJdidpxLLA.png" /></figure><p>The DGX Spark is NVIDIA’s compact AI supercomputer designed for desktop deployment. Its 128 GB unified memory architecture allows both CPU and GPU to access the same memory pool, which is critical for running large language models with extended context windows.</p><h3>Model Architecture</h3><h3>NVIDIA Nemotron 3 Nano 30B</h3><p>Nemotron 3 Nano 30B is a <strong>hybrid Mamba-2 + Attention Mixture-of-Experts (MoE)</strong> model a fundamentally different architecture from standard Transformers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*MyuO-jqbDDZQ2VAxm4wcHg.png" /></figure><h3>Why This Architecture Matters for Long Context</h3><p>The hybrid Mamba-2 + Attention design is uniquely suited for ultra-long context:</p><ol><li><strong>Mamba-2 layers use recurrent state, not KV cache.</strong> Only the 6 attention layers require key-value caching. Traditional Transformers with 52 attention layers would need ~8x more KV cache memory.</li><li><strong>MoE activates only 6 of 128 experts per token</strong>, making it computationally efficient despite the 30B total parameter count. Only 3.5B parameters are active for any given token.</li><li><strong>The recurrent Mamba-2 state is tiny</strong> only 190 MB for the entire recurrence buffer, regardless of context length.</li></ol><h3>Memory Breakdown at 1M Context</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*Z81FIlFeajCTDxOBEUQ3ZA.png" /></figure><h3>The 1M Context Window Challenge</h3><h3>What 1M Tokens Means</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*FuhbiIlVywsWdWPrG7PWVg.png" /></figure><h3>Why It’s Difficult</h3><p>Running 1M context is challenging because:</p><ol><li><strong>KV cache scales linearly</strong> with context length. At FP16 precision with standard Transformer architectures, 1M context can require 200+ GB of KV cache alone.</li><li><strong>Attention computation scales quadratically</strong> with sequence length (O(n²)), making long sequences exponentially slower.</li><li><strong>Memory bandwidth</strong> becomes the bottleneck — the entire KV cache must be read for each new token generated.</li></ol><h3>How I Solved It</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JoPeUpgEISNb-QGphedP3g.png" /></figure><h3>Infrastructure &amp; Setup</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gvj3tEAXfT2FSsSh-vxunA.png" /></figure><h3>Software Stack</h3><pre>cd /home/yasarkocyigit/llama.cpp<br>CUDACXX=/usr/local/cuda/bin/nvcc cmake -B build \<br>  -DGGML_CUDA=ON \<br>  -DCMAKE_CUDA_ARCHITECTURES=121<br>cmake --build build -j$(nproc)</pre><h3>llama.cpp Build Configuration</h3><pre>cd /home/yasarkocyigit/llama.cpp<br>CUDACXX=/usr/local/cuda/bin/nvcc cmake -B build \<br>  -DGGML_CUDA=ON \<br>  -DCMAKE_CUDA_ARCHITECTURES=121<br>cmake --build build -j$(nproc)</pre><p><strong>Critical note:</strong> The DGX Spark’s Blackwell GB10 GPU uses compute capability <strong>12.1 (sm_121)</strong>, not 12.0. Compiling with the wrong architecture (120) results in:</p><pre>CUDA error: no kernel image is available for execution on the device</pre><h3>Server Launch Configuration</h3><pre>./llama-server \<br>  -m /path/to/nemotron-q4_k_m.gguf \<br>  -c 1048576 \          # 1M context window<br>  -ngl 999 \            # Offload all layers to GPU<br>  -np 1 \               # Single slot (maximize memory for context)<br>  -ctk q4_0 \           # KV cache key quantization: 4-bit<br>  -ctv q4_0 \           # KV cache value quantization: 4-bit<br>  --host 0.0.0.0 \<br>  --port 8080</pre><h3>Memory Allocation (from server logs)</h3><pre>llama_kv_cache: size = 1728.00 MiB (1048576 cells, 6 layers, 4/1 seqs)<br>  K (q4_0):  864.00 MiB<br>  V (q4_0):  864.00 MiB<br>llama_memory_recurrent: size = 190.47 MiB (4 cells, 52 layers, 4 seqs)<br>  R (f32):   6.47 MiB<br>  S (f32): 184.00 MiB<br>CUDA0 compute buffer size = 3090.79 MiB<br>CUDA_Host compute buffer size = 2058.51 MiB</pre><h3>Needle-in-a-Haystack Test Methodology</h3><h3>What Is Needle-in-a-Haystack?</h3><p>The Needle-in-a-Haystack (NIAH) test is a standard benchmark for evaluating a language model’s ability to retrieve specific information from a large context. A unique “needle” — a distinctive piece of information — is embedded at a specific position within a large body of irrelevant “haystack” text. The model must find and accurately report the needle.</p><h3>Our Test Design</h3><p><strong>The Needle:</strong></p><pre>The secret project codename is AURORA-7749 and it was initiated on March 15, 2019<br>by Dr. Helena Vasquez in the underground laboratory beneath Mount Kailash.</pre><p><strong>The Haystack:</strong> Randomly assembled paragraphs covering diverse topics (maritime navigation, photosynthesis, cathedral architecture, volcanic eruptions, genetics, Roman engineering, sound physics, butterfly migration, economics, mountain formation, immunology, Renaissance art, the water cycle) interspersed with randomized factoid sentences.</p><p><strong>Needle Placement:</strong> Middle of the context (50% depth)</p><p><strong>Scoring Criteria (3 points):</strong></p><ol><li>Project codename found: AURORA-7749</li><li>Initiator name found: Dr. Helena Vasquez</li><li>Date found: March 15, 2019</li></ol><p><strong>Pass threshold:</strong> 2/3 points = PARTIAL, 3/3 = PASS</p><h3>Test Progression</h3><p>Seven context sizes tested: 10K → 50K → 100K → 250K → 500K → 750K → 1M tokens</p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*05NKy92oEr_lSFJVKtQPag.png" /></figure><h3>Accuracy: 100% Across All Context Sizes</h3><p>Every single test returned the correct needle with all three details (codename, name, date) — a perfect 21/21 score across all tests.</p><h3>Example Model Response (at 699,676 tokens)</h3><pre>**Secret project codename:** AURORA‑7749<br>**Initiation date:** March 15, 2019<br>**Initiator:** Dr. Helena Vasquez (in the underground laboratory beneath Mount Kailash)</pre><p>The model’s internal reasoning:</p><pre>We need to answer: &quot;What is the secret project codename, when was it initiated,<br>and who initiated it?&quot; The text includes a line: &quot;The secret project codename is<br>AURORA-7749 and it was initiated on March 15, 2019 by Dr. Helena Vasquez in the<br>underground laboratory beneath Mount Kailash.&quot;</pre><h3>Performance Analysis</h3><h3>Prompt Processing Speed vs Context Size</h3><pre>Tokens     | Speed (tok/s) | Relative to 7K<br>-----------|---------------|---------------<br>7,036      | 2,157.2       | 100% (baseline)<br>35,180     | 2,044.2       | 94.8%<br>70,145     | 1,695.2       | 78.6%<br>174,725    | 1,182.0       | 54.8%<br>349,651    |   748.8       | 34.7%<br>524,382    | 1,041.1       | 48.3%<br>699,676    |   875.2       | 40.6%</pre><p><strong>Observations:</strong></p><ul><li>Prompt processing speed degrades gracefully at 700K tokens, the model still processes at <strong>875 tok/s</strong>, which is ~40% of the small-context speed.</li><li>The 524K test was faster than 350K (1,041 vs 749 tok/s), likely due to batch processing optimizations and caching effects from the prior run.</li><li>Even at the maximum tested context, the entire 700K prompt is processed in under <strong>14 minutes</strong>.</li></ul><h3>Generation Speed vs Context Size</h3><pre>Tokens     | Gen Speed (tok/s) | Relative to 7K<br>-----------|-------------------|---------------<br>7,036      | 73.3              | 100% (baseline)<br>35,180     | 63.3              | 86.4%<br>70,145     | 49.9              | 68.1%<br>174,725    | 31.7              | 43.2%<br>349,651    | 18.8              | 25.6%<br>524,382    | 21.6              | 29.5%<br>699,676    | 17.3              | 23.6%</pre><p><strong>Observations:</strong></p><ul><li>Generation speed at 700K tokens is <strong>17.3 tok/s</strong> — still readable in real-time (average human reading speed is ~4 words/s ≈ ~5 tok/s).</li><li>The speed degradation is expected: each generated token must attend to the full KV cache.</li><li>The Mamba-2 hybrid design significantly mitigates this — only 6 of 52 layers need attention over the full context. The remaining 46 layers use constant-time recurrent computation.</li></ul><h3>Time to First Token (Prompt Processing)</h3><p>Context SizeTotal Processing Time7K tokens6 seconds70K tokens44 seconds175K tokens2.6 minutes350K tokens7.9 minutes524K tokens8.5 minutes700K tokens13.5 minutes</p><p>For interactive use cases, contexts up to ~100K tokens provide sub-minute prompt processing. For batch processing or document analysis, even 700K context is practical at under 14 minutes.</p><h3>Ollama Baseline Benchmarks</h3><p>Before switching to llama.cpp for 1M context, we ran comprehensive benchmarks using Ollama at standard context sizes.</p><h3>Speed Benchmarks (Ollama, default context)</h3><p>MetricValueGeneration speed~75 tok/sPrompt evaluation speed350–635 tok/s</p><h3>Intelligence Benchmarks (Ollama)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*K3B8JOKoBlmbowWC8Iw6Ig.png" /></figure><h3>Ollama Context Window Limits</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*VHp_j_D2po3CRgpPSzg7Jw.png" /></figure><h3>Ollama Needle-in-a-Haystack (up to 768K)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*OB4gbdRoEabsOoUj_X36jw.png" /></figure><p>Ollama maxes out at <strong>768K context</strong> because it uses FP16 KV cache with no quantization option for Mamba hybrid architectures. The FP16 KV cache at 1M requires 96 GB, which combined with the 24 GB model exceeds the 128 GB unified memory.</p><h3>Inference Engine Comparison</h3><p>We tested three inference engines during this benchmark. Here’s what worked and what didn’t:</p><h3>Ollama</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KMtfxhOitjBsgdWmBBoS8g.png" /></figure><h3>vLLM (via Docker)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fCsHTwsvcbOL3evBBv9s6Q.png" /></figure><h3>llama.cpp (compiled from source)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NUrPrHlh83uAQJ8Rw7XyOg.png" /></figure><h3>Failed Approaches &amp; Lessons Learned</h3><h3>Attempt 1: Ollama with 1M Context</h3><p><strong>What happened:</strong> Ollama crashed with OOM when setting num_ctx to 1048576. <strong>Root cause:</strong> FP16 KV cache for 1M context = 96 GB. Model (24 GB) + KV cache (96 GB) = 120 GB exceeds available memory after OS and other processes. <strong>Lesson:</strong> Ollama doesn&#39;t support KV cache quantization for Mamba hybrid architectures.</p><h3>Attempt 2: Ollama with Q8_0/Q4_0 KV Cache</h3><p><strong>What happened:</strong> Crash on model load. <strong>Root cause:</strong> Mamba hybrid architecture not supported for KV cache quantization in Ollama. <strong>Lesson:</strong> Ollama’s KV cache quantization is architecture-dependent.</p><h3>Attempt 3: vLLM with Various Quantizations</h3><p><strong>What happened:</strong> Three different model formats (NVFP4, FP8, BF16) all failed with different errors. <strong>Root cause:</strong> vLLM’s Mamba hybrid + MoE support on Blackwell (sm_121) is incomplete. <strong>Lesson:</strong> Cutting-edge hardware (Blackwell) + cutting-edge architecture (Mamba-2 + MoE) = limited framework support.</p><h3>Attempt 4: llama.cpp with CUDA Architecture 120</h3><p><strong>What happened:</strong> Model loaded, KV cache allocated successfully (1.7 GB), but crashed during CUDA kernel execution. <strong>Root cause:</strong> Compiled with sm_120 but DGX Spark GPU requires sm_121. <strong>Error:</strong> CUDA error: no kernel image is available for execution on the device <strong>Lesson:</strong> Always verify exact compute capability. DGX Spark Blackwell is sm_121, not sm_120. <strong>Silver lining:</strong> This attempt proved that Q4_0 KV cache allocation works at only 1.7 GB for 1M context.</p><h3>Attempt 5: llama.cpp with CUDA Architecture 121 ✓</h3><p><strong>What happened:</strong> Everything worked perfectly. <strong>Key insight:</strong> The only change from Attempt 4 was CMAKE_CUDA_ARCHITECTURES=121 instead of 120.</p><h3>Key Takeaways</h3><ol><li><strong>DGX Spark can run 1M context window.</strong> Despite being a compact desktop device, the DGX Spark’s 128 GB unified memory and Blackwell GPU make ultra-long context inference feasible.</li><li><strong>KV cache quantization is the key enabler.</strong> Q4_0 quantization reduces the KV cache from 96 GB to 1.7 GB — a 56x reduction — without any measurable loss in retrieval accuracy.</li><li><strong>Mamba-2 hybrid architecture is purpose-built for long context.</strong> With only 6 of 52 layers needing KV cache, the memory footprint is inherently smaller than pure Transformer models.</li><li><strong>MoE keeps compute efficient.</strong> Only 3.5B of 30B parameters are active per token, making the model fast despite its large total parameter count.</li><li><strong>Needle retrieval is perfect at all tested sizes.</strong> The model correctly found the hidden information in 100% of tests, from 7K to 700K tokens.</li><li><strong>Practical speeds at long context.</strong> At 700K tokens, the model still generates at 17 tok/s (readable in real-time) and processes prompts at 875 tok/s.</li><li><strong>The right tool matters.</strong> Ollama maxes out at 768K, vLLM doesn’t support this model on Blackwell, but llama.cpp with the right compilation flags achieves 1M context.</li></ol><h3>Technical Insights</h3><ul><li><strong>Unified memory is underrated.</strong> The DGX Spark’s 128 GB unified memory eliminates the GPU VRAM bottleneck that limits other setups.</li><li><strong>4-bit KV cache has no measurable quality impact</strong> for this retrieval task. The model still produces perfectly accurate and well-formatted responses.</li><li><strong>Compilation flags matter.</strong> A single digit difference (120 vs 121) in the CUDA architecture flag was the difference between &quot;no kernel image available&quot; and perfect operation.</li><li><strong>The Mamba-2 recurrent state is negligible</strong> — 190 MB regardless of context length, compared to the attention KV cache which scales linearly.</li></ul><h3>Reproduction Guide</h3><h3>Prerequisites</h3><ul><li>NVIDIA DGX Spark (or any system with 128 GB+ unified/GPU memory)</li><li>CUDA 13.0+ with Blackwell support</li><li>Nemotron 3 Nano 30B model in GGUF format (Q4_K_M, ~23 GB)</li></ul><h3>Step 1: Get the Model</h3><pre># Via Ollama (easiest way to download)<br>ollama pull nemotron:30b<br># Model is stored at: /usr/share/ollama/.ollama/models/blobs/</pre><h3>Step 2: Build llama.cpp</h3><pre>git clone https://github.com/ggml-org/llama.cpp.git<br>cd llama.cpp</pre><pre># CRITICAL: Use the correct CUDA architecture for your GPU<br># DGX Spark = sm_121, NOT sm_120<br>CUDACXX=/usr/local/cuda/bin/nvcc cmake -B build \<br>  -DGGML_CUDA=ON \<br>  -DCMAKE_CUDA_ARCHITECTURES=121</pre><pre>cmake --build build -j$(nproc)</pre><h3>Step 3: Launch the Server</h3><pre>./build/bin/llama-server \<br>  -m /path/to/nemotron-q4_k_m.gguf \<br>  -c 1048576 \<br>  -ngl 999 \<br>  -np 1 \<br>  -ctk q4_0 \<br>  -ctv q4_0 \<br>  --host 0.0.0.0 \<br>  --port 8080</pre><h3>Step 4: Test</h3><pre>curl http://localhost:8080/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;nemotron&quot;,<br>    &quot;messages&quot;: [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Hello, what model are you?&quot;}],<br>    &quot;max_tokens&quot;: 100<br>  }&#39;</pre><h3>Key Parameters Explained</h3><p>FlagPurpose-c 1048576Set context window to 1M tokens-ngl 999Offload all layers to GPU-np 1Single processing slot (maximizes memory for one long context)-ctk q4_0Quantize KV cache keys to 4-bit-ctv q4_0Quantize KV cache values to 4-bit</p><h3>Raw Server Logs</h3><h3>Memory Allocation at Startup</h3><pre>llama_kv_cache: size = 1728.00 MiB (1048576 cells, 6 layers, 4/1 seqs)<br>  K (q4_0):  864.00 MiB<br>  V (q4_0):  864.00 MiB</pre><pre>llama_memory_recurrent: size = 190.47 MiB (4 cells, 52 layers, 4 seqs)<br>  R (f32):   6.47 MiB<br>  S (f32): 184.00 MiB</pre><pre>CUDA0 compute buffer size = 3090.79 MiB<br>CUDA_Host compute buffer size = 2058.51 MiB</pre><h3>700K Token Test Timing (from server log)</h3><pre>prompt eval time = 799441.50 ms / 699676 tokens (1.14 ms per token, 875.21 tokens per second)</pre><p><em>Benchmark conducted on March 15, 2026, on NVIDIA DGX Spark with llama.cpp (commit d23355afc).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57b4d0809991" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running 30B LLMs on My Desk for $0/month: NVIDIA DGX Spark Benchmark Report]]></title>
            <link>https://medium.com/@kocyigityasar/running-30b-llms-on-my-desk-for-0-month-nvidia-dgx-spark-benchmark-report-fdaccae13c98?source=rss-1ab647a5c7d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdaccae13c98</guid>
            <dc:creator><![CDATA[Yasar Kocyigit]]></dc:creator>
            <pubDate>Thu, 05 Mar 2026 18:23:23 GMT</pubDate>
            <atom:updated>2026-03-05T18:23:23.268Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Qwen3.5 vs. Nemotron-3-Nano vs. QwQreal benchmarks, zero cloud costs, one clear winner.</strong></p><p>The 30-billion parameter range is the sweet spot for local LLM deployment. These models are large enough to produce genuinely useful output for coding, reasoning, and analysis tasks, yet small enough to run comfortably on a single GPU without cloud API bills. But which one should you actually use?</p><p>I took three of the most promising 30B-class models Alibaba’s Qwen3.5, NVIDIA’s Nemotron-3-Nano, and Alibaba’s QwQ and ran them through six standardized tests on my NVIDIA GB10 Spark DGX desktop. The results were definitive, and one model had a critical flaw that made it completely unusable.</p><h3>Test Environment</h3><ul><li><strong>Hardware:</strong> NVIDIA GB10 (Spark DGX)</li><li><strong>Runtime:</strong> Ollama (local installation)</li><li><strong>API:</strong> Ollama Chat API (/api/chat) with think: false</li><li><strong>Cost:</strong> $0 per query</li></ul><p>All three models were quantized to Q4_K_M for a fair comparison. Tests covered simple code generation (FizzBuzz), complex code generation (thread-safe LRU cache with TTL), bug fixing (binary search), SQL query writing, conceptual explanation (async vs. threading), and math reasoning (sum of primes).</p><h3>The Contenders</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2kKabd_DCRUz_qOjLvSEoA.png" /></figure><p>On paper, all three look competitive. Similar parameter counts, similar resource requirements, all free to use. The standout spec is Nemotron’s one-million-token context window 25x larger than either competitor. But specs only tell half the story.</p><h3>Speed: Nemotron Leads, QwQ Falls Off a Cliff</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l0XT-6mBop411Cu8EftbDQ.png" /></figure><p>Nemotron averaged 71.7 tokens per second, roughly 25% faster than Qwen3.5 at 57.3 tok/s. Both are perfectly usable for interactive development work fast enough to feel responsive without noticeable lag.</p><p>QwQ was a different story entirely. At 10.2 tok/s, it was 7x slower than Nemotron and painfully sluggish in practice. A complex coding task that Nemotron completed in 7 seconds took QwQ over 200 seconds and still produced no usable output.</p><p>Nemotron’s speed advantage comes from its hybrid architecture. By combining Mamba-2 layers for efficient sequence modeling with MoE layers for sparse expert routing, fewer parameters activate per token. The result is faster inference without sacrificing capacity.</p><p>After three tests confirmed QwQ’s consistently poor performance, I excluded it from the remaining benchmarks.</p><h3>Quality: Both Survivors Excel, But Nemotron Has an Edge</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lvz6LcrX6Ru7Da2KoJeKcA.png" /></figure><p>Qwen3.5 and Nemotron both passed all six tests. QwQ passed zero every test returned empty output as its token budget was consumed by invisible internal reasoning.</p><p>What separated the two working models was sophistication. Consider the thread-safe LRU cache test, which asked for a production-grade implementation with TTL support and thread safety.</p><p><strong>Qwen3.5</strong> produced a clean, correct implementation using threading.Lock(), docstrings, and input validation. Professional-grade code with good documentation habits.</p><p><strong>Nemotron</strong> went further. It used __slots__ for memory optimization. It chose time.monotonic() over time.time() - the correct choice for measuring elapsed time, since time.time() can jump during clock adjustments. It used threading.RLock() instead of Lock(), which is safer for nested calls. It added type hints throughout. It even separated concerns with a dedicated TTLCacheItem dataclass.</p><p>The same pattern repeated across other tests. For SQL queries, both models wrote correct queries with date-range best practices. But Nemotron added a practical variations section covering cancelled order filtering, fiscal year boundaries, and alternative date formats the kind of context a senior engineer would include in a code review.</p><p>For bug fixing, Qwen3.5 identified three bugs in a broken binary search. Nemotron found four, including a subtle IndexError from float index values, with deeper explanations of why each bug breaks the algorithm.</p><p>Neither model struggled with the simpler tests. For the FizzBuzz and explanation tasks, both delivered concise, accurate results. The differences only emerged under complexity and that’s exactly where it matters.</p><h3>The QwQ Problem: When Intelligence Can’t Get Out</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4em3pKphhgu__PjgjjY7Gg.png" /></figure><p>QwQ:32B has a fundamental problem on Ollama that makes it unusable for real work. The model was designed for deep chain-of-thought reasoning, and it always engages that reasoning mode regardless of what you tell it.</p><p>I tried every available control mechanism. Setting think: false in the API had no effect. Adding /no_think to the prompt had no effect. Increasing the token budget from 512 to 2,048 just gave the model more room to think silently - still zero visible output.</p><p>The root cause is architectural. QwQ uses a qwen2-based architecture that lacks the thinking capability flag Ollama needs to control thinking mode. Without that flag, every API parameter designed to suppress thinking is simply ignored.</p><p>Here’s the frustrating part: when you examine QwQ’s leaked thinking text, you can see it reasoning correctly. During the bug fix test, its internal chain-of-thought was identifying the same bugs that Nemotron and Qwen3.5 found. The intelligence is genuinely there. It just can’t produce output.</p><p>QwQ may work better through alternative inference engines like vLLM or TGI that handle thinking tokens differently. But on Ollama today the most popular local inference runtime it’s a non-starter.</p><h3>Overall Scores: A Clear Hierarchy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1yTRzIfWDHGnvq8YEpJ9cA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ehfccVXss7sy4dYREi0F9A.png" /></figure><p>*QwQ’s reasoning score reflects its internal thinking quality (visible in leaked text), not usable output.</p><p>Nemotron led in six of seven categories. Qwen3.5 only edged it out in Resource Efficiency, thanks to a slightly smaller disk footprint. QwQ’s sole strong score 8.0 in Reasoning is an asterisk at best, since that reasoning never reaches the user.</p><h3>Resource Comparison: Remarkably Similar, With One Key Exception</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IxMNb3wA_w2eHxgzn8YnEg.png" /></figure><p>All three models sit between 19 and 24 GB for disk and VRAM, making them all viable on the same hardware. Model load times ranged from 9.5 seconds (Qwen3.5) to 21.7 seconds (QwQ).</p><p>The real differentiator is context window. Nemotron’s 1,048,576 tokens (just over 1M) dwarfs the 40,960 offered by both competitors. That’s not an incremental advantage it’s a categorical one. With a 1M context window, you can feed Nemotron an entire repository, a full codebase, or a long technical document and get coherent analysis back. The other two models would need that content chunked and summarized first.</p><h3>What I Learned</h3><p><strong>Thinking tokens are the hidden failure mode of local LLM deployment.</strong> QwQ looked excellent on paper. Strong architecture, good reputation, competitive parameter count. But an infrastructure-level incompatibility with Ollama made it the worst performer by a massive margin. If you’re evaluating models for local use, test thinking token behavior before anything else.</p><p><strong>Hybrid architectures work.</strong> Nemotron’s Mamba-2 + MoE design delivered both the fastest inference and the highest quality output. The conventional wisdom that MoE models sacrifice quality for speed didn’t hold here. NVIDIA’s architecture choices paid off across the board.</p><p><strong>Benchmarks on paper don’t predict benchmarks in practice.</strong> All three models had similar parameter counts, similar quantization, and similar resource requirements. The performance gap between the best (9.1/10) and worst (3.9/10) was enormous. You have to test on your actual inference stack.</p><p><strong>The 30B class is ready for serious work.</strong> Running at 57–72 tok/s with zero API costs, these models handle coding, debugging, SQL generation, and reasoning tasks at a level that’s genuinely useful for day-to-day development. Local inference isn’t a hobby project anymore.</p><h3>My Recommendation</h3><p>If you’re running Ollama on local hardware, install Nemotron-3-Nano. It’s the fastest, the most capable, and its 1M context window opens up use cases the other models simply can’t handle. Qwen3.5 is a solid backupreliable and well-tested but Nemotron beats it in almost every dimension.</p><p>Skip QwQ on Ollama entirely until the thinking token control issue is resolved.</p><pre>ollama pull nemotron-3-nano:30b</pre><p>That’s it. You’re running a 9.1/10 coding assistant for free.</p><p><em>All benchmarks were run on an NVIDIA GB10 (Spark DGX) with Ollama, at zero cost. All models are open-source and free to use.</em></p><p><em>Yasar Kocyigit</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdaccae13c98" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Qwen3-Coder-Next:Q8_0 vs Qwen3:30B-A3B-Q4_K_M Detailed Comparison Report]]></title>
            <link>https://medium.com/@kocyigityasar/qwen3-coder-next-q8-0-vs-qwen3-30b-a3b-q4-k-m-detailed-comparison-report-b40e482d97ce?source=rss-1ab647a5c7d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b40e482d97ce</guid>
            <dc:creator><![CDATA[Yasar Kocyigit]]></dc:creator>
            <pubDate>Thu, 05 Mar 2026 16:28:25 GMT</pubDate>
            <atom:updated>2026-03-05T16:28:25.088Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Date:</strong> March 5, 2026 <strong>Platform:</strong> NVIDIA GB10 (Spark DGX) <strong>Ollama Version:</strong> Local installation <strong>Test Environment:</strong> Remote testing via SSH (Ollama API + CLI)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e8Z7fQ2ol2j8T7PkRZHJYQ.png" /></figure><h3>1. Model Technical Specifications</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VO3qJSEQB5roRIkhu2W6LA.png" /></figure><h3>Key Architectural Differences:</h3><ul><li><strong>qwen3-coder-next</strong>: Dense model — all 79.7B parameters are active during inference. Specifically trained and optimized for code generation.</li><li><strong>qwen3:30b-a3b</strong>: MoE (Mixture of Experts) model — 30.5B total parameters but only a subset is active per inference pass. This makes it significantly faster relative to its size.</li></ul><h3>2. Speed Comparison (Tokens/Second)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XoRAgAMn6vppJCPX2Sja8w.png" /></figure><h3>3. Quality Comparison (Detailed Test Results)</h3><h3>Test 1: FizzBuzz (Simple Code Generation)</h3><p><strong>qwen3-coder-next:q8_0</strong> — PASS</p><pre>def fizzbuzz(n):<br>    for i in range(1, n + 1):<br>        if i % 15 == 0:<br>            print(&quot;FizzBuzz&quot;)<br>        elif i % 3 == 0:<br>            print(&quot;Fizz&quot;)<br>        elif i % 5 == 0:<br>            print(&quot;Buzz&quot;)<br>        else:<br>            print(i)</pre><ul><li>Clean, correct, optimized (checks 15 first)</li><li>No explanation (as instructed)</li></ul><p><strong>qwen3:30b-a3b-q4_K_M </strong>PASS (with thinking leakage)</p><pre>def fizzbuzz(n):<br>    for i in range(1, n+1):<br>        if i % 3 == 0 and i % 5 == 0:<br>            print(&quot;FizzBuzz&quot;)<br>        elif i % 3 == 0:<br>            print(&quot;Fizz&quot;)<br>        elif i % 5 == 0:<br>            print(&quot;Buzz&quot;)<br>        else:<br>            print(i)</pre><ul><li>Correct but leaked thinking text despite /no_think flag</li><li>Added explanation despite “only code” instruction</li><li>Code is correct but i % 3 == 0 and i % 5 == 0 is less optimal than i % 15 == 0</li></ul><p><strong>Winner: qwen3-coder-next</strong> (cleaner output, better instruction following)</p><h3>Test 2: Thread-Safe LRU Cache with TTL (Complex Code Generation)</h3><p><strong>qwen3-coder-next:q8_0</strong> EXCELLENT</p><pre>import time<br>import threading<br>from collections import OrderedDict</pre><pre>class ThreadSafeLRUCache:<br>    def __init__(self, capacity: int, default_ttl: float = None):<br>        self.capacity = capacity<br>        self.default_ttl = default_ttl<br>        self.cache = OrderedDict()<br>        self.lock = threading.RLock()</pre><pre>    def _is_expired(self, key):<br>        if key not in self.cache:<br>            return True<br>        entry = self.cache[key]<br>        if entry[1] is not None and time.time() &gt; entry[1]:<br>            return True<br>        return False</pre><pre>    def get(self, key):<br>        with self.lock:<br>            if self._is_expired(key):<br>                self._remove(key)<br>                return None<br>            self.cache.move_to_end(key)<br>            return self.cache[key][0]</pre><pre>    def put(self, key, value, ttl=None):<br>        ttl = ttl if ttl is not None else self.default_ttl<br>        expire_time = time.time() + ttl if ttl is not None else None<br>        with self.lock:<br>            if key in self.cache:<br>                self._remove(key)<br>            elif len(self.cache) &gt;= self.capacity:<br>                self._popitem()<br>            self.cache[key] = (value, expire_time)<br>            self.cache.move_to_end(key)</pre><pre>    def _remove(self, key):<br>        if key in self.cache:<br>            del self.cache[key]</pre><pre>    def _popitem(self):<br>        if self.cache:<br>            self.cache.popitem(last=False)</pre><pre>    def cleanup(self):<br>        with self.lock:<br>            expired_keys = [key for key in self.cache if self._is_expired(key)]<br>            for key in expired_keys:<br>                del self.cache[key]</pre><pre>    def __len__(self):<br>        with self.lock:<br>            return len([k for k in self.cache if not self._is_expired(k)])</pre><pre>    def clear(self):<br>        with self.lock:<br>            self.cache.clear()</pre><ul><li>Fully featured: get, put, cleanup, clear, <strong>len</strong></li><li>Uses RLock (reentrant lock correct choice for nested calls)</li><li>TTL support with default_ttl and per-key TTL override</li><li>Completed in 11.24 seconds</li></ul><p><strong>qwen3:30b-a3b-q4_K_M</strong> FAILED</p><ul><li>512 tokens: EMPTY RESPONSE (all tokens consumed by thinking)</li><li>1024 tokens: EMPTY RESPONSE</li><li>4096 tokens: EMPTY RESPONSE (spent 64.5 seconds producing nothing!)</li><li>Even with /no_think: EMPTY RESPONSE</li></ul><p><strong>Winner: qwen3-coder-next</strong> (massive gap one produced excellent code, the other produced nothing)</p><h3>Test 3: Bug Fix (Binary Search)</h3><p><strong>qwen3-coder-next:q8_0</strong> EXCELLENT Correctly identified all 3 bugs:</p><ol><li>/ should be // (integer division required)</li><li>high = len(arr) can cause IndexError when mid equals len(arr)</li><li>low = mid causes infinite loop, should be low = mid + 1</li></ol><p>Provided two corrected versions (inclusive range and half-open interval approaches).</p><p><strong>qwen3:30b-a3b-q4_K_M</strong> EMPTY RESPONSE</p><ul><li>Even with /no_think, produced no output</li><li>Entire 512-token budget consumed by thinking</li></ul><p><strong>Winner: qwen3-coder-next</strong> (comprehensive analysis vs no response at all)</p><h3>Test 4: SQL Query</h3><p><strong>qwen3-coder-next:q8_0</strong> — EXCELLENT</p><pre>SELECT<br>    c.name,<br>    c.email,<br>    SUM(o.amount) AS total_amount<br>FROM customers c<br>INNER JOIN orders o ON c.id = o.customer_id<br>WHERE o.created_at &gt;= &#39;2024-01-01&#39;<br>    AND o.created_at &lt; &#39;2025-01-01&#39;<br>GROUP BY c.id, c.name, c.email<br>ORDER BY total_amount DESC<br>LIMIT 5;</pre><ul><li>Correct, optimized, and well-explained</li><li>Used date range filter best practice (&lt; &#39;2025-01-01&#39; instead of YEAR())</li><li>Explained index usage implications</li><li>Provided alternative YEAR() syntax with performance caveat</li></ul><p><strong>qwen3:30b-a3b-q4_K_M</strong> EMPTY RESPONSE</p><p><strong>Winner: qwen3-coder-next</strong></p><h3>Test 5: Async/Await vs Threading Explanation</h3><p><strong>qwen3-coder-next:q8_0</strong> VERY GOOD</p><ul><li>Concise 5-sentence explanation</li><li>Distinguished cooperative vs preemptive multitasking</li><li>Covered I/O-bound vs CPU-bound use cases</li><li>Indirect GIL reference</li><li>177 tokens, 4.71s</li></ul><p><strong>qwen3:30b-a3b-q4_K_M</strong> GOOD (with /no_think)</p><ul><li>Concise 5-sentence explanation</li><li>Direct GIL reference</li><li>Shorter but accurate</li><li>394 tokens (includes thinking overhead), 4.88s</li></ul><p><strong>Winner: Tie</strong> (both provided correct and adequate explanations)</p><h3>Test 6: Math Reasoning (Sum of Primes 1–50)</h3><p>Correct answer: 2+3+5+7+11+13+17+19+23+29+31+37+41+43+47 = <strong>328</strong></p><p><strong>qwen3-coder-next:q8_0 </strong>PARTIAL (token limit)</p><ul><li>Created detailed table checking each number for primality</li><li>Could not complete answer within 512-token limit (reached number 20)</li><li>Approach was correct</li></ul><p><strong>qwen3:30b-a3b-q4_K_M</strong> EMPTY RESPONSE</p><ul><li>With thinking mode: all tokens consumed by thinking</li><li>With /no_think: still empty response</li></ul><p><strong>Winner: qwen3-coder-next</strong> (at least showed correct working process)</p><h3>4. Critical Issue: qwen3:30b-a3b Thinking Token Problem</h3><p>The most significant finding is that qwen3:30b’s <strong>thinking mode cannot be reliably controlled via the API</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gnfiKGgw9JtGEOEzy9y1eA.png" /></figure><p><strong>Conclusion:</strong> Thinking tokens are deducted from the total token budget, and for complex questions the model spends its entire budget “thinking” without producing any visible output. This severely limits the model’s practical usability via the Ollama API.</p><h3>5. Resource Usage</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eT4AfaNmGzGiBlGGG-1tCw.png" /></figure><h3>6. Use Case Recommendations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f4onZQ6g29A9FKm1Db34Gg.png" /></figure><h3>7. Overall Evaluation and Conclusion</h3><h3>Score Card (out of 10)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lXjh36P7Jb_jbXNOmCQeBg.png" /></figure><p>(*) qwen3:30b’s low quality score stems from the <strong>thinking token problem</strong>, not necessarily the model’s inherent quality. Scores may change if API-level thinking token control becomes available.</p><h3>Final Verdict</h3><p><strong>qwen3-coder-next:q8_0 is the clear winner.</strong></p><ul><li>Despite qwen3:30b being <strong>2x faster</strong> in raw token generation, its <strong>inability to reliably produce responses</strong> is a critical flaw.</li><li>qwen3-coder-next delivered consistent, high-quality, usable output in every single test.</li><li>qwen3:30b is only viable for simple, short-answer scenarios when used with /no_think.</li><li>For real-world tasks like complex code generation, bug analysis, and SQL <strong>qwen3-coder-next is unquestionably superior</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lk-YYruSZsF3MdKYK7E0nQ.png" /></figure><h3>Recommendations</h3><ol><li><strong>Primary development tool:</strong> Use qwen3-coder-next:q8_0</li><li><strong>Quick simple queries:</strong> qwen3:30b-a3b with /no_think can be used for speed</li><li><strong>qwen3:30b thinking issue:</strong> Re-evaluate when Ollama introduces num_think or thinking token controls in future versions</li><li><strong>Resource-constrained setup:</strong> qwen3:30b (18GB vs 84GB) is viable, but factor in the quality trade-off</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b40e482d97ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Built an AI-Powered SSIS to Databricks Migration Framework Using Claude’s Skill System]]></title>
            <link>https://medium.com/@kocyigityasar/how-i-built-an-ai-powered-ssis-to-databricks-migration-framework-using-claudes-skill-system-fb642e3049d3?source=rss-1ab647a5c7d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb642e3049d3</guid>
            <dc:creator><![CDATA[Yasar Kocyigit]]></dc:creator>
            <pubDate>Sun, 01 Mar 2026 01:42:06 GMT</pubDate>
            <atom:updated>2026-03-01T01:42:06.560Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LS9fFi9YGQUr76rN6bdpAg.png" /></figure><p><em>An open-source toolkit that turns SSIS packages into Databricks notebooks with Medallion architecture, Unity Catalog, and a Claude AI skill that does the heavy lifting.</em></p><h3>The Trigger</h3><p>Anthropic recently published <a href="https://resources.anthropic.com/hubfs/The-Complete-Guide-to-Building-Skill-for-Claude.pdf"><strong>“The Complete Guide to Building Skills for Claude”</strong></a>a 33-page guide explaining how to teach Claude repeatable workflows through structured instruction files called “skills.”</p><p>The concept is simple: instead of explaining your process to an AI every time, you package your expertise into a folder with a SKILL.md file, and Claude learns to execute that workflow consistently.</p><p>I read the guide and thought: <em>this is exactly what SSIS migration needs.</em></p><p>If you’ve ever migrated SSIS packages to a modern platform, you know the pain. The logic isn’t in readable code it’s buried in XML (DTSX files), scattered across data flow transforms, embedded SQL, connection managers, and event handlers. Every package is different, but the translation patterns are the same.</p><p>So I built <strong>migration-spec-kit</strong>: an open-source framework that structures the entire SSIS-to-Databricks migration lifecycle, complete with a Claude skill that can parse DTSX files and generate Databricks notebooks automatically.</p><p><strong>GitHub:</strong> <a href="https://github.com/yasarkocyigit/migration-spec-kit">https://github.com/yasarkocyigit/migration-spec-kit</a></p><h3>What’s Inside</h3><p>The repo follows a phased approach — from source inventory to validation with every artifact organized in numbered folders:</p><p><strong>01-source/</strong> Your original SSIS packages and stored procedures go here. Read-only. Never modified.</p><p><strong>02-analysis/</strong> AI-generated (or manual) analysis output: what each package does, inter-package dependencies, and a complexity score (Simple/Medium/High) with effort estimates.</p><p><strong>03-mapping/ </strong>This is the reference library. Three detailed mapping documents cover:</p><ul><li><strong>Component mapping</strong> 30+ SSIS tasks and transforms mapped to their Databricks equivalents. Execute SQL Task becomes a notebook cell. Data Flow Task becomes a PySpark DataFrame pipeline. Conditional Split becomes .filter(). Lookup becomes .join() with broadcast hints.</li><li><strong>Connection mapping</strong> How to translate OLE DB, Flat File, and ADO.NET connection managers into JDBC connections, dbutils.secrets, and Unity Catalog external connections. With working code examples.</li><li><strong>Data type mapping</strong> Every SQL Server data type mapped to its Spark/Delta equivalent. Including edge cases like money → DECIMAL(19,4), datetimeoffset → TIMESTAMP (normalize to UTC), and uniqueidentifier → STRING.</li></ul><p><strong>04-target/</strong> The generated Databricks code. Organized by Medallion layer:</p><ul><li><strong>Bronze notebooks</strong> Raw ingestion via JDBC with incremental watermarks</li><li><strong>Silver notebooks</strong> Deduplication, cleansing, data quality flags, active/inactive splits</li><li><strong>Gold notebooks</strong> SCD Type 2 dimension merges, fact table loads with surrogate key resolution, late-arriving dimension handling</li></ul><p>Plus Unity Catalog DDLs for all three layers, a Databricks Workflow JSON (replacing SQL Server Agent), and shared utility functions.</p><p><strong>05-validation/</strong> Post-migration validation queries: row counts, data quality checks (nulls, duplicates, referential integrity, SCD2 history gaps), and a reconciliation report template with sign-off fields.</p><p><strong>06-docs/ </strong>Migration runbook (step-by-step), rollback plan (with Delta time travel), and an architecture decision log.</p><h3>The Claude Skill</h3><p>The skills/ssis-migration/ folder contains a purpose-built Claude skill following Anthropic&#39;s specification. It includes:</p><ul><li><strong>SKILL.md</strong> Instructions that teach Claude how to parse DTSX XML, extract control flow and data flow components, and generate Bronze/Silver/Gold notebooks</li><li><strong>references/</strong>A DTSX structure guide (XML element mappings, namespace handling, component class IDs) and a Databricks patterns guide (code templates for every migration pattern)</li><li><strong>scripts/parse_dtsx.py</strong> A standalone Python parser that extracts structured metadata from DTSX files</li></ul><p>Upload the skill to Claude (Settings → Skills) or place it in your Claude Code skills directory. Then simply ask:</p><p><em>“Analyze my SSIS package ETL_Load_Customers.dtsx and convert it to a Databricks notebook using Medallion architecture with Unity Catalog.”</em></p><p>The skill handles the rest: reading the DTSX XML, identifying tasks and transforms, resolving execution order from precedence constraints, and producing notebooks with proper audit columns, error handling, and dbutils.widgets for parameters.</p><h3>The Example Migration</h3><p>The repo ships with a complete working example that demonstrates a realistic scenario:</p><p><strong>Source side:</strong></p><ul><li>ETL_Load_Customers.dtsxIncremental customer load with Derived Column, Conditional Split, and Lookup transforms</li><li>ETL_Load_Orders.dtsx Multi-output data flow with aggregation, error output to flat file, and email notification on failure</li><li>sp_MergeCustomerDim.sqlSCD Type 2 merge using HASHBYTES for change detection</li><li>sp_MergeOrderFact.sql Fact merge with surrogate key resolution, late-arriving dimensions, and currency conversion</li></ul><p><strong>Target side:</strong></p><ul><li>Bronze notebook: JDBC incremental read → Delta append with audit columns</li><li>Silver notebook: Dedup via ROW_NUMBER window → MERGE into Silver table → inactive customer logging</li><li>Gold notebook (customers): SHA-256 hash comparison → expire changed records → insert new versions</li><li>Gold notebook (orders): Surrogate key joins → currency conversion → Delta MERGE → region summary aggregation</li></ul><p>Every notebook includes comments referencing the original SSIS component it replaces, so you can trace exactly where each piece of logic came from.</p><h3>How to Use It</h3><p>The simplest path:</p><ol><li>Clone the repo</li><li>Drop your .dtsx files into 01-source/dtsx/</li><li>Add your stored procedures to 01-source/stored-procedures/</li><li>Use Claude with the included skillor the parser script to analyze and generate</li></ol><p>The mapping docs and patterns are designed as a starting point. Modify everything to match your architecture different layer names, different naming conventions, different cluster configurations. The framework is flexible.</p><p>For teams running enterprise migrations with dozens of packages, the phased structure keeps things organized. The complexity assessment helps prioritize which packages to migrate first. The validation framework ensures nothing gets lost in translation.</p><h3>What’s Next</h3><p>This is version 1. There are SSIS patterns not yet covered Script Tasks with C# code, custom components, package configurations via XML files, and multi-server deployments. I plan to expand the mapping docs and add more example packages over time.</p><p>If you try it out, I’d genuinely appreciate hearing about your experience. What worked well? What patterns are missing? What broke? Open an issue, submit a PR, or just drop a comment here.</p><p>The goal is to make SSIS migration less painful for everyone going through it. And if an AI skill can handle the repetitive translation work, we can focus on the parts that actually need human judgment architecture decisions, data quality rules, and business logic validation.</p><p><strong>Repository:</strong> <a href="https://github.com/yasarkocyigit/migration-spec-kit">https://github.com/yasarkocyigit/migration-spec-kit</a></p><p><em>#DataEngineering #SSIS #Databricks #ETL #Migration #UnityCatalog #MedallionArchitecture #Claude #AI #OpenSource #PySpark #DeltaLake #SQLServer</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb642e3049d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Tested Agentic AI for Data Pipelines on Three Different Platforms:Here’s What I Learned]]></title>
            <link>https://medium.com/@kocyigityasar/i-tested-agentic-ai-for-data-pipelines-on-three-different-platforms-heres-what-i-learned-fca6d4035984?source=rss-1ab647a5c7d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fca6d4035984</guid>
            <dc:creator><![CDATA[Yasar Kocyigit]]></dc:creator>
            <pubDate>Sun, 15 Feb 2026 15:59:51 GMT</pubDate>
            <atom:updated>2026-02-15T15:59:51.547Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A hands-on journey through a custom Agentic Control Plane on Microsoft Fabric, a from-scratch open-source lakehouse on a Mac Mini, and Databricks Genie</em></p><p>If you work in data engineering long enough, you start noticing a pattern: business users want a simple change add a column, tweak a filter, onboard a new source and it turns into a two-week sprint because everything requires an engineer to touch code, test it, deploy it, and hope nothing breaks downstream.</p><p>I’ve been thinking about this problem for a while. Not in a “let’s replace engineers with AI” kind of way, but more in a “what if the repetitive, config-level work could be handled by an AI agent that actually understands the pipeline?” kind of way.</p><p>So I decided to test that idea. Not once, but on three very different platforms, to see where this approach actually works and where it falls apart. This article is the culmination of that journey what I built, what broke, and what I’d do differently.</p><p><em>This is Part 4 of my series on Agentic AI for Data Engineering. If you want the backstory:</em></p><ul><li><em>Part 1: </em><a href="https://medium.com/@kocyigityasar/reducing-engineering-dependency-in-data-pipelines-with-agentic-ai-89b1c77f6426"><em>Reducing Engineering Dependency in Data Pipelines with Agentic AI</em></a></li><li><em>Part 3: </em><a href="https://medium.com/@kocyigityasar/from-metadata-driven-to-agentic-evolving-the-data-pipeline-architecture-part-3-cefb227ac51f"><em>From Metadata-Driven to Agentic: Evolving the Data Pipeline Architecture</em></a></li></ul><h3>The Core Idea</h3><p>The concept is straightforward. In most data engineering teams, there’s a bottleneck: the engineer. Not because they’re slow, but because everything routes through them. Schema changes, new table onboarding, data quality rule updates all of it requires someone who understands the pipeline internals.</p><p>What if an AI agent could sit between the business request and the pipeline configuration, understand intent, and make changes safely — with guardrails, validation, and human approval?</p><p>That’s what I set out to build and test across three very different environments.</p><h3>Attempt 1: Agentic Control Plane on Microsoft Fabric</h3><p>I started in the Microsoft ecosystem because that’s where a lot of my enterprise work lives. But I didn’t just wire up a chatbot to a database I built a full <strong>Agentic Control Plane</strong> for Microsoft Fabric with a proper reasoning architecture.</p><h3>What I Built</h3><p>The system follows a <strong>Recall → Plan → Gate → Execute → Observe</strong> loop. It’s a Next.js application written in TypeScript that combines the reasoning power of LLMs (Claude / GPT-4o) with deterministic safety controls. The idea was to move beyond simple “chat-to-SQL” and create something that actually thinks before it acts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5O1CbI0G1fZ81IULzvfgsA.png" /></figure><h3>The Architecture in Detail</h3><p><strong>Memory Layer (pgvector):</strong> I found early on that without persistent memory, the AI restarts every conversation from zero. In data engineering, context is everything schemas, business rules, past errors. So I spun up PostgreSQL with pgvector and wrote a TypeScript adapter that converts interactions into embeddings. The agent can now do semantic searches like &quot;How did we fix the pipeline error last week?&quot; and retrieve relevant context before answering.</p><p><strong>Planner Node:</strong> I wanted to prevent the agent from rushing into execution or hallucinating a quick fix that breaks production. The Planner is a distinct logic layer that receives the user goal, generates a structured JSON plan (e.g., [{ step: &quot;check_schema&quot; }, { step: &quot;generate_query&quot; }]), and validates it against known constraints before passing it to execution.</p><p><strong>The Gate (Human-in-the-Loop):</strong> This is the piece I’m most proud of. A GateInterface intercepts every proposed action and classifies its risk level:</p><ul><li><strong>Auto-Approve:</strong> Read-only operations (SELECT, LIST, LOG) pass through automatically.</li><li><strong>Human-in-the-Loop:</strong> High-risk operations (run_pipeline, drop_table, delete) trigger a pause. The system generates a structured approval request and waits for explicit user confirmation in the UI before proceeding. This logic is hard-coded in gate.ts — the AI cannot bypass it.</li></ul><p><strong>Fabric MCP Tools (15+ integrations):</strong> The execution layer connects directly to the Microsoft Fabric API through the Model Context Protocol:</p><ul><li><strong>Pipelines</strong>: list, run, cancel, get_status, get_history</li><li><strong>Notebooks</strong>: Execute Spark notebooks with parameter injection</li><li><strong>Lakehouses</strong>: List tables, load data files (File → Table)</li><li><strong>Semantic Models</strong>: Trigger dataset refreshes</li><li><strong>Workspaces</strong>: List items, monitor capacity</li></ul><p><strong>Observability:</strong> Every request gets a unique TraceID (UUID) that tracks the full lifecycle: PLAN → APPROVAL_REQUEST → PENDING → SUCCESS/FAILURE. An immutable audit log records who did what and when. Unlike black-box agents, every decision is auditable.</p><h3>A Real Interaction Example</h3><p>Here’s how a typical exchange works:</p><blockquote><strong><em>User:</em></strong><em> “Delete the ‘Old_Logs’ table from metadata.”</em></blockquote><blockquote><strong><em>Agent (internally):</em></strong></blockquote><ul><li><strong>Recall</strong> — Searches memory for context about ‘Old_Logs’.</li><li><strong>Plan</strong> — Generates plan: [check_table_exists, assess_dependencies, delete_table].</li><li><strong>Gate</strong> — Detects HIGH RISK action (delete_table). Pauses execution.</li><li><strong>Response</strong> — “I have generated an approval request to delete table ‘Old_Logs’. Please approve to proceed.”</li><li><strong>User clicks Approve</strong> → Agent executes and logs success with TraceID.</li></ul><h3>What Was Hard</h3><p><em>Getting this running directly in Azure was more involved than I expected. </em>Fabric API authentication with Service Principals, MCP server deployment, networking between components, OAuth token managementeach piece made sense on its own, but wiring them all together in the cloud added friction at every step. I also tested Azure AI Foundry’s agent framework directly, deploying a SQL MCP Server on Azure Container Apps to connect agents to Azure SQL databases, and ran into similar complexity there.</p><p><em>So I made a practical decision: I moved the entire setup to my local machine. </em>Docker Compose, local PostgreSQL with pgvector, the Next.js app running on localhost same architecture, same code, but without the cloud networking and authentication overhead getting in the way. This let me iterate much faster and focus on the actual agent logic rather than debugging Azure connectivity issues.</p><p>It’s worth noting: none of this is a knock on the Microsoft ecosystem. The tools are capable and mature. But when you’re prototyping something new, every extra layer between you and the thing you’re building slows you down. Running locally removed that friction and let me validate the Recall-Plan-Gate-Execute-Observe pattern properly before thinking about cloud deployment.</p><h3>Verdict</h3><p>The Fabric Control Plane is the most architecturally complete thing I built. The Recall-Plan-Gate-Execute-Observe pattern is genuinely production-worthy, and the HITL safety gate gives me real confidence. But it’s a significant engineering effort this approach makes sense for teams with dedicated platform engineering support, not for quick prototyping.</p><blockquote><strong><em>GitHub:</em></strong><em> </em><a href="https://github.com/yasarkocyigit/agentic-ai-mcp"><em>github.com/yasarkocyigit/agentic-ai-mcp</em></a></blockquote><blockquote><strong><em>Medium (Series Part 3):</em></strong><em> </em><a href="https://medium.com/@kocyigityasar/from-metadata-driven-to-agentic-evolving-the-data-pipeline-architecture-part-3-cefb227ac51f"><em>From Metadata-Driven to Agentic: Evolving the Data Pipeline Architecture</em></a></blockquote><h3>Attempt 2: Full Open-Source Stack (Built from Scratch on a Mac Mini)</h3><p>After the Microsoft experience, I wanted to strip everything down. No vendor lock-in, no managed services. Just Docker containers, open-source tools, and configuration files all running locally on an M4 Mac Mini.</p><p>The motivation was simple: you can’t hire a Data Engineer even a digital one if you don’t have a data platform for them to work on. You can’t ask an agent to “optimize the pipeline” if there is no pipeline. So I built the office first, then invited the employee.</p><h3>What I Built</h3><p>A complete, production-grade data lakehouse implementing the Medallion Architecture (Bronze → Silver → Gold), fully driven by YAML configuration zero hardcoded SQL or transformation logic. Nine services, one docker compose up -d.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z_ZwK17wPzgA4z_r6aAWgQ.png" /></figure><h3>The Tech Stack</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LLjt_LG1unc1Ycpu9qancg.png" /></figure><h3>Why Apache Iceberg (Not Delta Lake)</h3><p>I started with Delta Lake but migrated to Iceberg. A few reasons:</p><ul><li><strong>Vendor Neutrality</strong>: Iceberg isn’t owned by a single platform. It works equally well with Spark, Trino, Flink, and Snowflake.</li><li><strong>Hidden Partitioning</strong>: You query by timestamp, Iceberg handles partition pruning transparently. No manual partition column management.</li><li><strong>Schema Evolution</strong>: Rename columns, change types — without rewriting petabytes.</li><li><strong>Time Travel</strong>: The agent can make mistakes. With Iceberg, we can ROLLBACK TO SNAPSHOT instantly. It gives the AI a safety net.</li></ul><h3>The Config-as-Code Approach</h3><p>This is where it gets interesting for the agentic AI use case. Every pipeline is defined in YAML — all transformations, data quality rules, connections, and SLAs:</p><pre># config/pipelines.yml<br>silver:<br>  - name: clean_orders<br>    source: &quot;s3a://bronze/tpch/orders&quot;<br>    primary_key: order_key<br>    scd_type: 1<br>    transformations:<br>      - { source: o_orderkey,    target: order_key,    type: rename }<br>      - { source: o_totalprice,  target: total_price,  type: cast, expression: double }<br>      - { source: o_orderstatus, target: order_status,  type: trim }<br>    data_quality:<br>      - { name: pk_not_null,    column: order_key,   type: not_null, severity: critical }<br>      - { name: price_positive, column: total_price, type: range, expression: &quot;0,&quot;, severity: warning }</pre><p>A shared metadata engine (_metadata.py) reads these configs at runtime, resolves transformations, applies data quality rules, and manages state. Adding a new table requires zero code changes — just a new YAML entry.</p><p>The Gold layer uses Spark 4.1+ <strong>Declarative Pipelines</strong> (SDP) with @dp.materialized_view decorators. Instead of writing <em>how</em>to process data (imperative PySpark), we define <em>what</em> we want (declarative). This is crucial for the agent — it&#39;s hard for an LLM to write perfect complex PySpark code, but it&#39;s easy for an LLM to generate a clean YAML config.</p><p>Data quality rules map to pipeline behavior based on severity: critical halts the pipeline, warning filters out bad rows, info just logs.</p><h3>The “Physical AI Employee” Concept</h3><p>With the platform running, I started designing the agent layer — what I’ve been calling “OpenClawd.” The idea is to give the agent specific <strong>MCP Tools</strong> that act as its hands:</p><ol><li><strong>run_spark_job</strong> — Submit a Spark job to the local cluster.</li><li><strong>query_trino</strong> — Run verifying SQL queries (&quot;Did the row count match?&quot;).</li><li><strong>read_logs</strong> — If a job fails, read the Airflow logs to diagnose the error (OutOfMemory vs ColumnNotFound).</li><li><strong>deploy_config</strong> — Write the YAML configs that drive the pipelines.</li></ol><p>The feedback loop is what makes this interesting. The agent doesn’t just write code — it owns tasks:</p><blockquote><strong><em>Task:</em></strong><em> “Ingest the new Supplier dataset.”</em></blockquote><blockquote><strong><em>Agent:</em></strong><em> Checks metadata → Generates Bronze YAML config → Deploys it → Triggers Airflow → Queries Trino to verify data landed → If row count is 0, self-corrects the config and retries.</em></blockquote><p>By running the entire stack on M4 silicon, the agent can iterate in seconds, not minutes. No cloud latency in the loop.</p><h3>What Went Well</h3><ul><li>Total control over every component. I understood every layer because I built every layer.</li><li>The config-as-code pattern genuinely works — it’s the architecture that makes agentic data engineering practical.</li><li>Iceberg + Trino sharing the same JDBC catalog means tables created by Spark are instantly queryable.</li><li>OpenLineage through Marquez gives full visibility into data flow.</li><li>Great for learning and for proving the concept to clients.</li></ul><h3>What Was Hard</h3><p>Infrastructure. Setting up 9 Docker services, getting Spark and Iceberg JAR dependencies right (iceberg-spark-runtime, AWS SDK v2, Hadoop AWS, PostgreSQL JDBC), configuring the JDBC catalog, wiring up MinIO buckets, debugging networking between containers — it all takes time.</p><p>The platform works well once it’s running. Getting to “running” requires patience and a good understanding of distributed systems.</p><h3>Verdict</h3><p>If you want maximum flexibility and vendor independence, this is the way. The config-as-code pattern is powerful and genuinely reduces the barrier for AI agents to interact with the pipeline. But you’re taking on the full infrastructure burden. For a consulting engagement or a team that values control, this is excellent. For a quick proof-of-concept, it might be more than you need.</p><blockquote><strong><em>GitHub:</em></strong><em> </em><a href="https://github.com/yasarkocyigit/agentic-data-engineer"><em>github.com/yasarkocyigit/agentic-data-engineer</em></a></blockquote><blockquote><strong><em>Medium:</em></strong><em> </em><a href="https://medium.com/@kocyigityasar/building-a-databricks-class-lakehouse-on-a-mac-mini-the-foundation-of-my-physical-ai-employee-f38e650e62f9"><em>Building a Databricks-Class Lakehouse on a Mac Mini</em></a></blockquote><h3>Attempt 3: Databricks AI Agents (Genie)</h3><p>After building everything from scratch, I wanted to see what the experience looks like on a managed platform designed for this kind of work. Databricks has been investing heavily in AI capabilities, and their <a href="https://docs.databricks.com/aws/en/genie/">Genie</a> product is essentially a natural language interface over your data — backed by Unity Catalog governance.</p><h3>What Makes It Different</h3><p>With Databricks, a lot of the infrastructure I built manually comes for free:</p><ul><li><strong>Unity Catalog</strong> handles governance, lineage, and access control.</li><li><strong>Delta Lake</strong> provides the table format with ACID transactions.</li><li><strong>Compute</strong> is managed — no Spark cluster configuration.</li><li><strong>The AI agent (Genie)</strong> understands your schemas, column descriptions, and relationships through catalog metadata.</li></ul><p>Setting up an AI agent in Databricks essentially means: register your tables in Unity Catalog with good descriptions, configure a Genie space, and users start asking questions. That’s it.</p><p>No MCP servers to deploy. No Docker containers to manage. No YAML configs to write. No pgvector to configure.</p><h3>What Went Well</h3><ul><li><strong>Setup time was dramatically shorter.</strong> What took days in Azure and weeks in the open-source stack took hours in Databricks.</li><li><strong>The governance story is clean.</strong> Unity Catalog means the agent respects permissions — if a user can’t see a table, the agent can’t query it on their behalf.</li><li><strong>The feedback loop is fast.</strong> Iterate on agent instructions, test queries, and refine behavior quickly.</li><li><strong>Built-in monitoring.</strong> See what queries the agent generated, review them, improve the system prompt.</li></ul><h3>What Was Hard</h3><p>Less hard, more “different.” You’re operating within Databricks’ paradigm. The flexibility is lower compared to the open-source stack you can’t implement a custom Recall-Plan-Gate-Execute loop like I did with the Fabric Control Plane. But for most enterprise use cases, the built-in capabilities cover what you need.</p><h3>Verdict</h3><p>Databricks Genie gave me the best experience in terms of time-to-value. If your data is already in Databricks or you’re planning to move there, this is the path of least resistance. The managed infrastructure and Unity Catalog integration remove a huge amount of friction.</p><h3>Comparing the Three Approaches</h3><p>Here’s my assessment after spending significant time with all three:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IkoT8XJafCIwoqqf8r_47w.png" /></figure><h3>The Real Tradeoff: Flexibility vs. Convenience</h3><p>The open-source stack gives you complete control every component, every behavior, no vendor lock-in. But you own all the infrastructure too.</p><p>Databricks gives you the fastest path to a working solution with solid governance, but you’re inside their ecosystem.</p><p>The Fabric Control Plane sits in between you get to design your own reasoning architecture (which I think matters a lot for production safety), but you’re deep in the Microsoft world.</p><h3>What I Actually Learned</h3><p>Beyond platform comparisons, a few broader observations stood out.</p><h3>Config-as-Code Is the Enabler</h3><p>The single most important architectural decision for agentic AI in data pipelines is making everything configuration-driven. When pipeline logic lives in YAML or a metadata table rather than in code, the surface area for AI interaction becomes manageable and safe. The agent doesn’t need to be a Spark expert it just needs to understand the config schema.</p><h3>Safety Can’t Be Bolted On</h3><p>In the Fabric project, I built the Gate from the start. In the open-source stack, I had to design it myself. Databricks handles it through Unity Catalog. Whatever platform you choose, the question of “what is the agent allowed to do?” needs to be answered on day one. Start with read-only access and expand carefully.</p><h3>The Agent Is Only as Good as the Metadata</h3><p>Consistent across all three platforms. Good table descriptions, clear column names, documented relationships → the agent performs well. Sparse or inconsistent metadata → the agent struggles. Just like a new engineer would.</p><h3>Memory Changes Everything</h3><p>Adding pgvector to the Fabric project was a turning point. Without persistent memory, the agent resets every conversation. With memory, it builds up context over time — past errors, schema changes, business rules. It goes from a tool to something closer to a team member.</p><h3>What’s Next</h3><p>I’m continuing to develop both the open-source stack and the Fabric Control Plane as reference architectures. The config-as-code foundation works regardless of which AI framework you layer on top.</p><p>For production, I’m leaning toward Databricks for teams already on that platform, and the Fabric Control Plane pattern for teams that need custom safety logic and reasoning chains.</p><p>The technology is moving fast. Six months ago, most of what I described here wasn’t possible without significant custom development. Today, it’s a weekend project on Databricks and a focused couple of weeks on the open-source stack.</p><p>If you’re a data engineer wondering whether agentic AI is worth exploring it is. Start small, start read-only, and let the architecture prove itself.</p><p><em>All projects are open-source:</em></p><ul><li><strong>Agentic Control Plane (Fabric/Azure):</strong> <a href="https://github.com/yasarkocyigit/agentic-ai-mcp">github.com/yasarkocyigit/agentic-ai-mcp</a></li><li><strong>Open-Source Lakehouse:</strong> <a href="https://github.com/yasarkocyigit/agentic-data-engineer">github.com/yasarkocyigit/agentic-data-engineer</a></li></ul><p><em>Previous posts in this series:</em></p><ul><li><a href="https://medium.com/@kocyigityasar/reducing-engineering-dependency-in-data-pipelines-with-agentic-ai-89b1c77f6426">Part 1: Reducing Engineering Dependency in Data Pipelines with Agentic AI</a></li><li><a href="https://medium.com/@kocyigityasar/from-metadata-driven-to-agentic-evolving-the-data-pipeline-architecture-part-3-cefb227ac51f">Part 3: From Metadata-Driven to Agentic — Evolving the Architecture</a></li><li><a href="https://medium.com/@kocyigityasar/building-a-databricks-class-lakehouse-on-a-mac-mini-the-foundation-of-my-physical-ai-employee-f38e650e62f9">Building a Databricks-Class Lakehouse on a Mac Mini</a></li></ul><p><em>If you’re building something similar or want to discuss the architecture, feel free to connect on </em><a href="https://www.linkedin.com/in/yasarkocyigit/"><em>LinkedIn</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fca6d4035984" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hired: My First AI Data Engineer -Clawdbot (And It runs on my Mac Mini)]]></title>
            <link>https://medium.com/@kocyigityasar/hired-my-first-ai-data-engineer-and-it-runs-on-my-mac-mini-820739c176af?source=rss-1ab647a5c7d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/820739c176af</guid>
            <dc:creator><![CDATA[Yasar Kocyigit]]></dc:creator>
            <pubDate>Thu, 12 Feb 2026 16:35:20 GMT</pubDate>
            <atom:updated>2026-02-12T16:58:23.118Z</atom:updated>
            <content:encoded><![CDATA[<p><em>From building the foundation to onboarding “ClawdBot” here is how I turned my local data stack into a collaborative agentic environment.</em></p><p>In my <a href="https://medium.com/@kocyigityasar/building-a-databricks-class-lakehouse-on-a-mac-mini-the-foundation-of-my-physical-ai-employee-f38e650e62f9">previous post</a>, I shared how I built a Databricks-class Lakehouse (Spark, Iceberg, Trino, Airflow, MinIO) running entirely on a Mac Mini M4. That was the <strong>Foundation</strong>.</p><p>Today, I’m sharing the <strong>Evolution</strong>.</p><p>I didn’t just build this stack to run manual pipelines. I built it to be the workspace for my first AI employee: <strong>ClawdBot</strong>.</p><h3>The Goal: Collaboration, Not Automation</h3><p>Most “Agentic AI” demos show an AI magically doing everything in a black box. That terrifies me as a Data Engineer. I don’t want magic; I want engineering.</p><p>My goal was to treat the AI Agent exactly like a <strong>Junior Data Engineer</strong>:</p><ol><li>It must work in the same environment I do.</li><li>It must commit code to GitHub.</li><li>It must <strong>NOT</strong> deploy to production without my approval.</li></ol><h3>The Architecture of Collaboration</h3><p>I designed a workflow where the Human (me) and the Agent (ClawdBot) work in parallel, meeting at the <strong>Collaboration Hub</strong> (GitHub).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mm1RTFMQUlrXoW-zpWPm4w.png" /></figure><h3>1. The Agent Environment (The “Physical” Employee)</h3><p>ClawdBot runs on the Mac Mini cluster. It has access to:</p><ul><li><strong>Terminal:</strong> To run Airflow commands, dbt tests, and Docker.</li><li><strong>IDE:</strong> To write and edit Python/SQL code.</li><li><strong>Local Data:</strong> The full Iceberg Lakehouse we built previously.</li></ul><h3>2. The Safety Valve: GitHub Flow</h3><p>This is the most critical part. ClawdBot does <strong>not</strong> have permission to push to the</p><p>main branch.</p><ul><li><strong>Step 1:</strong> ClawdBot receives a task (e.g., “Ingest the TPC-H data”).</li><li><strong>Step 2:</strong> It writes the code and tests it locally on the Mac Mini.</li><li><strong>Step 3:</strong> It opens a <strong>Pull Request (PR)</strong> on GitHub.</li><li><strong>Step 4:</strong> I review the code on my laptop. If it looks good, I merge it. If not, I comment, and ClawdBot fixes it.</li></ul><h3>Why This Changes Everything</h3><p>This workflow solves the “Hallucination Problem” in production. By forcing the AI to go through the standard software engineering lifecycle (Code -&gt; Test -&gt; PR -&gt; Review -&gt; Merge), we catch errors <em>before</em> they hit the production pipeline.</p><p>I am no longer writing every line of code. I am <strong>architecting</strong> the solution and <strong>reviewing</strong> the implementation.</p><p>In the next post, I’ll deep dive into the <strong>MCP (Model Context Protocol)</strong> server that connects Claude 4.6 to my local Airflow and Docker environment.</p><p><strong>Stay tuned.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=820739c176af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Databricks-Class Lakehouse on a Mac Mini: The Foundation of my “Physical AI Employee”]]></title>
            <link>https://medium.com/@kocyigityasar/building-a-databricks-class-lakehouse-on-a-mac-mini-the-foundation-of-my-physical-ai-employee-f38e650e62f9?source=rss-1ab647a5c7d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f38e650e62f9</guid>
            <dc:creator><![CDATA[Yasar Kocyigit]]></dc:creator>
            <pubDate>Mon, 09 Feb 2026 02:52:11 GMT</pubDate>
            <atom:updated>2026-02-09T02:52:11.776Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r08YrsZsUdAVjGngGC6x0w.png" /></figure><p><a href="https://medium.com/@kocyigityasar/when-bot-laptops-join-the-team-a-futuristic-take-on-openclaw-skill-supply-chains-and-5693557ca5d0"><em>(Continuation from “Reducing Engineering Dependency in Data Pipelines with Agentic AI”)</em></a></p><p>I ordered an M4 Mac Mini. Not just to run local LLMs, but to build a <strong>Physical AI Employee</strong> that lives on my desk.</p><p>But you can’t hire a Data Engineer even a digital one if you don’t have a data platform for them to work on. You can’t ask an agent to “optimize the pipeline” if there is no pipeline.</p><p>Technically, I could have prompted OpenClawd to “build me a data platform” from zero. It certainly has the capability. But based on my experience, successful platforms need a strong, opinionated foundation a “Golden Path.” I didn’t want the agent guessing at the architecture; I wanted to define the rules of the game.</p><p>So, I built the core framework myself. I set up the “Office” the standards, the tools, the infrastructure. By establishing this solid foundation first, I’m not just asking the agent to build <em>a</em> pipeline; I’m enabling it to continuously build and innovate upon <em>our</em> specific, optimized architecture.</p><p>Here is the technical blueprint of the <strong>Agentic Lakehouse</strong> I built.</p><h3>️ The Architecture: “The Local Cloud”</h3><p>The goal was to replicate a modern Metadata-Driven Lakehouse. We moved away from “scripting ETL” to “declarative asset definition”.</p><h3>1. The Storage Format: Why Apache Iceberg?</h3><p>We started with Delta Lake, but migrated to <strong>Apache Iceberg</strong>. Why? While Delta is fantastic (and tightly integrated with Databricks), <strong>Iceberg</strong> feels like the true “open standard” for the modern data stack.</p><ul><li><strong>Vendor Neutrality</strong>: Iceberg isn’t owned by a single platform. It works equally well with Spark, Trino, Flink, and Snowflake.</li><li><strong>Hidden Partitioning</strong>: This is a killer feature. In Hive/Delta, you often have to manually deal with partition columns (day, month). Iceberg handles this transparently. You query by timestamp, and Iceberg knows which partitions to skip.</li><li><strong>Schema Evolution</strong>: We can rename columns or change types without rewriting the entire petabyte of data.</li><li><strong>Time Travel</strong>: Our Agent can make mistakes. With Iceberg, we can execute ROLLBACK TO SNAPSHOT instantly. It gives the AI a &quot;Ctrl+Z&quot; button for data engineering.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uz1heN7imeTibfF62PrfqA.png" /></figure><h3>2. The Compute: Spark 4.1.1 + Declarative Pipelines</h3><p>We are running the bleeding-edge <strong>Apache Spark 4.1.1</strong>. But we aren’t writing 1000-line PySpark scripts. We utilize <strong>Spark Declarative Pipelines (SDP)</strong>.</p><ul><li><strong>The Concept</strong>: Instead of writing <em>how</em> to process data (imperative), we define <em>what</em> we want (declarative).</li><li><strong>For the AI</strong>: This is crucial. It’s hard for an LLM to write perfect, complex PySpark code. It is <em>easy</em> for an LLM to generate a clean YAML config defining “Source”, “Transformation”, and “Target”.</li></ul><h3>3. The Serving Layer: Trino (The SQL Engine)</h3><p>Data sitting in Parquet files is useless if you can’t query it fast. We deployed <strong>Trino</strong> (formerly PrestoSQL).</p><ul><li><strong>Role</strong>: It acts as our “Data Warehouse” engine.</li><li><strong>Capability</strong>: I can connect <strong>DataGrip</strong> or <strong>Power BI</strong> directly to localhost:8083 and run standard SQL queries against our Iceberg tables stored in MinIO. It separates Compute (Spark) from Analytics (Trino).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hHnDU1dnJ9MwbssjrzYtzQ.png" /></figure><h3>4. The Orchestrator: Airflow 3.0</h3><p>The nervous system. Airflow 3.0 triggers the Spark jobs, manages dependencies, and recovers from failures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EDUCh2R4WL0HE-rpc8VwiQ.png" /></figure><h3>5. The Storage: MinIO</h3><p>A high-performance, S3-compatible object storage. It tricks the entire stack into thinking it’s running on AWS, but it’s just a Docker container on the Mac.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DavlZCkUW6_E8_mIuGlLuQ.png" /></figure><h3>The “Physical AI Employee” (OpenClawd)</h3><p>Now that the office is built, how do we “train” the employee?</p><p>We are not fine-tuning a model on definitions of “Data Engineering”. We are building an <strong>Agentic System</strong> using <strong>MCP (Model Context Protocol)</strong>. This turns the Mac Mini M4 from a server into an intelligent worker.</p><h3>Phase 1: Context Stuffing (The Onboarding)</h3><p>Just like a new hire, the AI needs to know <em>our</em> way of doing things.</p><ul><li><strong>RAG (Retrieval-Augmented Generation)</strong>: We feed the agent our entire codebase, our task.md, and our technical documentation.</li><li><strong>Golden Paths</strong>: It knows that “Bronze” tables are always Parquet, and “Silver” tables are always Iceberg. It knows our naming conventions (clean_orders, not orders_v2).</li></ul><h3>Phase 2: MCP Tools (The Hands)</h3><p>An LLM dictates text. An Agent takes action. We are giving OpenClawd specific <strong>MCP Tools</strong> that act as its hands:</p><ol><li><strong>run_spark_job</strong>: It can submit a Spark job to the local cluster.</li><li><strong>query_trino</strong>: It can run verifying SQL queries (&quot;Did the row count match?&quot;).</li><li><strong>read_logs</strong>: If a job fails, it reads the Airflow logs itself to diagnose the error (e.g., &quot;OutOfMemory&quot; vs &quot;ColumnNotFound&quot;).</li><li><strong>deploy_config</strong>: It can write the YAML configs that drive the pipelines.</li></ol><h3>Phase 3: The Feedback Loop (The Experience)</h3><p>This is where the magic happens. The agent doesn’t just “write code”. It “owns tasks”.</p><ul><li><strong>Scenario</strong>: “Ingest the new Supplier dataset.”</li><li><strong>Agent Action</strong>:</li></ul><ol><li>Checks metadata to see if the table exists.</li><li>Generates the ‘Bronze’ YAML config.</li><li>Deploys it.</li><li>Triggers Airflow.</li><li><strong>Critical Step</strong>: It queries Trino to verify the data landed. If the row count is 0, it knows it failed, self-corrects the config, and retries.</li></ol><h3>Why Local? Why M4?</h3><p>By running this entire stack (Airflow, Spark, Trino, MinIO, Agent) on the M4 silicon, I remove the cloud latency loop. The agent can iterate on pipelines in seconds, not minutes.</p><p>This isn’t a chatbot. It’s a localized, autonomous Data Engineering unit. And it’s just getting its first ticket assigned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f38e650e62f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>