<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[DataAI - Medium]]></title>
        <description><![CDATA[Data &amp; AI topics - Medium]]></description>
        <link>https://medium.com/dataai?source=rss----9724fcf03e33---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>DataAI - Medium</title>
            <link>https://medium.com/dataai?source=rss----9724fcf03e33---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 15:55:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/dataai" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Vibe Code a Dashboard from Scratch]]></title>
            <link>https://medium.com/dataai/vibe-code-a-dashboard-from-scratch-3eeb3c89ff21?source=rss----9724fcf03e33---4</link>
            <guid isPermaLink="false">https://medium.com/p/3eeb3c89ff21</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Matt]]></dc:creator>
            <pubDate>Mon, 06 Apr 2026 12:40:35 GMT</pubDate>
            <atom:updated>2026-04-06T12:40:36.546Z</atom:updated>
            <content:encoded><![CDATA[<h3>Vibe Coding a Dashboard</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KXvibbhe-G66IUdx2SPeSg.png" /><figcaption>Generated by Gemini</figcaption></figure><p>Recently Jeffrey Shaffer posted about an experience he had building a dashboard that was generated through AI prompting. Naturally, I thought that seemed like a fun challenge and something that would be worth experimenting.</p><p>I’ll be the first to admit that my end output wasn’t nearly as good of a final product, but I had some very interesting takeaways from this project. If you are in the data visualization or data storytelling space I would highly recommend you try this process at least once.</p><p>I relied exclusively on AI tools to develop ALL of the code that was used in my final output. Yes, that caused a headache at many points along the way, but this led to a lot of learning. Some of my key takeaways and insights are below:</p><ol><li>Your AI tool may get stuck and cant find a solution. When this happens try a different tool.</li><li>The end-to-end development process is likely going to be far slower than building something directly in a tool like Power BI or Tableau.</li><li>The AI tools have a limited ability to identify and implement visualization best practices (without additional prompting).</li></ol><p>Basically, at this point, you probably don’t need to worry about AI building best-in-class visualizations with minimal effort. As the data viz practitioner it will continue to be important to understand data viz best practices and design standards.</p><h3>Tech and Data Overview</h3><p>First of all, if you are trying to replicate something like I did it’s helpful to have the context of the tools I was using to build out my dashboard. When it comes to the data, I used an API feed of San Antonio road pavement quality. My initial goal was to use traffic data, but I had to pivot as the AI tools struggled to get those feeds set up properly.</p><p>I built the dashboard itself using the Databricks Free Edition and d3.js to build a locally hosted dashboard. The goal here was to use only free and readily available tools. The Databricks Free Edition is a great way to be able to play around with the Databricks UI while connecting to real data. I’ve also been hearing a lot about d3.js dashboards and wanted to experiment with them. Effectively, two birds one stone.</p><p>The AI tools that I used were Claude in a VERY limited capacity and Google’s Gemini. For Claude, I just used the free version with limited tokens. Gemini was the powerhouse behind getting the process built out and was limited to Gemini 3 Fast. I suspect I could have benefited from using a more advanced version of Gemini but again the goal was to keep this as available to a broad reaching audience as possible.</p><h3>Setting up the API call</h3><p>This part was executed within Databricks. The goal here is to set up the initial data feed from the San Antonio pavement data API endpoint. Here is the prompt I used to set up the API feed.</p><blockquote>My goal is to create an end to end data visualization tool. This tool will visualize San Antonio pavement data patterns using geospatial maps and trend lines over time. The data must be sourced via API. The output visualization must be deployed via a D3 web dashboard.</blockquote><blockquote>Generate code that can be used within the Databricks Free Edition to collect the initial data via the API connection to the Texas department of transportation. Assume I have limited experience connecting to API endpoints.</blockquote><p>After repeated prompting I kept getting errors with the API call. Many of these errors were due to ill formatted querying against the API feed. Ultimately, this is where I pivoted to Claude to get the starting code seed. Claude was able to build code that connected on the first try and then I refined that code in Gemini after I ran out of Claude tokens.</p><p>In the Databricks Free Edition there are limitations to the way that data feeds can be structured for downstream use. Ultimately, after A LOT of prompting and iteration I was able to generate the following code that sets up the initial API call that then saves a json data feed. Below is the code (you’ll notice the references to traffic data from my initial research that I pivoted away from):</p><pre>import pandas as pd<br>import json<br>import base64<br>from pyspark.sql.functions import col, lit, when<br><br># 1. Convert to Spark<br>df = spark.createDataFrame(traffic_data)<br><br># 2. DATA REPAIR: Using Backticks to resolve the column name error<br># Spark requires ` ` when a column name contains a dot .<br>target_col = &quot;`attributes.PCI`&quot;<br><br>if &quot;attributes.PCI&quot; in df.columns:<br>    # Cast to integer for D3 math<br>    df_clean = df.withColumn(&quot;pci_score&quot;, col(target_col).cast(&quot;int&quot;))<br>else:<br>    # Fallback if the column is truly missing<br>    df_clean = df.withColumn(&quot;pci_score&quot;, lit(0))<br><br># 3. Create &#39;Health&#39; categories for high-contrast D3 colors<br>df_clean = df_clean.withColumn(&quot;health&quot;, <br>    when(col(&quot;pci_score&quot;) &gt;= 85, &quot;Excellent&quot;)<br>    .when(col(&quot;pci_score&quot;) &gt;= 70, &quot;Good&quot;)<br>    .when(col(&quot;pci_score&quot;) &gt;= 50, &quot;Fair&quot;)<br>    .when(col(&quot;pci_score&quot;) &gt; 0, &quot;Poor&quot;)<br>    .otherwise(&quot;Unknown&quot;)<br>)<br><br># 4. Final Prep (Ensuring we keep geometry for the map!)<br>final_pd = df_clean.toPandas()<br><br># 5. GENERATE DOWNLOAD BUTTON<br>json_string = final_pd.to_json(orient=&#39;records&#39;)<br>b64 = base64.b64encode(json_string.encode()).decode()<br>payload = f&quot;data:application/json;base64,{b64}&quot;<br><br>html_button = f&#39;&#39;&#39;<br>&lt;div style=&quot;background-color: #121212; padding: 20px; border: 1px solid #00ff41; border-radius: 8px; text-align: center; font-family: sans-serif;&quot;&gt;<br>    &lt;h3 style=&quot;color: #00ff41; margin-top: 0;&quot;&gt;Step 1: PCI Data Extracted&lt;/h3&gt;<br>    &lt;p style=&quot;color: #e0e0e0;&quot;&gt;Resolved Column Conflict using Backticks.&lt;/p&gt;<br>    &lt;a href=&quot;{payload}&quot; download=&quot;pavement_data.json&quot; <br>       style=&quot;background-color: #00ff41; color: #000; padding: 12px 25px; <br>              text-decoration: none; border-radius: 4px; font-weight: bold; <br>              display: inline-block; margin-top: 10px;&quot;&gt;<br>       DOWNLOAD REPAIRED JSON<br>    &lt;/a&gt;<br>&lt;/div&gt;<br>&#39;&#39;&#39;<br>displayHTML(html_button)</pre><h3>Building the Dashboard</h3><p>Now comes the really fun part. Unfortunately, I was out of Claude tokens at this point because I used them all to get the API feed stood up… Setting up the d3.js dashboard was a bit tricky because of the structure that is needed to general set up that is needed across multiple files.</p><p>To set up a d3.js dashboard you need to have a dedicated folder that houses 4 different files. These files tell your web app what data to reference, how to render the images, how to structure the overall dashboard, and an index file. Coordinating the code across each of these files will lead to the final dashboard output. However, as with all code, the different files could also be the source of a lot of code discrepancies.</p><p>Below is the code I used to generate my final dashboard:</p><h4>index.html</h4><pre>&lt;!DOCTYPE html&gt;<br>&lt;html lang=&quot;en&quot;&gt;<br>&lt;head&gt;<br>    &lt;meta charset=&quot;UTF-8&quot;&gt;<br>    &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1.0&quot;&gt;<br>    &lt;title&gt;San Antonio | Pavement Infrastructure Analysis&lt;/title&gt;<br>    <br>    &lt;script src=&quot;https://d3js.org/d3.v7.min.js&quot;&gt;&lt;/script&gt;<br>    <br>    &lt;link href=&quot;https://fonts.googleapis.com/css2?family=Inter:wght@300;400;600&amp;family=JetBrains+Mono&amp;display=swap&quot; rel=&quot;stylesheet&quot;&gt;<br>    <br>    &lt;link rel=&quot;stylesheet&quot; href=&quot;style.css&quot;&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br><br>    &lt;header class=&quot;app-header&quot;&gt;<br>        &lt;div class=&quot;header-content&quot;&gt;<br>            &lt;h1&gt;San Antonio | &lt;span&gt;Pavement Infrastructure Analysis&lt;/span&gt;&lt;/h1&gt;<br>            &lt;div class=&quot;status-badge&quot;&gt;<br>                &lt;span class=&quot;indicator&quot;&gt;&lt;/span&gt; Real-time Condition Monitor<br>            &lt;/div&gt;<br>        &lt;/div&gt;<br>    &lt;/header&gt;<br><br>    &lt;main class=&quot;dashboard-container&quot;&gt;<br>        <br>        &lt;aside class=&quot;sidebar&quot;&gt;<br>            &lt;section class=&quot;control-group&quot;&gt;<br>                &lt;h2&gt;Controls&lt;/h2&gt;<br>                &lt;p class=&quot;instruction&quot;&gt;Click legend items to toggle visibility&lt;/p&gt;<br>                <br>                &lt;div id=&quot;map-legend&quot; class=&quot;legend-container&quot;&gt;<br>                    &lt;/div&gt;<br>            &lt;/section&gt;<br><br>            &lt;section class=&quot;kpi-group&quot;&gt;<br>                &lt;div class=&quot;kpi-card&quot;&gt;<br>                    &lt;label&gt;Visible Segments&lt;/label&gt;<br>                    &lt;div id=&quot;kpi-miles&quot; class=&quot;kpi-value&quot;&gt;0&lt;/div&gt;<br>                &lt;/div&gt;<br>                &lt;div class=&quot;kpi-card&quot;&gt;<br>                    &lt;label&gt;Predominant Health&lt;/label&gt;<br>                    &lt;div id=&quot;kpi-type&quot; class=&quot;kpi-value excellent&quot;&gt;--&lt;/div&gt;<br>                &lt;/div&gt;<br>            &lt;/section&gt;<br>        &lt;/aside&gt;<br><br>        &lt;div class=&quot;viz-content&quot;&gt;<br>            <br>            &lt;div class=&quot;viz-card map-card&quot;&gt;<br>                &lt;div class=&quot;card-header&quot;&gt;<br>                    &lt;h3&gt;Spatial Distribution: Road Health&lt;/h3&gt;<br>                &lt;/div&gt;<br>                &lt;div id=&quot;map-viz&quot; class=&quot;svg-container&quot;&gt;<br>                    &lt;div class=&quot;loader&quot;&gt;Loading Geospatial Data...&lt;/div&gt;<br>                &lt;/div&gt;<br>            &lt;/div&gt;<br><br>            &lt;div class=&quot;viz-card trend-card&quot;&gt;<br>                &lt;div class=&quot;card-header&quot;&gt;<br>                    &lt;h3&gt;PCI Score Distribution (Pavement Condition Index)&lt;/h3&gt;<br>                &lt;/div&gt;<br>                &lt;div id=&quot;trend-viz&quot; class=&quot;svg-container&quot;&gt;<br>                    &lt;/div&gt;<br>            &lt;/div&gt;<br><br>        &lt;/div&gt;<br>    &lt;/main&gt;<br><br>    &lt;script src=&quot;dashboard.js&quot;&gt;&lt;/script&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><h4>style.css</h4><pre>/* --- 1. GLOBAL RESET &amp; FONTS --- */<br>:root {<br>    --bg-dark: #0f0f0f;<br>    --card-bg: #1a1a1a;<br>    --border-color: #333;<br>    --text-main: #e0e0e0;<br>    --text-dim: #888;<br>    <br>    /* Condition Colors matched to D3 */<br>    --excellent: #00ff41;<br>    --good: #00e5ff;<br>    --fair: #ffff00;<br>    --poor: #ff4500;<br>    --unknown: #444444;<br>}<br><br>body {<br>    background-color: var(--bg-dark);<br>    color: var(--text-main);<br>    font-family: &#39;Inter&#39;, -apple-system, BlinkMacSystemFont, sans-serif;<br>    margin: 0;<br>    display: flex;<br>    flex-direction: column;<br>    height: 100vh;<br>    overflow: hidden;<br>}<br><br>/* --- 2. HEADER --- */<br>.app-header {<br>    height: 60px;<br>    padding: 0 24px;<br>    display: flex;<br>    align-items: center;<br>    background: #000;<br>    border-bottom: 1px solid var(--border-color);<br>}<br><br>.app-header h1 {<br>    font-size: 1.2rem;<br>    font-weight: 600;<br>    letter-spacing: -0.5px;<br>}<br><br>.app-header span {<br>    color: var(--text-dim);<br>    font-weight: 300;<br>}<br><br>/* --- 3. LAYOUT ENGINE --- */<br>.dashboard-container {<br>    display: grid;<br>    grid-template-columns: 320px 1fr; /* Fixed Sidebar, Fluid Viz */<br>    gap: 20px;<br>    padding: 20px;<br>    flex-grow: 1;<br>    box-sizing: border-box;<br>}<br><br>/* Sidebar Styling */<br>.sidebar {<br>    display: flex;<br>    flex-direction: column;<br>    gap: 20px;<br>}<br><br>.control-group, .kpi-group {<br>    background: var(--card-bg);<br>    border: 1px solid var(--border-color);<br>    border-radius: 12px;<br>    padding: 20px;<br>}<br><br>/* Viz Content Stacking (Map on top of Trend) */<br>.viz-content {<br>    display: flex;<br>    flex-direction: column;<br>    gap: 20px;<br>    min-width: 0; /* Prevents D3 from overflowing container */<br>}<br><br>/* --- 4. CARDS &amp; VISUALS --- */<br>.viz-card {<br>    flex: 1;<br>    background: var(--card-bg);<br>    border: 1px solid var(--border-color);<br>    border-radius: 12px;<br>    padding: 20px;<br>    display: flex;<br>    flex-direction: column;<br>    position: relative;<br>}<br><br>.card-header h3 {<br>    margin: 0 0 15px 0;<br>    font-size: 0.9rem;<br>    text-transform: uppercase;<br>    letter-spacing: 1px;<br>    color: var(--text-dim);<br>}<br><br>.svg-container {<br>    flex-grow: 1;<br>    width: 100%;<br>    min-height: 0;<br>}<br><br>/* --- 5. LEGEND &amp; CONTROLS --- */<br>.legend-container {<br>    margin-top: 15px;<br>}<br><br>.legend-item {<br>    display: flex;<br>    align-items: center;<br>    padding: 6px 8px;<br>    border-radius: 6px;<br>    transition: background 0.2s;<br>}<br><br>.legend-item:hover {<br>    background: rgba(255, 255, 255, 0.05);<br>}<br><br>.instruction {<br>    font-size: 0.8rem;<br>    color: var(--text-dim);<br>    margin-bottom: 10px;<br>}<br><br>/* --- 6. KPI STYLING --- */<br>.kpi-card {<br>    margin-bottom: 15px;<br>}<br><br>.kpi-card:last-child { margin-bottom: 0; }<br><br>.kpi-card label {<br>    display: block;<br>    font-size: 0.75rem;<br>    color: var(--text-dim);<br>    text-transform: uppercase;<br>    margin-bottom: 5px;<br>}<br><br>.kpi-value {<br>    font-family: &#39;JetBrains Mono&#39;, monospace;<br>    font-size: 1.8rem;<br>    font-weight: 600;<br>}<br><br>/* Dynamic KPI Colors */<br>.kpi-value.excellent { color: var(--excellent); text-shadow: 0 0 15px rgba(0,255,65,0.2); }<br>.kpi-value.good { color: var(--good); }<br>.kpi-value.fair { color: var(--fair); }<br>.kpi-value.poor { color: var(--poor); }<br>.kpi-value.unknown { color: var(--unknown); }<br><br>/* --- 7. D3 SPECIFIC OVERRIDES --- */<br>path.road-path {<br>    transition: stroke-width 0.2s;<br>}<br><br>path.road-path:hover {<br>    stroke-width: 4px !important;<br>    cursor: crosshair;<br>}<br><br>.domain, .tick line {<br>    stroke: #444; /* Axis colors */<br>}<br><br>.tick text {<br>    fill: #888;<br>    font-size: 10px;<br>}</pre><h4>dashboard.js</h4><pre>// 1. GLOBAL STATE<br>let globalData = [];<br>let filteredData = [];<br>const colorScheme = {<br>    &quot;Excellent&quot;: &quot;#00ff41&quot;, <br>    &quot;Good&quot;: &quot;#00e5ff&quot;,      <br>    &quot;Fair&quot;: &quot;#ffff00&quot;,      <br>    &quot;Poor&quot;: &quot;#ff4500&quot;,      <br>    &quot;Unknown&quot;: &quot;#444444&quot;<br>};<br><br>// Initialize with ALL filters active<br>let activeFilters = new Set([&quot;Excellent&quot;, &quot;Good&quot;, &quot;Fair&quot;, &quot;Poor&quot;, &quot;Unknown&quot;]);<br><br>// 2. DATA LOADING ENGINE<br>d3.json(&#39;pavement_data.json&#39;).then(data =&gt; {<br>    // This will bring back your sample observation for debugging<br>    console.log(&quot;Sample Observation:&quot;, data[0]);<br><br>    globalData = data.map(d =&gt; {<br>        // Convert Unix Timestamp (ms) to Year for the trend chart<br>        const rawDate = d[&#39;attributes.InstallDate&#39;];<br>        const yearInt = rawDate ? new Date(rawDate).getFullYear() : 0;<br><br>        return {<br>            ...d,<br>            health: d.health || &quot;Unknown&quot;,<br>            pci: d.pci_score || 0,<br>            year: yearInt,<br>            geometry: { <br>                type: &quot;LineString&quot;, <br>                coordinates: d[&#39;geometry.paths&#39;] ? d[&#39;geometry.paths&#39;][0] : [] <br>            }<br>        };<br>    }).filter(d =&gt; d.geometry.coordinates.length &gt; 0); // Remove empty geometries<br><br>    filteredData = [...globalData];<br>    initDashboard();<br>}).catch(err =&gt; console.error(&quot;Loading Error:&quot;, err));<br><br>// 3. INITIALIZATION<br>function initDashboard() {<br>    renderLegend();<br>    updateKPIs();<br>    <br>    // Give CSS 100ms to calculate the grid before drawing SVGs<br>    setTimeout(() =&gt; {<br>        renderMap();<br>        renderTrend();<br>    }, 100);<br>}<br><br>// 4. MAP RENDER (Using Auto-Fit)<br>function renderMap() {<br>    const container = d3.select(&quot;#map-viz&quot;);<br>    const width = container.node().clientWidth;<br>    const height = container.node().clientHeight;<br>    <br>    const svg = container.html(&quot;&quot;).append(&quot;svg&quot;)<br>        .attr(&quot;width&quot;, width)<br>        .attr(&quot;height&quot;, height);<br>    <br>    const g = svg.append(&quot;g&quot;);<br><br>    // 1. Create a FeatureCollection for D3 to measure<br>    const geojson = {<br>        type: &quot;FeatureCollection&quot;,<br>        features: filteredData.map(d =&gt; ({<br>            type: &quot;Feature&quot;,<br>            geometry: d.geometry<br>        }))<br>    };<br><br>    // 2. AUTO-SCALE: This forces the data to fill the box<br>    const projection = d3.geoMercator()<br>        .fitSize([width, height], geojson); // &lt;--- THIS IS THE MAGIC LINE<br><br>    const pathGen = d3.geoPath().projection(projection);<br><br>    g.selectAll(&quot;path&quot;)<br>        .data(filteredData)<br>        .enter().append(&quot;path&quot;)<br>        .attr(&quot;d&quot;, d =&gt; pathGen(d.geometry))<br>        .attr(&quot;stroke&quot;, d =&gt; colorScheme[d.health] || &quot;#888&quot;)<br>        .attr(&quot;stroke-width&quot;, 1.5)<br>        .attr(&quot;fill&quot;, &quot;none&quot;)<br>        .style(&quot;opacity&quot;, 0.8);<br><br>    // Add Zoom<br>    svg.call(d3.zoom().on(&quot;zoom&quot;, (event) =&gt; g.attr(&quot;transform&quot;, event.transform)));<br>}<br><br>// 5. LEGEND &amp; FILTERING<br>function renderLegend() {<br>    const legend = d3.select(&quot;#map-legend&quot;).html(&quot;&quot;);<br>    <br>    Object.keys(colorScheme).forEach(category =&gt; {<br>        const item = legend.append(&quot;div&quot;)<br>            .attr(&quot;class&quot;, &quot;legend-item&quot;)<br>            .style(&quot;display&quot;, &quot;flex&quot;)<br>            .style(&quot;align-items&quot;, &quot;center&quot;)<br>            .style(&quot;margin-bottom&quot;, &quot;8px&quot;)<br>            .style(&quot;cursor&quot;, &quot;pointer&quot;)<br>            .style(&quot;opacity&quot;, activeFilters.has(category) ? 1 : 0.3)<br>            .on(&quot;click&quot;, function() {<br>                if (activeFilters.has(category)) activeFilters.delete(category);<br>                else activeFilters.add(category);<br>                <br>                filteredData = globalData.filter(d =&gt; activeFilters.has(d.health));<br>                d3.select(this).style(&quot;opacity&quot;, activeFilters.has(category) ? 1 : 0.3);<br>                <br>                renderMap();<br>                renderTrend();<br>                updateKPIs();<br>            });<br><br>        item.append(&quot;span&quot;)<br>            .style(&quot;width&quot;, &quot;12px&quot;).style(&quot;height&quot;, &quot;12px&quot;)<br>            .style(&quot;background-color&quot;, colorScheme[category])<br>            .style(&quot;margin-right&quot;, &quot;10px&quot;).style(&quot;border-radius&quot;, &quot;2px&quot;);<br><br>        item.append(&quot;label&quot;).text(category).style(&quot;cursor&quot;, &quot;pointer&quot;).style(&quot;color&quot;, &quot;#eee&quot;);<br>    });<br>}<br><br>// 6. TREND CHART (PCI Distribution)<br>function renderTrend() {<br>    const container = d3.select(&quot;#trend-viz&quot;);<br>    const width = container.node().clientWidth - 80;<br>    const height = Math.max(container.node().clientHeight - 80, 150);<br><br>    const svg = container.html(&quot;&quot;).append(&quot;svg&quot;)<br>        .attr(&quot;width&quot;, &quot;100%&quot;)<br>        .attr(&quot;height&quot;, &quot;100%&quot;)<br>        .append(&quot;g&quot;)<br>        .attr(&quot;transform&quot;, `translate(50, 20)`);<br><br>    // 1. Define X Scale<br>    const x = d3.scaleLinear().domain([0, 100]).range([0, width]);<br><br>    // 2. Generate Bins<br>    const histogram = d3.bin()<br>        .value(d =&gt; d.pci)<br>        .domain(x.domain())<br>        .thresholds(10);<br><br>    const bins = histogram(filteredData);<br><br>    // 3. Define Y Scale <br>    const y = d3.scaleLinear()<br>        .domain([0, d3.max(bins, d =&gt; d.length) || 10])<br>        .range([height, 0]);<br><br>    // 4. Draw Grouped/Stacked Bars<br>    bins.forEach(bin =&gt; {<br>        // Group the data inside this specific bin by health status<br>        const counts = d3.rollup(bin, v =&gt; v.length, d =&gt; d.health);<br>        let yOffset = 0;<br><br>        // Draw a sub-rect for every health type present in this PCI range<br>        Array.from(counts).forEach(([health, count]) =&gt; {<br>            const barHeight = height - y(count);<br>            <br>            svg.append(&quot;rect&quot;)<br>                .attr(&quot;x&quot;, x(bin.x0) + 1)<br>                .attr(&quot;width&quot;, Math.max(0, x(bin.x1) - x(bin.x0) - 1))<br>                .attr(&quot;y&quot;, y(count) - yOffset) <br>                .attr(&quot;height&quot;, barHeight)<br>                .attr(&quot;fill&quot;, colorScheme[health] || &quot;#888&quot;) // &lt;--- Syncs with Map!<br>                .attr(&quot;opacity&quot;, 0.8)<br>                .append(&quot;title&quot;) // Tooltip for clarity<br>                .text(`${health}: ${count} segments`);<br><br>            yOffset += barHeight; // Stack them if multiple types fall in one bin<br>        });<br>    });<br><br>    // Axes<br>    svg.append(&quot;g&quot;).attr(&quot;transform&quot;, `translate(0,${height})`).call(d3.axisBottom(x));<br>    svg.append(&quot;g&quot;).call(d3.axisLeft(y).ticks(5));<br>}<br><br>// 7. KPIs<br>function updateKPIs() {<br>    const total = filteredData.length;<br>    <br>    // Group and find the most common health status<br>    const counts = d3.rollup(filteredData, v =&gt; v.length, d =&gt; d.health);<br>    const topEntry = d3.greatest(counts, d =&gt; d[1]);<br>    <br>    // Safety check: if no data, default to N/A / unknown<br>    const topHealth = topEntry ? topEntry[0] : &#39;Unknown&#39;;<br>    const healthClass = topHealth.toLowerCase();<br><br>    // Update the HTML<br>    d3.select(&#39;#kpi-miles&#39;).text(total.toLocaleString());<br>    <br>    d3.select(&#39;#kpi-type&#39;)<br>        .text(topHealth)<br>        // This removes old classes and adds the new one for the color sync<br>        .attr(&quot;class&quot;, `kpi-value ${healthClass}`);<br>}<br><br></pre><p>After the code was created and synced across each of the files I went it a local command line editor on my laptop to set up a locally hosted version of the dashboard. In that editor I navigated to the folder containing these 4 files and typed in the following code (<em>python -m http.server 8000</em>) to set up a locally hosted web app. Then I opened a new browser and entered <em>localhost:8000 </em>to load the dashboard.</p><h3>Conclusion</h3><p>At the end of the project I ended up getting the following dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZVEoAfDHP9DneQVSLMa6JA.png" /></figure><p>Keep in mind all of this was built from AI directly. If I were to build this on my own for a real work project this is far from the desired final output. However, this approach does show a ton of promise and is something I’ll be exploring further. I relied entirely on Gemini to be able to derive the design and all relevant KPIs as well.</p><p>At this point, the development process here was significantly slower and much less iterative than what I’m used to being able to do in Tableau. If I need to get something fast out to stakeholders this isn’t the approach I would use, but building a generalizable framework on top of this is very appealing as well.</p><p>If you’re even remotely curious about doing something like this let me know. I’m curious to see how other people approach a similar problem and to see your successes.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p><em>If this post was helpful leave me a comment here or connect with me on </em><a href="https://www.linkedin.com/in/mhuff04/"><em>LinkedIn</em></a><em>.</em></p><p><em>Note that this post was not written by AI (except for the code… obviously).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3eeb3c89ff21" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataai/vibe-code-a-dashboard-from-scratch-3eeb3c89ff21">Vibe Code a Dashboard from Scratch</a> was originally published in <a href="https://medium.com/dataai">DataAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Test-Time Training: TTT-Discover]]></title>
            <link>https://medium.com/dataai/test-time-training-ttt-discover-a8be30142598?source=rss----9724fcf03e33---4</link>
            <guid isPermaLink="false">https://medium.com/p/a8be30142598</guid>
            <dc:creator><![CDATA[Carey Chou]]></dc:creator>
            <pubDate>Thu, 05 Feb 2026 04:49:06 GMT</pubDate>
            <atom:updated>2026-02-05T04:48:44.680Z</atom:updated>
            <content:encoded><![CDATA[<p>Most of our experience with Large Language Models (LLMs) involves a frozen brain. You send a prompt, and the model — with weights fixed in time — predicts the next token. If the model fails at a complex coding task or a mathematical proof, we simply prompt it again or try “Best-of-N” sampling, hoping it “gets lucky.”</p><p>The release of <strong>TTT-Discover</strong> (from the paper <em>“Learning to Discover at Test Time” </em><a href="https://arxiv.org/pdf/2601.16175">https://arxiv.org/pdf/2601.16175</a>) introduces a radical alternative: <strong>What if the model could learn from its own peak in real-time to fine-tune the model for a single problem?</strong></p><h3>The Problem: The “Frozen Brain” Plateau</h3><p>Standard inference techniques like Best-of-N are “democratic.” They treat every attempt as an independent event. If you ask an LLM to optimize a GPU kernel, it might give you 100 versions. 99 might be slow or broken, and one might be a breakthrough.</p><p>In a standard system, the model never realizes it found a miracle. It doesn’t learn from that success to make its next 100 attempts even better. It stays stagnant.</p><h3>The Solution: TTT-Discover &amp; The J-Beta Objective</h3><p>TTT-Discover turns the LLM into a Test-Time Optimizer. Instead of just sampling, the model enters a loop of acting, being evaluated by an external verifier (like a compiler), and then — crucially — updating its own weights before the next attempt.</p><p>The heart of this discovery is the J_beta Entropic Utility objective.</p><h3>Standard RL vs. J-Beta</h3><p>In standard Reinforcement Learning (REINFORCE), we maximize the <strong>average reward</strong>. This is great for making a chatbot “general purpose.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*ER6A4DiG-PfcP0aDSBIzXA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*rbmwpBUd6YFfs7QVp6RfHA.png" /></figure><p>But for discovery, the average is noise. You don’t want an “average” GPU kernel; you want the one that breaks the world record.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*aJEvhWmpvaM_39UsjMCdWQ.png" /></figure><p>By exponentially weighting the rewards, the model focuses its entire gradient update on the <strong>outliers</strong> — the rare, high-performing solutions. It ignores the 99 failures and “specializes” its weights toward the logic of the one success.</p><h3>The Implementation: End-to-End Discovery with LoRA</h3><p>We use <strong>LoRA (Low-Rank Adaptation)</strong> to update only a small set of “specialist” weights during the test session. Below is a functional Python skeleton of the TTT-Discover loop:</p><pre>import torch<br>import torch.nn.functional as F<br>from transformers import AutoModelForCausalLM, AutoTokenizer<br>from peft import LoraConfig, get_peft_model<br>import numpy as np<br><br># 1. THE EVALUATOR (The &quot;Environment&quot;)<br>def evaluate_kernel_performance(code_string):<br>    &quot;&quot;&quot;<br>    Placeholder: In a real blog, this would be a subprocess <br>    calling &#39;triton&#39; or &#39;g++&#39; to benchmark the code.<br>    &quot;&quot;&quot;<br>    # Simulate a reward based on code length or a random discovery<br>    reward = len(code_string) / 1000.0 + np.random.uniform(0, 0.5)<br>    return reward<br><br># 2. LOG-PROBABILITY EXTRACTOR<br>def get_sequence_log_probs(model, tokenizer, prompt, generations):<br>    &quot;&quot;&quot;Calculates log P(generation | prompt) for the policy gradient.&quot;&quot;&quot;<br>    # Concatenate prompt + generation<br>    texts = [prompt + gen for gen in generations]<br>    inputs = tokenizer(texts, return_tensors=&quot;pt&quot;, padding=True, truncation=True).to(model.device)<br>    <br>    # Forward pass<br>    outputs = model(**inputs)<br>    logits = outputs.logits # [B, Seq, Vocab]<br>    <br>    # Identify where the generation starts<br>    prompt_len = tokenizer(prompt, return_tensors=&quot;pt&quot;)[&#39;input_ids&#39;].shape[1]<br>    <br>    # Shift to align logits with target tokens<br>    shift_logits = logits[:, prompt_len-1:-1, :].contiguous()<br>    shift_labels = inputs.input_ids[:, prompt_len:].contiguous()<br>    <br>    # Gather log_probs of the actual tokens sampled<br>    log_probs = F.log_softmax(shift_logits, dim=-1)<br>    token_log_probs = torch.gather(log_probs, 2, shift_labels.unsqueeze(-1)).squeeze(-1)<br>    <br>    # Mask out padding tokens<br>    mask = (shift_labels != tokenizer.pad_token_id).float()<br>    return (token_log_probs * mask).sum(dim=-1)<br><br># 3. THE TTT-DISCOVER ENGINE<br>class DiscoveryAgent:<br>    def __init__(self, model_id=&quot;meta-llama/Llama-3-8B&quot;):<br>        self.tokenizer = AutoTokenizer.from_pretrained(model_id)<br>        self.tokenizer.pad_token = self.tokenizer.eos_token<br>        <br>        # Initialize with LoRA<br>        base_model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map=&quot;auto&quot;)<br>        peft_config = LoraConfig(r=16, lora_alpha=32, target_modules=[&quot;q_proj&quot;, &quot;v_proj&quot;])<br>        self.model = get_peft_model(base_model, peft_config)<br>        <br>        self.optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-5)<br>        self.buffer = [] # To store {&#39;code&#39;: str, &#39;reward&#39;: float, &#39;visits&#39;: int}<br><br>    def run_discovery(self, problem_prompt, steps=50, batch_size=4):<br>        print(f&quot;--- Starting TTT-Discover for Problem ---\n&quot;)<br>        <br>        for i in range(steps):<br>            # A. Select Seed (PUCT Logic)<br>            seed = self.select_seed()<br>            current_prompt = f&quot;{problem_prompt}\nBest current solution:\n{seed[&#39;code&#39;]}&quot; if seed else problem_prompt<br>            <br>            # B. Act (Sample Rollouts)<br>            inputs = self.tokenizer(current_prompt, return_tensors=&quot;pt&quot;).to(self.model.device)<br>            output_tokens = self.model.generate(<br>                **inputs, max_new_tokens=128, num_return_sequences=batch_size, do_sample=True, temperature=0.8<br>            )<br>            <br>            # Decode only the NEW tokens<br>            generations = [self.tokenizer.decode(o[inputs.input_ids.shape[1]:], skip_special_tokens=True) for o in output_tokens]<br>            <br>            # C. Evaluate (External Reward)<br>            rewards = torch.tensor([evaluate_kernel_performance(g) for g in generations]).to(self.model.device)<br>            <br>            # Save to buffer<br>            for g, r in zip(generations, rewards):<br>                self.buffer.append({&#39;code&#39;: g, &#39;reward&#39;: r.item(), &#39;visits&#39;: 0})<br>            <br>            # D. Test-Time Training (J-Beta Update)<br>            # Calculate Adaptive Beta based on reward spread<br>            beta = 1.0 / (rewards.std() + 1e-6)<br>            weights = F.softmax(beta * rewards, dim=0)<br>            <br>            # Get Log-Probs for gradients<br>            log_probs = get_sequence_log_probs(self.model, self.tokenizer, current_prompt, generations)<br>            <br>            # Loss = - weighted log_prob (maximizing reward)<br>            loss = -(weights.detach() * log_probs).sum()<br>            <br>            loss.backward()<br>            self.optimizer.step()<br>            self.optimizer.zero_grad()<br>            <br>            print(f&quot;Step {i} | Best Reward: {max(r[&#39;reward&#39;] for r in self.buffer):.4f} | Beta: {beta:.2f}&quot;)<br><br>    def select_seed(self):<br>        if not self.buffer: return None<br>        # Simplified PUCT: Pick the highest reward solution from buffer<br>        return max(self.buffer, key=lambda x: x[&#39;reward&#39;])<br><br># 4. MAIN EXECUTION<br>if __name__ == &quot;__main__&quot;:<br>    agent = DiscoveryAgent()<br>    problem = &quot;Write a high-performance Triton kernel for 2D Depthwise Convolution.&quot;<br>    agent.run_discovery(problem, steps=10)</pre><p>By allowing models to learn at test time, we enable them moving toward <strong>Agentic Autonomy</strong></p><ul><li><strong>Solve OOD Problems:</strong> Solve tasks that weren’t in their training data by experimenting.</li><li><strong>Surpass Human Baselines:</strong> TTT-Discover has already set new state-of-the-art records in GPU kernel optimization and mathematical problems (like Erdős’ minimum overlap).</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8be30142598" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataai/test-time-training-ttt-discover-a8be30142598">Test-Time Training: TTT-Discover</a> was originally published in <a href="https://medium.com/dataai">DataAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The New Human Roles of AI: Why Boundaries, Not Models, Define the Future of Work]]></title>
            <link>https://medium.com/dataai/the-new-human-roles-of-ai-why-boundaries-not-models-define-the-future-of-work-e9706e7c5a48?source=rss----9724fcf03e33---4</link>
            <guid isPermaLink="false">https://medium.com/p/e9706e7c5a48</guid>
            <dc:creator><![CDATA[Carey Chou]]></dc:creator>
            <pubDate>Tue, 16 Dec 2025 18:15:25 GMT</pubDate>
            <atom:updated>2025-12-16T18:15:21.716Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Wall Street Journal recently published an article titled <em>“Four New Jobs That May Be in Our AI Future.” </em></strong><a href="https://www.wsj.com/tech/ai/new-jobs-ai-future-fefd4b35?mod=ai_lead_pos3">https://www.wsj.com/tech/ai/new-jobs-ai-future-fefd4b35?mod=ai_lead_pos3</a></p><p>At first glance, it reads like a prediction about emerging job titles. But the deeper insight of the article is not about jobs at all.</p><p>It is about <strong>boundaries</strong>.</p><p>As AI systems become more capable, organizations are discovering that the hardest problems are no longer technical — they are about where responsibility, judgment, and accountability sit between humans and machines.</p><p>The four roles described in the article exist because AI does not eliminate boundaries. It creates new ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*gtaSHh-1XUhUWpiRWdI5xA.png" /></figure><h3>The Real Pattern: AI Creates New Boundaries That Humans Must Own</h3><p>Each of the four roles in the WSJ article emerges at a critical boundary where AI alone is insufficient:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*8eHgy5-H6ZvNC4t6pxzg5A.png" /></figure><p>AI systems operate probabilistically, adapt over time, and increasingly influence real-world decisions. That means someone must own the space between what the AI does and what the organization is willing to stand behind. These roles are how that ownership shows up in practice.</p><h3>From WSJ “New Jobs” to Industry AI Roles</h3><p>To make these roles actionable for enterprises, it helps to translate them into <strong>industry-ready roles</strong> that align with operating models, governance, and scale.</p><h4>From AI Explainer → AI Translator</h4><p><strong>Owning the boundary between AI output and human judgment</strong></p><p>The WSJ describes AI explainers as people who translate complex AI systems for managers, judges, and regulators. In industry, this role is better understood as an <strong>AI Translator</strong>.</p><p>The AI Translator sits at the boundary between:</p><ul><li>What the model produces</li><li>What leaders, regulators, and customers need to understand and trust</li></ul><p>This role:</p><ul><li>Explains <em>why</em> an AI made a recommendation, not just <em>what</em> it said</li><li>Connects model behavior to business, legal, and ethical implications</li><li>Supports accountability when AI influences high-stakes decisions</li></ul><p>AI Translators become essential in domains like:</p><ul><li>Credit and lending</li><li>Healthcare recommendations</li><li>Hiring and eligibility decisions</li><li>Pricing, safety, and compliance-driven environments</li></ul><h4>From AI Chooser → AI Architect</h4><p><strong>Owning the boundary between technology capability and business intent</strong></p><p>The “AI chooser” role exists because not all AI is the same — and misapplied AI is often worse than no AI at all.</p><p>In enterprises, this role becomes the <strong>AI Architect</strong>.</p><p>The AI Architect sits at the boundary between:</p><ul><li>What AI technologies can technically do</li><li>What the business actually needs to achieve</li></ul><p>This role:</p><ul><li>Decides where AI should be used — and where it should not</li><li>Chooses between predictive, generative, rules-based, or hybrid approaches</li><li>Balances speed, cost, explainability, and risk</li><li>Designs AI as part of an end-to-end operating model, not a standalone tool</li></ul><p>AI Architects prevent organizations from chasing hype and instead ensure AI is intentional, contextual, and scalable.</p><h4>From AI Auditors &amp; Cleaners → Responsible AI &amp; AI Operations</h4><p><strong>Owning the boundary between AI behavior and acceptable risk</strong></p><p>The WSJ groups auditors and cleaners together, but in practice they form the backbone of <strong>Responsible AI and AI Operations</strong>.</p><p>These roles sit at the boundary between:</p><ul><li>What the model is doing in production</li><li>What the organization considers fair, compliant, and acceptable</li></ul><p>This boundary exists because AI risk is not static.</p><p>Models drift. Data changes. Feedback loops emerge. Bias can appear months after deployment.</p><p>AI Auditors:</p><ul><li>Continuously monitor performance, bias, and drift</li><li>Validate outcomes across populations and time</li></ul><p>AI Cleaners (or Operators):</p><ul><li>Retrain or recalibrate models</li><li>Adjust constraints, features, or thresholds</li><li>Intervene when systems operate outside guardrails</li></ul><p>Together, these roles turn AI governance from a one-time review into an ongoing operational discipline.</p><h3>From AI Trainer → AI Enablement &amp; Change Agent</h3><p><strong>Owning the boundary between AI capability and human adoption</strong></p><p>The final role in the article addresses a truth many organizations learn too late. The biggest risk in AI is not model failure — it’s adoption failure.</p><p>In industry, the AI trainer becomes an <strong>AI Enablement or Change Agent</strong>.</p><p>This role sits at the boundary between:</p><ul><li>What AI can theoretically do</li><li>What people are actually ready to use and trust</li></ul><p>AI Enablement focuses on:</p><ul><li>Teaching people how to work <em>with</em> AI, not just how to use tools</li><li>Helping employees question, validate, and apply AI outputs</li><li>Using AI itself to personalize learning and upskilling</li><li>Embedding AI into daily workflows instead of one-off pilots</li></ul><p>Without this role, even the best AI systems stall.</p><h3>The Bigger Insight: AI Is Forcing a Redesign of Human Responsibility</h3><p>The WSJ article may frame these as “new jobs,” but the deeper message is this:</p><blockquote><em>As AI systems become more powerful, humans move closer to ownership of decisions — not farther away.</em></blockquote><p>Across industries, we are seeing a new human architecture emerge — one defined by <strong>boundary ownership</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*sLgMN4bubG-y9eV9k4EC-g.png" /></figure><p>Organizations that fail to intentionally design these roles will still need them — but they will emerge informally, inconsistently, and reactively.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9706e7c5a48" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataai/the-new-human-roles-of-ai-why-boundaries-not-models-define-the-future-of-work-e9706e7c5a48">The New Human Roles of AI: Why Boundaries, Not Models, Define the Future of Work</a> was originally published in <a href="https://medium.com/dataai">DataAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Biggest Mistakes to Avoid When Implementing Generative AI]]></title>
            <link>https://medium.com/dataai/the-biggest-mistakes-to-avoid-when-implementing-generative-ai-d9f3e701d764?source=rss----9724fcf03e33---4</link>
            <guid isPermaLink="false">https://medium.com/p/d9f3e701d764</guid>
            <category><![CDATA[genarative-ai]]></category>
            <dc:creator><![CDATA[Anshulika Singh]]></dc:creator>
            <pubDate>Tue, 16 Dec 2025 17:08:57 GMT</pubDate>
            <atom:updated>2025-12-16T17:08:57.146Z</atom:updated>
            <content:encoded><![CDATA[<p>Generative AI has moved from experimental labs into boardrooms, workflows, and product roadmaps at record speed. Organizations everywhere are exploring how to use it to reduce costs, increase productivity, and spark innovation. But in the rush to adopt, many fall into predictable and expensive traps.</p><p>If you are planning to implement generative AI in your team or organization, here are the mistakes you must avoid to realize real value without chaos, risk, or disappointment.</p><h4>Starting With Technology Instead of a Business Problem</h4><p>One of the biggest missteps companies make is leading with the question: “What can we do with AI?” instead of “What problem are we trying to solve?”</p><p>When the technology leads the conversation, companies invest in flashy use cases that don’t meaningfully move the needle. Tools get adopted, but outcomes don’t.</p><p>💡 Do this instead: Start with a pain point, inefficiency, bottleneck, or growth opportunity. Then ask how generative AI can enable the solution, not be the solution itself.</p><h4>Overestimating the Tech, Underestimating the Change Management</h4><p>Generative AI is not a plug-and-play magic wand. The biggest obstacles to adoption are rarely technical, they’re cultural. Teams fear replacement. Leadership fears risk. Processes get disrupted. Workflows need redesign. Without intentional change management, even the best AI solutions stall.</p><p>💡 Do this instead: Invest in training, build trust, communicate benefits early, and establish a clear “AI + human” collaboration model.</p><h4>Running Experiments With No Clear Path to Production</h4><p>Pilots are exciting. Demos impress. But many organizations get stuck in an endless loop of pilots that never scale because:</p><ul><li>Ownership is unclear</li><li>Data is messy</li><li>Infrastructure isn’t ready</li><li>Legal has not blessed the use case</li><li>No KPI is defined</li></ul><p>Without a production strategy, pilots generate activity, not impact.</p><p>💡 Do this instead: Define success metrics, assign accountable owners, and plan for integration before you build the prototype.</p><h4>Ignoring Data Quality and Governance</h4><p>Gen AI is only as good as the data it learns from. Yet many assume the model can magically “figure it out. Poor data leads to poor answers. Lack of governance leads to compliance headaches. Also, forgetting data privacy can lead to catastrophic exposure.</p><p>💡 Do this instead: Implement strong data hygiene, access controls, and governance standards before scaling workloads.</p><h4>Not Defining Guardrails for Safety and Ethics</h4><p>Employees will use generative AI even without official approval. What’s risky is when there are no rules around:</p><ul><li>What data can be entered</li><li>How outputs should be validated</li><li>How to avoid hallucinations</li><li>Intellectual property concerns</li><li>Bias and fairness checks</li><li>Customer-facing content approvals</li></ul><p>💡 Do this instead: Create clear guidelines and use policies. Promote responsible use. Set a review process for high-impact outputs.</p><h4>Treating Gen AI as a Cost Saver Only</h4><p>Many companies approach AI with one goal: efficiency, but the real value of generative AI is not just savings, it’s growth.</p><p>It can reinvent products, personalize customer experiences, accelerate insights, and expand what your workforce is capable of.</p><p>If you limit it to cost-cutting, you miss the bigger opportunity.</p><p>💡 Do this instead: Balance efficiency use cases with innovation and customer experience opportunities.</p><h4>Failing to Measure Real ROI</h4><p>Generative AI can produce nice demos but still fail to produce real business value. Teams often measure activity instead of measurable improvement.</p><p>💡 Do this instead: Track metrics like:</p><ul><li>Time saved</li><li>Cost avoided</li><li>Revenue generated</li><li>Quality improvement</li><li>Customer satisfaction</li><li>Reduction in cycle time</li><li>The value should be visible, quantifiable, and attributable.</li></ul><h4>Expecting Perfection Instead of Designing for Iteration</h4><p>Gen AI is probabilistic, not deterministic. It will make mistakes. Waiting for perfect accuracy delays all progress. The most successful adopters treat AI like a product: test, learn, refine, scale.</p><p>💡 Do this instead: Start small, release early, iterate based on feedback, and continuously retrain or fine-tune.</p><p><strong>Conclusion</strong>: Adopt with Strategy, Not Hype. Generative AI is transformative but only when adopted thoughtfully. Whether you are leading a team, a function, or an enterprise initiative, avoid these common pitfalls and focus on what actually drives value:</p><ul><li>Clear business problem</li><li>Strong governance</li><li>Human-centric design</li><li>Scalable architecture</li><li>Measurable outcomes</li></ul><p>Organizations that move with clarity not just excitement will be the ones that win in the AI era.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9f3e701d764" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataai/the-biggest-mistakes-to-avoid-when-implementing-generative-ai-d9f3e701d764">The Biggest Mistakes to Avoid When Implementing Generative AI</a> was originally published in <a href="https://medium.com/dataai">DataAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everyone’s Using AI to Replace Rule-Based Systems. We Used OpenEvolve to Improve Them]]></title>
            <link>https://medium.com/dataai/everyones-using-ai-to-replace-rule-based-systems-we-used-openevolve-to-improve-them-02101ebe862a?source=rss----9724fcf03e33---4</link>
            <guid isPermaLink="false">https://medium.com/p/02101ebe862a</guid>
            <dc:creator><![CDATA[Jin Hao Huang]]></dc:creator>
            <pubDate>Mon, 08 Dec 2025 16:32:46 GMT</pubDate>
            <atom:updated>2025-12-08T16:32:44.962Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The prevailing AI narrative is clear: replace old techniques with better, smarter AI solutions. Rule-based systems? Legacy thinking. Keyword matching? Outdated. But what if the real opportunity isn’t replacing these foundational approaches — it’s using AI to improve them? This is the story of how we used OpenEvolve to generate optimized keyword search rules that delivered 42% cost savings, fully transparent logic, and a reminder that the best use of AI isn’t always at inference time. Sometimes it’s generating the rules that run forever after.</em></p><h3>Not Every Problem Needs an AI Solution at Runtime</h3><p>Imagine you need to extract structured attributes from messy product descriptions for a major grocery retailer — things like flavor profiles, dietary classifications, and product categories.</p><p>Our LLM-powered workflow to create these product attributes was working beautifully. It could handle semantic complexity, understand context, manage ambiguity — all the capabilities that make LLMs valuable. After proving value, we started to scale to handle hundreds of thousands of products, each with dozens of attributes leading to millions of unique combinations. But throughout this scaling process, we saw space to be even more efficient.</p><p><strong>Here’s what we noticed</strong>: Many of our attribution cases were straightforward, as products labels and descriptions had clear keyword patterns. For example, products with “2%” were dairy and products labeled “Organic” were… organic. We were paying premium LLM costs for decisions that simpler pattern matching could handle just as accurately.</p><p>Recent research presented at AAAI 2025 validated this intuition — keyword-based approaches can achieve over 90% accuracy for information retrieval tasks when properly implemented, rivaling more expensive methods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*BWIVIAGgsFmMbetK-SJeXw.png" /></figure><p><em>Source: </em><a href="https://www.linkedin.com/posts/shreyassub_amazonbedrock-ai-machinelearning-activity-7293710273742274561-cZyh"><em>Shreyas Subramanian, PhD’s Post</em></a></p><h3>AlphaEvolve: AI at Build Time, Rules at Runtime</h3><p>Around this time, Google released their AlphaEvolve whitepaper, an evolutionary system using AI to automatically optimize code and generate novel solutions. Traditional genetic algorithms have struggled with code optimization for decades: random mutations almost always produce broken or meaningless variations, making the search space practically infinite. AlphaEvolve’s breakthrough was using LLMs as the mutation engine to finally solve this bottleneck. Instead of random perturbations, you get semantically intelligent variations like “what if we added fuzzy matching for typos?” That’s directed exploration, not blind search.</p><p>Building on the approach <a href="https://medium.com/careychou-ideaoneer/teaching-ai-to-segment-shopping-baskets-how-we-use-alphaevolve-with-map-elites-q-learning-205f706716fb">referenced here</a> that recreates in simple terms parts of the algorithm in the whitepaper, the hypothesis crystallized: use evolution to automatically generate attribute-specific keyword search code that’s:</p><ul><li><strong>Cost-effective</strong>: Generate once, reuse infinitely at near-zero cost</li><li><strong>Transparent</strong>: Editable Python code instead of black-box LLM calls</li><li><strong>Scalable</strong>: Systematic deployment across hundreds of attributes</li></ul><p>As we planned our implementation strategy, the landscape shifted. <a href="https://huggingface.co/blog/codelion/openevolve"><strong>OpenEvolve was released open-source — one week after Google’s whitepaper.</strong></a> The community had built a working alternative with the enhancements we needed.</p><p>We pivoted immediately.</p><p>Rather than investing more time building AI solutions from scratch, we could leverage a battle-tested implementation with powerful features already built in. In AI’s rapidly evolving landscape, this adaptability and understanding of hybrid solutions that weren’t purely LLM-driven delivered production value in weeks instead of months.</p><h3>OpenEvolve: AI-Powered Keyword Search Rules</h3><p>OpenEvolve similarly combines LLM creativity with evolutionary optimization algorithms like MAP-Elites to automatically discover and refine code solutions.</p><p>Think of <strong>MAP-Elites like a library with specialized sections</strong>. Traditional optimization keeps one book called “the best solution” and keeps rewriting it. MAP-Elites maintains an entire library — the best exact-match solution on shelf A, the best fuzzy-match solution on shelf B, the best regex-pattern solution on shelf C, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vAaLhVy0mlHXI3Rp2gLhDw.png" /></figure><p><em>Source: </em><a href="https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/AlphaEvolve.pdf?utm_source=deepmind.google&amp;utm_medium=referral&amp;utm_campaign=gdm&amp;utm_content="><em>AlphaEvolve Whitepaper</em></a></p><p>When generating new ideas, the LLM pulls from different shelves, combining strengths from fundamentally different approaches. A solution might merge exact matching (shelf A) with fuzzy logic (shelf B) to create something novel (shelf D). This diversity prevents getting stuck in local optima — you’re exploring the entire solution space, not just refining one path.</p><p><strong>In our case for keyword search, this means discovering and preserving diverse approaches and taking inspiration from the best performers to create the hybrid, optimized final solution.</strong> For example, we have one “shelf” for solutions that only executed direct string matching, another that focused on regex patterns and fuzzy matching, and a third solution type purely on attribute synonyms and domain-specific similarities. When generating new solutions, OpenEvolve draws from multiple shelves simultaneously, combining the precision of exact matching with the flexibility of regex and domain-specific synonyms. Each generation builds on the previous, iteratively refining these hybrid approaches until we arrive at a final solution that synthesizes the strengths of all three strategies.</p><h3>How to Teach AI to Write Optimized Rules</h3><p>OpenEvolve required three inputs to generate optimized keyword search rules:</p><p><strong>1. Configuration</strong>: Parameters defining the evolutionary process.</p><p><strong>2. Initial Code</strong>: A basic template as a starting point — evolution handles the sophistication. This template just needs to provide a structural foundation for the LLM to understand the problem space. For our keyword search, we started with simple keyword lists. OpenEvolve evolved these into complex multi-strategy solutions with fuzzy matching, regex patterns, and confidence scoring.</p><pre># Initial Code Template (Before Evolution)<br><br>def predict_attribute_values(product_description):<br>    &quot;&quot;&quot;<br>    Basic keyword matching template - OpenEvolve evolves this into <br>    sophisticated multi-strategy solutions<br>    &quot;&quot;&quot;<br>    product_text = product_description.lower()<br>    predictions = []<br>    <br>    # Simple keyword lists as starting point<br>    type_a_keywords = [&quot;keyword1&quot;, &quot;keyword2&quot;]<br>    type_b_keywords = [&quot;keyword3&quot;, &quot;keyword4&quot;]<br>    <br>    # Basic exact matching logic<br>    if any(keyword in product_text for keyword in type_a_keywords):<br>        predictions.append(&quot;Type_A&quot;)<br>    <br>    if any(keyword in product_text for keyword in type_b_keywords):<br>        predictions.append(&quot;Type_B&quot;)<br>    <br>    return predictions<br><br># Evolution transforms this into:<br># - Fuzzy matching for typos and variations<br># - Regex patterns for complex text structures  <br># - Confidence scoring for cascade decisions<br># - Multi-strategy combinations optimized per attribute</pre><p><strong>3. The Evaluator</strong>: This was the hard part. We built a function measuring <strong>F1 score</strong>, which balances precision (accuracy) and recall (coverage) — translating business requirements (“tolerate unknowns, minimize wrong answers”) into mathematical objectives.</p><p>Building the evaluator forced clarity: <strong>What does “good enough” actually mean?</strong></p><ul><li>How do we balance the trade-off between avoiding wrong attribution cases (precision) versus finding correct attribute values (recall)?</li><li>What F1 score threshold makes us confident enough to route products away from LLM review? Should this score dynamically change depending on the attribute or category?</li><li>How do we quantify the cost-benefit trade-off and the number of optimization iterations we run?</li><li>How do we identify “good quality/golden” data to evaluate our results against?</li></ul><p>By solving the above questions, we gained clarity and understanding of the project that gave us value independent of the actual optimization results. The process of defining our evaluation metrics lead to improved documentation, shared understanding between technical teams and stakeholders, and a framework for evaluating any future attribution approach.</p><p><strong>The logging advantage</strong>: OpenEvolve captures every iteration’s code changes, LLM reasoning, and performance metrics. This transparency proved invaluable for debugging, stakeholder communication, and building trust in the generated solutions.</p><pre># 2025-11-05 12:00:00: OpenEvolve Evolution Log - Iteration X <br><br># Improvement: <br>Add keyword variations and synonyms for better matching<br><br># Reasoning: <br>Expanding the keyword list will increase the likelihood of <br>correctly identifying attribute values, particularly for edge cases where <br>alternate terms are used in product descriptions.<br><br>&lt;&lt;&lt;&lt;&lt;&lt;&lt; SEARCH (Current Change)<br>primary_keywords = [&quot;type_a&quot;, &quot;variant_1&quot;]<br>=======<br><br>primary_keywords = [&quot;type_a&quot;, &quot;variant_1&quot;, &quot;alt_type_a&quot;, &quot;synonym_1&quot;]<br>&gt;&gt;&gt;&gt;&gt;&gt;&gt; REPLACE (Incoming Change)<br><br>---<br><br># Performance Impact:<br>- Accuracy: 0.73 → 0.78 (+6.8%)<br>- Precision: 81.3% (maintained)<br>- Recall: 54.2% → 59.1% (+4.9%)</pre><h3>Results that Rule</h3><p><strong>The hybrid strategy</strong>: Run all products through keyword search first. Confidence scoring determines whether to accept keyword search results or route to LLM for complex semantic reasoning.</p><pre># Cost Savings Calculation<br><br># Theoretical Max Cost Savings: <br>Rule_Generation_Rate × Average_Coverage_Per_Attribute = ~50%<br><br># Generation Cost Deduction:<br>One-time OpenEvolve generation costs = ~5-10%<br><br># Net Cost Savings:<br>Theoretical Max Cost Savings - Generation Costs <br>  = 50% - (5-10%)<br>  = 42.5%</pre><p><strong>Results:</strong></p><ul><li>Approximately <strong>42% cost savings</strong> expected at full deployment</li><li><strong>100% performance improvement</strong> from initial code to evolved solution (improving from 0.41 to 0.82 F1 score)</li><li>For example:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZVZCu5mYprqx4spPA9YSiQ.png" /></figure><p>Validation proved the approach worked at scale. But the most valuable outcomes weren’t just the numbers.</p><p><strong>The Unexpected Wins:</strong></p><ol><li><strong>Data Quality Auditor: </strong>Evolution logs revealed missing values in our existing list of attribute values, from which we were able to discover new values that didn’t exist before. The system surfaced gaps that were invisible in black-box LLM approaches.</li><li><strong>Domain Knowledge Extraction: </strong>Subject matter experts requested log access — not for debugging, but for insights. The system discovered patterns they’d unconsciously known but never articulated.</li><li><strong>Transparency as a Feature: </strong>Evolved code is visible and editable. When business logic changes, stakeholders can modify rules directly instead of retraining models. For high-stakes decisions, this visibility proved as valuable as cost savings.</li></ol><h3>Key Takeaways</h3><p><strong>On AI-generated rules:</strong></p><ul><li>Rule-based systems aren’t obsolete — they’re ready to be augmented</li><li>Prior exploration of evolutionary approaches proved viability from first principles</li><li>Transparent, editable rules beats LLM black-box inference for many production problems</li></ul><p><strong>On finding the right problems:</strong></p><ul><li>Not every workflow needs an LLM at runtime</li><li>Look for high-volume decisions with identifiable patterns and clear evaluation criteria</li><li>Ask: Can we quantify success? Is the problem complex enough? Will evolved solutions beat manual development?</li></ul><p><strong>When evolutionary optimization makes sense:</strong></p><ul><li>✅ Complex, multi-objective problems with clear metrics</li><li>✅ Non-linear solution spaces requiring exploration</li><li>✅ Performance-critical code with compounding value</li><li>❌ Simple problems with established solutions</li><li>❌ Unclear evaluation criteria or subjective outcomes</li></ul><p><strong>On building evaluators:</strong></p><ul><li>Forces you to define “good enough” mathematically</li><li>That clarity becomes documentation and shared understanding</li><li>Value exists independent of optimization results</li></ul><h3>What’s Next?</h3><p>The OpenEvolve algorithm used here is impressive, and the business impact from the implementation is tangible and significant.</p><p>But the real lesson isn’t about any single technology. It’s about recognizing that AI’s role isn’t always at inference time. Sometimes the highest-leverage use of AI is generating the rules, patterns, and code that run at near-zero cost forever after.</p><p>OpenEvolve optimized our keyword search. But the real optimization? Recognizing that the best AI strategies don’t abandon foundational techniques — they augment and improve on them. The future isn’t rules <em>vs </em>AI. It’s rules <em>written by</em> AI.</p><p><em>Want to try this approach yourself? OpenEvolve is open-source and actively maintained. Start with a small test case, build a robust evaluator, and pay special attention to when evolutionary optimization actually makes sense for your problem. Not every challenge needs evolution — but when patterns exist, AI-generated rules can be transformative.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=02101ebe862a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataai/everyones-using-ai-to-replace-rule-based-systems-we-used-openevolve-to-improve-them-02101ebe862a">Everyone’s Using AI to Replace Rule-Based Systems. We Used OpenEvolve to Improve Them</a> was originally published in <a href="https://medium.com/dataai">DataAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Nested Learning for Recommender Systems: Bringing Fast and Slow Learning to Personalization]]></title>
            <link>https://medium.com/dataai/nested-learning-for-recommender-systems-bringing-fast-and-slow-learning-to-personalization-eef38209ace5?source=rss----9724fcf03e33---4</link>
            <guid isPermaLink="false">https://medium.com/p/eef38209ace5</guid>
            <dc:creator><![CDATA[Carey Chou]]></dc:creator>
            <pubDate>Fri, 21 Nov 2025 02:14:41 GMT</pubDate>
            <atom:updated>2025-11-28T15:16:19.459Z</atom:updated>
            <content:encoded><![CDATA[<p>Modern recommendation systems face a paradox:</p><ul><li>Users expect instant personalization — adaptation after a single click.</li><li>But recommender models rely on slow retraining cycles — hourly or nightly.</li><li>Transformers and deep models can store long-term knowledge, but cannot update that knowledge during inference.</li></ul><p>The result, as the recent <strong>Nested Learning (NeurIPS 2025 </strong><a href="https://abehrouz.github.io/files/NL.pdf">https://abehrouz.github.io/files/NL.pdf</a><strong>)</strong> paper from Google Research argues, is that today’s models suffer from a kind of <em>anterograde amnesia</em>:</p><blockquote><em>They cannot integrate new experiences (interactions) into long-term memory while running.</em></blockquote><p>Nested Learning proposes a biologically inspired framework for fixing this: multiple memory systems operating at different time scales, all inside the model.</p><p>This article will explain:</p><ol><li>The core ideas from the Nested Learning paper</li><li>How its equations (13/16, 17/20, 25) formalize fast and slow learning</li><li>How we can apply this pattern to retail recommendation using a simple Python implementation</li></ol><p>Then apply the concept in product recommendation as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*3oQgZGnl8KZQWkuRB_M2_Q.png" /></figure><h3>What Is Nested Learning?</h3><p>The paper argues that a neural network is not one learner, instead; it is <strong>several learners stacked inside each other</strong>, each running at a different speed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*T62LkVvtx1bG8KMBBZ6FYg.png" /></figure><p>Three components align perfectly with deep learning:</p><h4>Fast Memory: Attention</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/201/1*oYPi2QUg58brUnZteRl6Lg.png" /></figure><p>This is Eq. <strong>13/16</strong> in the paper.</p><h4>Middle Memory: Optimizer Momentum</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*PW62q9p5GKYuilpdN_1FKA.png" /></figure><p>Optimizers like SGD+momentum or Adam actually maintain an internal state: This is Eq. <strong>17–20</strong>. Momentum is not a hack; it is a memory of recent surprise signals.</p><h4>Slow Memory: Model Weights</h4><p>The slow weights update with in Eq. <strong>25</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*QwgSPQ3j6xk88ZnVlFsk3g.png" /></figure><h3>Equation Walkthrough</h3><p>Let’s walk through the three equations that matter most.</p><h4>(A) Fast Associative Memory (Eq. 13/16)</h4><p>The paper shows that <strong>attention is itself a gradient descent update</strong> on an internal memory matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/217/1*cKG9lEkgKQKYJv2Hu9dxIg.png" /></figure><p>Why outer product?</p><ul><li><strong>kₜ</strong> is the key</li><li><strong>vₜ</strong> is the value</li><li>Their outer product is a short-lived associative trace</li></ul><p>This lets the model adapt <em>instantly</em> to new behavior — without retraining.</p><h4>(B) Momentum as Memory (Eq. 17/20)</h4><p>Momentum stores a compressed gradient history:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*QNEhndAqZ5rvFIJ38FE8zA.png" /></figure><p>This becomes the <em>optimizer-level</em> fast memory, storing “what surprised me recently.” This plays a critical role in slow weight updates.</p><h4>(C) Outer Slow Weight Update (Eq. 25)</h4><p>At much slower intervals, the real model parameters update:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/185/1*bc4KX16bD5PYfiymtCfbHw.png" /></figure><p>The key insight: Slow weights do NOT update every step; they update using the compressed memory from the fast system. This gives us fast online adaptation + slow long-term learning.</p><h3>Turning Nested Learning into a Personalized Recommender System</h3><p>Most personalized recommenders today behave like Transformers:</p><ul><li>They infer &amp; <em>score</em> interactions online based on customer’s preference and interaction</li><li>But the model does not <strong>adapt</strong> or <strong>modify</strong> their parameters online</li><li>Updates require large batch retraining</li></ul><p>Let’s walk through what actually happens when a customer opens your e-commerce site. Imagine a customer lands on e-commerce homepage. In most recommenders today, The customer receives a static ranking from inferencing and scoring based on:</p><ul><li>the customer’s long-term profile</li><li>historical and in-session behavior</li><li>users who look like the customer</li></ul><p>Nothing in a traditional model adapts in real time to the customer’s behavior as he browses. Nested Learning changes that. In our implementation, every action the customer takes immediately reshapes the model’s fast memory, updating his recommendations as he clicks and scrolls. Over time, these fast-memory insights are gradually consolidated into the slow weights, becoming part of the model’s long-term knowledge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*I1zMLx89bLLRxwGRmlT8KQ.png" /></figure><h4>STEP 1 — Customer views an Item → Fast Memory Updates (Eq. 13 / 16)</h4><p>He clicks on a yogurt product. NL model forms the feature vector:</p><pre>x = concat(user_embedding[Emily], item_embedding[Yogurt])</pre><p>Then NL computes attention projections:</p><pre>k = x @ W_k<br>v = x @ W_v<br>q = x @ W_q</pre><p>Now Eq. <strong>13 / 16</strong> kicks in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/198/1*c-xvwXWyda4eMKc3n3u2QA.png" /></figure><p>In the code:</p><pre>self.memory += np.outer(v, k)</pre><p>This does something very important in adaptation: The recommender creates a short-term associative memory — A lightweight “trace” linking Emily’s current preferences to yogurt-type items. This memory could be <strong>session-scoped or multi-channel scoped</strong>: it reflects what the customer is interested in right now, not since last time model was fully trained. We don’t need to retrain anything yet — the fast memory handles this live.</p><h4>STEP 2 — Customer clicks or ignores items → Momentum Memory Updates (Eq. 17 / 20)</h4><p>This customer now clicks a second item — a protein bar. Or maybe he ignores it.</p><p>NL system computes the prediction error:</p><pre>error = prob - label<br>grad = error * x</pre><p>This gradient feeds into the <strong>momentum memory</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*CGaXQisP0AvrggCB1gTl8g.png" /></figure><pre>self.state.grad_memory =<br>      beta * self.state.grad_memory +<br>      (1 - beta) * grad</pre><p>This memory captures: the direction customer’s preferences are shifting</p><ul><li>If he clicks on dairy items → gradient moves weights toward dairy signals</li><li>If he ignores spicy snacks → gradients move weights away from those signals</li></ul><p>Noise is filtered out. Trends are kept. This is <em>mid-term memory</em> — not immediate like attention, not slow like retraining.</p><h4>STEP 3 — Fast Weights Update Instantly</h4><p>Fast weights adjust <em>right away</em> using the momentum memory:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*Tdeduan07Rui64luw0G4Gw.png" /></figure><pre>self.state.fast_weights =<br>     self.state.fast_weights - eta_fast * self.state.grad_memory</pre><p>As every new interactions changes the customer’s personalized ranking. Within just 3–5 interactions, the recommender has customized itself for Emily in ways that traditional systems cannot — it is like a “<strong>personalized recommendation model</strong>” for the customer.</p><h4>STEP 4 — Offline (Slow) Update Consolidates Across All Users (Eq. 25)</h4><p>When it’s time to retrain the full model, we can consolidate the fast weights into the slow weights using Eq. 28 and Eq. 29. These updates incorporate not only the accumulated fast weights but also the input feature vectors, ensuring that long-term memory becomes feature-aware rather than just a stored average.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*pk28eNld0W5IxH9sAK5ONA.png" /></figure><p>In today’s industry models:</p><ul><li>Personalization is driven by <em>offline retraining</em></li><li>Online adaptation is limited to bandits or heuristic rules</li><li>The core model cannot update itself during a session</li></ul><p>Nested Learning changes the architecture fundamentally:</p><h4>Real-time “Personalized Model”</h4><p>Every interaction reshapes the fast memory and fast weights. NL recommender adapts on the fly like a human store associate:</p><blockquote><em>“Ah, this customer picked Greek yogurt and protein bars — let me adapt my memory to learn more about protein-rich snacks.”</em></blockquote><h4>Momentum memory handles short-term preference drift</h4><p>If the customer suddenly shifts from “healthy snacks” to “baking supplies,”<br> the model updates seamlessly.</p><h4>Slow weights learn long-term, population-level behavior</h4><p>The long term memory system still captures:</p><ul><li>lifetime events</li><li>loyalty habits</li><li>household profiles</li><li>seasonal trends</li><li>slow shift of purchase patterns</li></ul><h4>The dual-memory system is biologically plausible</h4><p>Humans use:</p><ul><li><strong>short-term working memory</strong> (fast attention)</li><li><strong>medium-term consolidation</strong></li><li><strong>long-term memory</strong></li></ul><p>And Nested Learning mirrors this architecture.</p><p><strong>NOTE:</strong> This is not a production recommender but a conceptual implementation that mirrors the Nested Learning architecture.</p><pre>&quot;&quot;&quot;Recommender with fast online updates and periodic slow commits.&quot;&quot;&quot;<br><br>from __future__ import annotations<br><br>from dataclasses import dataclass, field<br>from typing import Dict, Iterable, Tuple<br><br>import numpy as np<br>from numpy.typing import ArrayLike, NDArray<br><br><br>@dataclass<br>class OnlineOfflineState:<br>    &quot;&quot;&quot;Tracks slow and fast weights alongside momentum memory.&quot;&quot;&quot;<br><br>    slow_weights: NDArray[np.float64]<br>    fast_weights: NDArray[np.float64]<br>    grad_memory: NDArray[np.float64]<br><br><br>def _sigmoid(x: NDArray[np.float64]) -&gt; NDArray[np.float64]:<br>    return 1.0 / (1.0 + np.exp(-x))<br><br><br>@dataclass<br>class OnlineOfflineRecommender:<br>    &quot;&quot;&quot;Nested learning recommender with fast online and slow offline updates.<br><br>    Online step:<br>      - Eq. (12)-(16): update user-specific fast attention memory.<br>      - Eq. (17)-(20): update momentum-style gradient memory.<br>      - Fast weights use the smoothed gradient immediately for scoring.<br>    Offline step (less frequent):<br>      - Eq. (25): apply the slow gradient descent step to slow weights using<br>        the accumulated gradient memory, then resynchronise fast weights.<br>    &quot;&quot;&quot;<br><br>    num_users: int<br>    num_items: int<br>    user_dim: int = 8<br>    item_dim: int = 8<br>    memory_dim: int = 8<br>    eta_fast: float = 0.05<br>    eta_slow: float = 0.01<br>    beta: float = 0.9<br>    slow_update_interval: int = 10<br>    rng: np.random.Generator = field(default_factory=lambda: np.random.default_rng(0))<br>    state: OnlineOfflineState = field(init=False)<br>    W_k: NDArray[np.float64] = field(init=False)<br>    W_v: NDArray[np.float64] = field(init=False)<br>    W_q: NDArray[np.float64] = field(init=False)<br>    W_fast_out: NDArray[np.float64] = field(init=False)<br>    memory: NDArray[np.float64] = field(init=False)<br>    user_embeddings: NDArray[np.float64] = field(init=False)<br>    item_embeddings: NDArray[np.float64] = field(init=False)<br>    step_count: int = field(init=False, default=0)<br><br>    def __post_init__(self) -&gt; None:<br>        input_dim = self.user_dim + self.item_dim<br>        slow_weights = self.rng.normal(scale=0.01, size=input_dim)<br>        fast_weights = slow_weights.copy()<br>        grad_memory = np.zeros_like(slow_weights)<br>        self.state = OnlineOfflineState(<br>            slow_weights=slow_weights,<br>            fast_weights=fast_weights,<br>            grad_memory=grad_memory,<br>        )<br>        self.W_k = self.rng.normal(scale=0.05, size=(input_dim, self.memory_dim))<br>        self.W_v = self.rng.normal(scale=0.05, size=(input_dim, self.memory_dim))<br>        self.W_q = self.rng.normal(scale=0.05, size=(input_dim, self.memory_dim))<br>        self.W_fast_out = self.rng.normal(scale=0.05, size=self.memory_dim)<br>        self.memory = np.zeros((self.memory_dim, self.memory_dim), dtype=np.float64)<br>        self.user_embeddings = self.rng.normal(scale=0.1, size=(self.num_users, self.user_dim))<br>        self.item_embeddings = self.rng.normal(scale=0.1, size=(self.num_items, self.item_dim))<br><br>    @property<br>    def input_dim(self) -&gt; int:<br>        return self.user_dim + self.item_dim<br><br>    def _features(self, user_id: int, item_id: int) -&gt; NDArray[np.float64]:<br>        return np.concatenate(<br>            [self.user_embeddings[user_id], self.item_embeddings[item_id]],<br>            axis=0,<br>        )<br><br>    def _attention(self, x: NDArray[np.float64], *, update: bool) -&gt; Dict[str, NDArray[np.float64]]:<br>        k = x @ self.W_k<br>        v = x @ self.W_v<br>        q = x @ self.W_q<br>        if update:<br>            self.memory += np.outer(v, k)<br>        y_fast_vec = self.memory @ q<br>        fast_scalar = float(self.W_fast_out @ y_fast_vec)<br>        return {&quot;k&quot;: k, &quot;v&quot;: v, &quot;q&quot;: q, &quot;fast_vec&quot;: y_fast_vec, &quot;fast_scalar&quot;: fast_scalar}<br><br>    def score(self, user_id: int, item_id: int) -&gt; float:<br>        x = self._features(user_id, item_id)<br>        attention = self._attention(x, update=False)<br>        y_slow = float(self.state.fast_weights @ x)<br>        logit = y_slow + attention[&quot;fast_scalar&quot;]<br>        return float(_sigmoid(np.array([logit]))[0])<br><br>    def recommend(self, user_id: int, item_ids: NDArray[np.int_]) -&gt; NDArray[np.float64]:<br>        return np.asarray([self.score(user_id, item) for item in item_ids], dtype=np.float64)<br><br>    def online_step(self, user_id: int, item_id: int, label: float) -&gt; Dict[str, float | NDArray[np.float64]]:<br>        x = self._features(user_id, item_id)<br>        attention = self._attention(x, update=True)<br>        y_fast = float(self.state.fast_weights @ x)<br>        logit = y_fast + attention[&quot;fast_scalar&quot;]<br>        prob = float(_sigmoid(np.array([logit]))[0])<br>        loss = -label * np.log(prob + 1e-9) - (1 - label) * np.log(1 - prob + 1e-9)<br><br>        error = prob - label<br>        grad = error * x<br><br>        self.state.grad_memory = self.beta * self.state.grad_memory + (1 - self.beta) * grad<br>        self.state.fast_weights = self.state.fast_weights - self.eta_fast * self.state.grad_memory<br><br>        self.step_count += 1<br>        committed = False<br>        if self.step_count % self.slow_update_interval == 0:<br>            self.offline_update()<br>            committed = True<br><br>        return {<br>            &quot;prob&quot;: np.array([prob]),<br>            &quot;loss&quot;: np.array([loss]),<br>            &quot;error&quot;: np.array([error]),<br>            &quot;grad&quot;: grad,<br>            &quot;grad_memory&quot;: self.state.grad_memory.copy(),<br>            &quot;fast_weights&quot;: self.state.fast_weights.copy(),<br>            &quot;slow_weights&quot;: self.state.slow_weights.copy(),<br>            &quot;attention_memory&quot;: self.memory.copy(),<br>            &quot;attention_fast_scalar&quot;: np.array([attention[&quot;fast_scalar&quot;]]),<br>            &quot;committed&quot;: np.array([1.0 if committed else 0.0]),<br>        }<br><br>    def offline_update(self):<br>        &quot;&quot;&quot;<br>        Apply the correct offline slow update following Eq. (28)-(29).<br>        W_{t+1} = W_t (I - x x^T) - eta_slow * (u_t ⊗ x_t)<br>        &quot;&quot;&quot;<br>    <br>        # Pull slow weights into variable<br>        W = self.state.slow_weights<br>    <br>        # gradient memory stores the EMA of gradient:  g_t = error * x_t<br>        # This corresponds to ∇_y L ⊗ x_t  (outer product)<br>        grad_W = self.state.grad_memory            # shape: (input_dim,)<br>    <br>        # --- Build contraction matrix (I - x xᵀ) ---<br>        # Need the last input x_t used online; store x_t in state during online_step<br>        x_t = self.state.last_x                    # shape: (input_dim,)<br>        I = np.eye(self.input_dim)<br>        contraction = I - np.outer(x_t, x_t)<br>    <br>        # --- Apply Eq. (28)/(29) ---<br>        # W_{t+1} = W_t (I - x_t x_t^T) - η * (grad ⊗ x_t)<br>        # Since W is a row vector here, simplify the multiplication<br>        W_new = W @ contraction - self.eta_slow * grad_W<br>    <br>        # Update slow and fast weights (fast resets to slow snapshot)<br>        self.state.slow_weights = W_new<br>        self.state.fast_weights = W_new.copy()<br><br>    def online_train(self, interactions: Iterable[Tuple[int, int, float]]) -&gt; None:<br>        for user_id, item_id, label in interactions:<br>            self.online_step(user_id, item_id, label)</pre><h4>Why This Matters for Product Recommendation</h4><p>Product recommendation has several realities:</p><ul><li>Users behave differently across sessions</li><li>Their taste can shift in minutes</li><li>New items appear constantly</li><li>Traditional retraining cycles cannot keep up</li></ul><p>Nested Learning offers a practical recipe for hybrid online + offline learning:</p><ul><li>Fast weights: adapt immediately after every session engagement per user</li><li>Slow weights: consolidate knowledge across users and time</li><li>Momentum memory: smooths noise and prevents overfitting to single clicks</li><li>Attention memory: captures short-term context like “what the user is focusing on right now”</li></ul><p>This mirrors how humans learn: something surprising modifies short-term memory quickly, but only repeated exposure consolidates it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eef38209ace5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataai/nested-learning-for-recommender-systems-bringing-fast-and-slow-learning-to-personalization-eef38209ace5">Nested Learning for Recommender Systems: Bringing Fast and Slow Learning to Personalization</a> was originally published in <a href="https://medium.com/dataai">DataAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling data science workflows]]></title>
            <link>https://medium.com/dataai/scaling-data-science-workflows-d67e5a1de953?source=rss----9724fcf03e33---4</link>
            <guid isPermaLink="false">https://medium.com/p/d67e5a1de953</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Matt]]></dc:creator>
            <pubDate>Tue, 04 Nov 2025 22:17:07 GMT</pubDate>
            <atom:updated>2025-11-04T22:17:06.353Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SKLVAVB2KH6YygVw" /><figcaption>Generated by Google Gemini</figcaption></figure><p>Imagine you’re a data scientist or technical analyst that has a collection of reports that you run regularly. None of these reports are incredibly difficult to run but they all require some finesse. Essentially they require some form of manual input. Maybe that input is a date range, list of products, or list of locations. Wouldn’t it be nice to automate the collection of these inputs?</p><p>In this post I’ll walk through an LLM based approach and general framework you can use to automate these pipelines. It helps to walk through a tactical example so I’ll look at a simple example where I want to have an automated process to generate behavior based audiences at scale.</p><h3>Process overview</h3><p>To incorporate this process you will need to have key pieces in place. Below is a diagram outlining each necessary component to the proposed workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Tb7rC-9083YuLrGOyPiNg.png" /></figure><p>Keep in mind that this proposal is intended to be a bare essentials process. This is a framework that you can add to or take away from at your leisure. For example, I’m only recommending a single LLM model run, but you could just as easily swap this out with a more agentic approach where you run multiple LLM’s simultaneously to get multiple recommendations that could then be evaluated by another agent. For this purpose though a lot of those complexities will just muddy the key points we need to review.</p><h3>User intake</h3><p>This should be as low-touch as possible for you. Ideally, you have an established intake process that non-technical stakeholders can access. An example, would be a simple Slack bot that you set up that will let your stakeholders input custom text for whatever audience they want. This allows your stakeholders to explain in detail what they are looking to accomplish with the audience they need.</p><p>Once you have your Slack bot stood up, you will need to have that output into a table in your respective data lake. As part of this process you will want to have clearly defined metadata as well for this particular request. For example, you would want to ensure that the Slack bot would output the following features:</p><ul><li>Request ID (possibly linked to a work ticket in Jira or other ticket management software)</li><li>Urgency</li><li>User description (will be passed to the LLM)</li></ul><h3>LLM model</h3><p>During this phase you will dig deep in tuning your model. Ideally, you have a backlog of requests you could use to manually test your prompt.</p><p>As you build your prompt you will want to keep the following in mind:</p><ul><li>Clearly define the structure of your inputs</li><li>Specify that your outputs should ONLY be in a table structure (or other data structure that you plan to ingest in workflows later on)</li><li>Clearly define all data requirements you need to use in your downstream models (e.g. date ranges, etc.)</li><li>Defined lists of models that the model can choose from</li></ul><p>Here is a sample prompt you can use as a starting point. Note that this prompt needs further refinement to restrict output to a single usable table format with variables that need to be evaluated.</p><blockquote>You are a senior level data science practitioner. Your purpose is to take a custom explanation of audience profiles as described by your stakeholders and translate those descriptions into filtering and model recommendations for defining the relevant audiences. The output you define must consist of exactly one model recommendation and your confidence, between 0 and 1, in your model choice. You must also provide any recommendations for minimum or maximum spend and visit thresholds. You must also generate the ideal look back window for your recommendation.</blockquote><h3>Model recommendation</h3><p>In this phase you would have a workflow that regularly picks up new audience requests in the data lake. Then you would iteratively loop through all new audience requests against the LLM you defined above. The goal here is to generate a simple table or series of tables that can again be picked up by other downstream processes.</p><p>At the end of this phase you will have tables with clear thresholds for features needed to support your model along with a specific model recommendation. For example, you would have clearly defined date ranges or geographic locations you want to use in defining your audiences. Remember that you will have defined the key features you need to pass into the assortment of audience models you can choose from.</p><h3>Models</h3><p>At this point you should have an output table with the relevant request ID’s and other model recommendations. Within your data platform you can then set up specific jobs that begin executing after your model recommendations have been made. The audience segmentation process can then be executed as individual workflows for each model type you have available or as a single, broad workflow through a directed acyclic graph (DAG).</p><p>The important piece here is that you will have a process that takes the output from the LLM and that will be used to define features and criteria that are needed for your audience segmentation frameworks. You can then add new methods for segmenting audiences over time as needed.</p><h3>Reporting layer</h3><p>The reporting layer is critical to the success of this automation process. This is where we introduce a human-in-the-loop process to evaluate the outputs of the LLM recommendation and assess whether or not the audience that was generated is tactically relevant. One consideration here is that you will want a low touch process that can be used to re-evaluate the initial model through the above process again.</p><p>Additionally, this is another space where you can introduce agentic frameworks to evaluate the outputs and provide recommendations on next steps. This isn’t recommended though for initial deployments until you have refined the above framework and have a good understanding of where gaps may exist in your framework.</p><h3>Application</h3><p>In Google’s Gemini I tested a couple of sample audiences that my hypothetical stakeholders may request. It is important to note that these examples are purely hypothetical and don’t actually reflect any segmentations I have created. Also, I used the prompt that I shared above to create these model recommendations.</p><h4>Example #1</h4><blockquote>I need an audience that can be used to target ads for recent Nike brand shoppers at my company.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*7hMsqcCw_5xbF2RufSildw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*yliyXJ2tNnMxYEKNbqqFOA.png" /></figure><h4>Example #2</h4><blockquote>I need to create an audience that identifies loyal shoppers of Pepsi that have a high likelihood of switching to coke if presented with an ad.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*G8fkWW_bVmE5rsnlEgEopg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*7cp-TXye7ThSeJrdu8XGZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*JM1naOqjsBD27njTC5rPTg.png" /></figure><h3>Summary</h3><p>While there’s still plenty of tuning that is need to support this particular hypothetical example of audiences it should be clear that there is a lot of merit to this particular approach. This approach works for other processes as well especially when you simply need to parameters for analysis but your code can’t scale due to size but automating the requests can.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d67e5a1de953" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataai/scaling-data-science-workflows-d67e5a1de953">Scaling data science workflows</a> was originally published in <a href="https://medium.com/dataai">DataAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting What Matters: A Field Report from the VN2 Inventory Challenge with TimesFM 2.5]]></title>
            <link>https://medium.com/dataai/forecasting-what-matters-a-field-report-from-the-vn2-inventory-challenge-with-timesfm-2-5-4743e652e48d?source=rss----9724fcf03e33---4</link>
            <guid isPermaLink="false">https://medium.com/p/4743e652e48d</guid>
            <dc:creator><![CDATA[Philippe Dagher]]></dc:creator>
            <pubDate>Sun, 05 Oct 2025 11:56:28 GMT</pubDate>
            <atom:updated>2025-10-05T11:56:28.280Z</atom:updated>
            <content:encoded><![CDATA[<h3>Forecasting What Matters: A Field Report from the VN2 Inventory Challenge with TimesFM 2.5 (+ Covariates)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KEGWXAl2-dA1kqTRgEbqUA.png" /></figure><h3><em>How a zero-/few-shot time-series foundation model handled a messy, intermittent retail dataset — and what changed when we added covariates.</em></h3><h3>Why this project</h3><p>VN2 is a practical inventory planning challenge: you’re asked to forecast short horizons and make ordering decisions that balance shortage vs holding costs across store × product SKUs. Real data, real constraints, intermittent demand, and a small decision window — exactly the kind of environment where model bias and calibration hurt more than leaderboard points.</p><p>I wanted to test a modern foundation model — <strong>TimesFM-2.5</strong> — in this setting. TimesFM is a decoder-only, pre-trained forecasting model from Google Research designed for zero-shot/few-shot use. Version 2.5 offers longer context and native probabilistic outputs (quantile head) with a compact 200M checkpoint on Hugging Face.</p><p>But one practical question matters on VN2: <strong>do covariates help?</strong> Earlier guidance and implementations around TimesFM highlighted support for external covariates in workflows and examples; in retail, they often stabilize bias and reduce avoidable error.</p><h3>What I actually built</h3><p><strong>Baseline:</strong> <strong>TimesFM-2.5</strong> zero-shot, no covariates.</p><p><strong>Enhanced:</strong> <strong>TimesFM-2.5 + covariates</strong>, via an <strong>in-context regression (xreg)</strong> layer that encodes static and dynamic features and linearly adjusts the foundation model’s predictions.</p><p><strong>Traditional &amp; ML baselines:</strong></p><ul><li><strong>Seasonal Moving Average (8-week)</strong> — the challenge’s simple seasonal benchmark.</li><li><strong>Model-routed SES/Croston/TBATS</strong> — per-SKU routing based on intermittency/seasonality, with safe fallbacks.</li><li><strong>LightGBM (direct multi-horizon)</strong> — engineered lags/rollings/trend + calendar/categorical interactions.</li><li><strong>ARIMAX (per-SKU)</strong> — with numeric exogenous features for a quick classical reference.</li></ul><blockquote><em>Code and notebooks: </em><a href="https://github.com/senoni-research/timesfm/tree/master"><strong><em>github.com/senoni-research/timesfm</em></strong></a><em> (my fork) and </em><a href="https://github.com/senoni-research/timesfm/tree/master/notebooks"><strong><em>/notebooks</em></strong></a><em> folder for full VN2 runs.</em></blockquote><ul><li>Core xreg: <a href="https://github.com/senoni-research/timesfm/blob/master/src/timesfm/xreg_lib.py">src/timesfm/xreg_lib.py</a></li><li>Integration into 2.5: <a href="https://github.com/senoni-research/timesfm/blob/master/src/timesfm/timesfm_2p5/timesfm_2p5_torch.py">src/timesfm/timesfm_2p5/timesfm_2p5_base.py</a></li><li>VN2 notebook: <a href="https://github.com/senoni-research/timesfm/blob/master/notebooks/vn2_inventory_forecast.ipynb">notebooks/vn2_inventory_forecast.ipynb</a></li></ul><p>Official TimesFM repo and the 2.5 checkpoint for reference: <a href="https://github.com/google-research/timesfm">https://github.com/google-research/timesfm</a></p><h3>Porting covariates from “v1-style” to TimesFM-2.5</h3><p>Google’s materials and community tutorials around TimesFM framed <strong>external covariates</strong> as a useful part of real-world time-series workflows. To run that pattern with <strong>2.5</strong>, I implemented a lightweight <strong>xreg layer</strong> that sits “around” the foundation model:</p><blockquote><strong>Encode covariates</strong></blockquote><ul><li><em>Static categorical:</em> Store, ProductGroup, Department → one-hot (drop-first, unknown-safe)</li><li><em>Static numeric:</em> historical mean demand, coefficient of variation</li><li><em>Dynamic categorical:</em> Month, WeekOfYear, Quarter (extended into the horizon)</li><li><em>Dynamic numeric:</em> WeekIndex (time counter, extended into the horizon)</li></ul><p><strong>Fit ridge regression on the training window</strong> to learn a calibration that maps covariates to the target (or residuals), with per-input normalization.</p><p><strong>Blend with TimesFM predictions</strong> using “<strong>xreg + timesfm</strong>”: fit on the target, then predict residuals and add them back to TimesFM’s output. (I also tested the reverse order; results were similar on this dataset but “xreg + timesfm” reduced bias more consistently.)</p><p>This is intentionally simple: transparent, debuggable, and robust to category drift — useful properties when SKUs come and go.</p><h3>Dataset &amp; split (VN2)</h3><ul><li><strong>Scope:</strong> all <strong>599</strong> SKUs (Store × Product) in the VN2 challenge.</li><li><strong>Frequency:</strong> weekly; <strong>157</strong> weeks of history.</li><li><strong>Horizon:</strong> <strong>3 weeks</strong> ahead (to match a short ordering cycle).</li><li><strong>Validation:</strong> holdout on the last 16 weeks; context window = 140 weeks.</li></ul><p>The evaluation emphasizes <strong>WMAPE</strong> (weighted by actuals, the metric that reflects portfolio-level impact), plus <strong>MAE</strong>, <strong>RMSE</strong>, and <strong>BIAS</strong> (signed error).</p><h3>Results on all SKUs</h3><p><strong>Primary takeaways (portfolio-level):</strong></p><ul><li><strong>TimesFM-2.5 + covariates</strong> delivered the <strong>best WMAPE, MAE, and RMSE</strong> across all approaches, with a notably lower (closer-to-zero) <strong>BIAS</strong> than TimesFM alone.</li><li><strong>Seasonal MA</strong> retained the flattest BIAS (closest to zero) among simple baselines — useful as a calibration yardstick.</li><li><strong>LightGBM (direct multi-horizon)</strong> was competitive but did not surpass the foundation-model-plus-xreg stack on this split.</li></ul><p><strong>Aggregate metrics (all 599 SKUs, 3-week horizon):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NLtlookluFk1J-OQ7BT7iA.png" /></figure><p>Two practical interpretations:</p><ul><li><strong>Bias &amp; inventory:</strong> on short horizons, bias hurts more than a slightly higher MAE. Covariates reduced TimesFM’s under-forecasting tendency here (−2.01 → −0.93).</li><li><strong>Robustness:</strong> TimesFM’s zero-shot baseline is already strong — adding covariates mainly <strong>stabilized</strong> and <strong>calibrated</strong> it, especially on low-volume and intermittent SKUs.</li></ul><blockquote><em>Notes for context: TimesFM is designed for zero-/few-shot use (longer context, decoder-only). The model and checkpoints are public; covariate usage patterns are discussed in community blogs and examples, though the integration you see here is my fork’s implementation.</em></blockquote><h3>What the covariates actually did</h3><p>When I looked SKU-by-SKU, three effects showed up repeatedly:</p><ol><li><strong>Seasonal anchoring via calendar features</strong> (Month/WeekOfYear/Quarter) trimmed predictable swings that a universal prior tends to smooth out.</li><li><strong>Store &amp; hierarchy terms</strong> (Store, ProductGroup, Department) narrowed systematic offsets for recurring local patterns (pricing, mix, shelf placement, local demand variance).</li><li><strong>A simple time index</strong> (WeekIndex) helped with slow drifts where a pure seasonal decomposition wasn’t enough.</li></ol><p>These are humble features — no promotions, no price, no weather — yet they were enough to <strong>reduce error and bias</strong> meaningfully across the portfolio.</p><h3>Where the baselines still shine</h3><ul><li><strong>Seasonal MA</strong> was best on bias. If your downstream cost function is highly asymmetric or biased toward stockout protection, a seasonal baseline remains a good safety rail.</li><li><strong>Routed SES/Croston/TBATS</strong> helped for very intermittent SKUs. If you operationalize this, keep the routing logic simple and add aggressive fallbacks.</li><li><strong>LightGBM</strong> benefited from <strong>direct multi-horizon</strong> training (no recursive leakage) and horizon-safe lags/rollings. With deeper feature sets (e.g., price/promo/stockouts) it could close the gap.</li></ul><h3>Limitations &amp; how I’d extend this</h3><ul><li><strong>Features are thin by design.</strong> If you have <strong>promotions, price, marketing, holiday calendars, stockouts</strong>, or <strong>weather</strong>, add them. The xreg layer is ready to encode/extend them into the horizon.</li><li><strong>No hyperparameter search.</strong> I used conservative defaults for LightGBM/ARIMAX and a small ridge in xreg. There’s easy headroom with CV and per-segment tuning.</li><li><strong>Cost, not just error.</strong> VN2’s goal is cost. Next step: feed forecasts and uncertainties into a cost-aware base-stock or service-level policy and measure realized holding/shortage trade-offs.</li></ul><h3>Reproducibility</h3><ul><li><strong>Notebook:</strong> <a href="https://github.com/senoni-research/timesfm/blob/master/notebooks/vn2_inventory_forecast.ipynb">notebooks/vn2_inventory_forecast.ipynb</a> (runs on CPU; my run used PyTorch 2.8 without CUDA).</li><li><strong>Data layout:</strong> place VN2 CSVs under ../vn2inventory/data/ as referenced in the notebook.</li><li><strong>Model:</strong> google/timesfm-2.5-200m-pytorch from Hugging Face.</li><li><strong>Official TimesFM repo</strong> for background and alternatives (Vertex AI, BigQuery): <a href="https://github.com/google-research/timesfm">https://github.com/google-research/timesfm</a></li><li><strong>Challenge description &amp; rules</strong> (if you want to align to the competition setup): <a href="https://www.datasource.ai/en/users/philippe-dagher/competitions/vn2-inventory-planning-challenge/datathon_detail/description">https://www.datasource.ai/en/users/philippe-dagher/competitions/vn2-inventory-planning-challenge/datathon_detail/description</a></li></ul><h3>Why this mattered to me</h3><p>I didn’t set out to “win” on a narrow metric. I wanted a <strong>transparent workflow</strong> that a planner can trust:</p><ul><li>Start from a competent <strong>foundation prior</strong> (TimesFM-2.5).</li><li>Add <strong>small, intelligible covariates</strong> that match how the business thinks (season, store, category).</li><li>Keep <strong>bias near zero</strong> to avoid silent failure modes in inventory decisions.</li><li>Keep the system <strong>simple enough to maintain</strong>, while staying open to richer features later.</li></ul><p>It’s not glamorous, but it’s a fast path from <em>model</em> to <em>operationally useful forecast</em>.</p><h3>Acknowledge the broader context</h3><p>Time-series foundation models are moving quickly (longer contexts, better probabilistic heads, few-shot adapters), and research on <strong>covariates with zero-/few-shot models</strong> is accelerating too. The path I used — an outer xreg layer — is one pragmatic answer among several emerging techniques. <a href="https://research.google/blog/time-series-foundation-models-can-be-few-shot-learners/">https://research.google/blog/time-series-foundation-models-can-be-few-shot-learners/</a></p><h3>Closing thought</h3><p>Inventory planning rewards <strong>clarity</strong> more than cleverness. On VN2, a compact foundation model plus a small covariate layer produced forecasts that were <strong>less biased, more stable, and measurably more accurate</strong> — without exotic engineering. If you’re carrying a similar portfolio, start simple: get a strong prior, add the covariates you can trust, and make bias everyone’s business.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4743e652e48d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataai/forecasting-what-matters-a-field-report-from-the-vn2-inventory-challenge-with-timesfm-2-5-4743e652e48d">Forecasting What Matters: A Field Report from the VN2 Inventory Challenge with TimesFM 2.5</a> was originally published in <a href="https://medium.com/dataai">DataAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Your AI Should Say “I Don’t Know”: Lessons from GPT-5’s Attribute Extraction]]></title>
            <link>https://medium.com/dataai/when-your-ai-should-say-i-dont-know-lessons-from-gpt-5-s-attribute-extraction-c56928bc7249?source=rss----9724fcf03e33---4</link>
            <guid isPermaLink="false">https://medium.com/p/c56928bc7249</guid>
            <dc:creator><![CDATA[Suneel Tewari]]></dc:creator>
            <pubDate>Thu, 02 Oct 2025 20:24:47 GMT</pubDate>
            <atom:updated>2025-10-02T20:24:43.774Z</atom:updated>
            <content:encoded><![CDATA[<p><em>In the race to adopt the latest AI models, it’s tempting to assume newer means better across the board. But when we tested GPT-5 against GPT-4o for product attribute extraction, we discovered something counter-intuitive: the newer model’s willingness to say “Not Specified” instead of guessing made it produce “worse” metrics while actually giving better results.</em></p><h4>The Setup: Attribute Extraction at Scale</h4><p>Imagine you need to extract structured attributes from messy product descriptions — things like flavor profiles, dietary classifications, and product categories. You’ve got two models to choose from:</p><ul><li><strong>GPT-4o</strong>: The reliable workhorse you’ve been using</li><li><strong>GPT-5</strong>: The shiny new model with “improved reasoning”</li></ul><p>Which do you pick? The answer isn’t as straightforward as you might think.</p><h3>What We Tested</h3><p>We ran both models through a typical attribute extraction pipeline:</p><pre>Product: &quot;Fage Total Plain Greek Yogurt&quot;<br>Attributes to predict values for:<br>- Flavor<br>- Yogurt Source<br>- Yogurt Type<br>- ...</pre><p>The task seems simple enough — until you realize the ambiguity lurking in real-world data.</p><h3>The Paradox: When Lower Precision Means Higher Quality</h3><p>Here’s where things get interesting. Our initial metrics looked damning for GPT-5:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vQHblWtM46_bplZT67EdPw.png" /><figcaption>*Cost and speed measured for our test batch of 43 products with 9 attributes each</figcaption></figure><p>Case closed, right? GPT-4o wins on almost every metric. Not so fast.</p><h3>The “Not Specified” Revolution</h3><p>When we dug into the actual predictions, we discovered GPT-5 was making a fundamentally different type of decision. Across multiple attribute categories, we saw a consistent pattern: where GPT-4o would make inferential leaps to force classifications, GPT-5 would return “Not Specified” when faced with genuine ambiguity.</p><p>This wasn’t random — it was systematic. GPT-5 consistently refused to guess when products lacked explicit markers or claims. It required stronger evidence before making categorical assignments.</p><p>Here’s the catch: our golden dataset was created using GPT-4o’s historical outputs. When GPT-4o made those inferential leaps, they became the “correct” answers in our evaluation set. When GPT-5 more conservatively returned “Not Specified,” it was marked wrong — hence the lower precision score (65% vs 76%).</p><p>GPT-5 wasn’t performing worse — it was refusing to hallucinate. This is the difference between a model that always gives you an answer and one that knows when it doesn’t know.</p><h3>The Technical Deep Dive: Understanding Model Behavior</h3><h3>Token Economics and Reasoning Chains</h3><pre># Simplified token analysis<br>gpt4o_tokens = {<br>    &quot;prompt&quot;: 110693,<br>    &quot;completion&quot;: 20212,  # Concise, decisive<br>    &quot;total_cost&quot;: 0.48<br>}<br><br>gpt5_tokens = {<br>    &quot;prompt&quot;: 110688,<br>    &quot;completion&quot;: 45478,  # 2.25x more output<br>    &quot;total_cost&quot;: 0.59<br>}</pre><p>GPT-5 generated over twice as many completion tokens per prediction. Since we used identical prompts and didn’t request additional reasoning chains, this suggests GPT-5’s internal processing generates more intermediate tokens before arriving at its final answer — even when not explicitly asked to “think step by step.”</p><p>Without access to the raw outputs, we can’t definitively say whether these extra tokens represent reasoning steps, metadata, or formatting differences. What we do know is that this token increase correlated with more nuanced decisions (choosing “Not Specified” over forced classifications).</p><p>The key insight isn’t about reasoning depth — it’s about decision-making behavior: GPT-5 was more willing to acknowledge uncertainty rather than force a classification. Whether the extra tokens directly enable this better judgment or are just coincidental overhead, the practical reality is the same: better uncertainty handling comes with a 2.25x token cost.</p><h3>The Latency Trade-off</h3><p>Yes, GPT-5 is slower — roughly 4.4x slower in our tests:</p><pre>def calculate_processing_time(num_items):<br>    gpt4o_time = num_items * 8.4  # seconds<br>    gpt5_time = num_items * 37.0   # seconds<br>    return {<br>        &quot;gpt4o_hours&quot;: gpt4o_time / 3600,<br>        &quot;gpt5_hours&quot;: gpt5_time / 3600<br>    }<br><br># For 2000 items:<br># GPT-4o: ~4.7 hours<br># GPT-5: ~20.5 hours</pre><p>But here’s the key insight: this latency is almost entirely parallelizable. With proper orchestration, you can achieve similar wall-clock times while benefiting from superior reasoning.</p><h3>The Surprising Discovery: Self-Evaluation Equivalence</h3><p>When we tested both models’ ability to self-evaluate their predictions (assigning confidence scores), we found both models produced <strong>identical</strong> confidence scores across all 387 predictions, but GPT-4o-mini cost 10x less. The lesson? Not every task benefits from the most advanced model.</p><pre># Confidence scoring comparison<br>for product in test_set:<br>    gpt4o_mini_confidence = evaluate(product, model=&quot;gpt-4o-mini&quot;)<br>    gpt5_confidence = evaluate(product, model=&quot;gpt-5&quot;)<br>    <br>    assert gpt4o_mini_confidence == gpt5_confidence  # Always true!</pre><h3>Practical Implementation: The Hybrid Approach</h3><p>Based on our findings, here’s the optimal configuration:</p><pre>class OptimalAttributeExtractor:<br>    def __init__(self):<br>        self.inference_model = &quot;gpt-5&quot;  # Better reasoning<br>        self.evaluation_model = &quot;gpt-4o-mini&quot;  # Same quality, lower cost<br>    <br>    def extract(self, product_description):<br>        # Use GPT-5 for extraction when accuracy matters<br>        attributes = self.inference_model.predict(product_description)<br>        <br>        # Use GPT-4o-mini for confidence scoring<br>        confidence = self.evaluation_model.evaluate(attributes)<br>        <br>        return attributes, confidence</pre><h3>When to Use Each Model</h3><p><strong>Use GPT-5 for inference when:</strong></p><ul><li>Ambiguity handling is critical</li><li>False positives are costlier than “unknown” results</li><li>You need explainable reasoning chains</li><li>The 23% cost increase is justified by reduced manual review</li></ul><p><strong>Stick with GPT-4o when:</strong></p><ul><li>You need forced classification (no “Not Specified” allowed)</li><li>Speed is critical and parallelization isn’t feasible</li><li>Your golden dataset is optimized for GPT-4o patterns</li><li>You’re doing self-evaluation tasks</li></ul><p>These trade-offs align with OpenAI’s recent research on <a href="https://openai.com/index/why-language-models-hallucinate/">why language models hallucinate</a>, which shows that current evaluation benchmarks actually penalize models for saying “I don’t know” — awarding zero points for uncertainty just like wrong answers. This creates a systemic bias toward guessing over honesty. In our case, the 23% cost premium for GPT-5’s uncertainty-aware responses was justified by the reduction in manual corrections downstream, though this trade-off will vary by use case.</p><h3>Key Takeaways</h3><ol><li><strong>Metrics can be misleading</strong>: Lower precision might indicate better contextual understanding, not worse performance.</li><li><strong>“Not Specified” is a feature, not a bug</strong>: A model that admits uncertainty is often more valuable than one that confidently hallucinates.</li><li><strong>Mix and match models</strong>: Use expensive models where they add value, cheaper ones where they don’t.</li><li><strong>Challenge your golden datasets</strong>: They might be encoding the biases of older models rather than ground truth.</li></ol><h3>What’s Next?</h3><p>The evolution from GPT-4o to GPT-5 represents a shift from “always give an answer” to “give the right answer or acknowledge uncertainty.” This philosophical change has profound implications for production systems.</p><p>As you evaluate new models for your pipelines, remember:</p><ul><li>Test beyond surface metrics</li><li>Examine actual predictions, not just scores</li><li>Consider the cost of false positives vs. unknowns</li><li>Design systems that leverage each model’s strengths</li></ul><p>The best system isn’t always the one with the newest model — it’s the one that uses the right model for each task.</p><p><em>Want to try this comparison yourself? The approach generalizes to any attribute extraction task. Start with a small test set, examine the actual predictions (not just metrics), and pay special attention to how each model handles ambiguous cases.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c56928bc7249" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataai/when-your-ai-should-say-i-dont-know-lessons-from-gpt-5-s-attribute-extraction-c56928bc7249">When Your AI Should Say “I Don’t Know”: Lessons from GPT-5’s Attribute Extraction</a> was originally published in <a href="https://medium.com/dataai">DataAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Over “Makeover Monday”]]></title>
            <link>https://medium.com/dataai/making-over-makeover-monday-e7d440813c18?source=rss----9724fcf03e33---4</link>
            <guid isPermaLink="false">https://medium.com/p/e7d440813c18</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[datafam]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[databricks-free-edition]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Feiza Data]]></dc:creator>
            <pubDate>Thu, 02 Oct 2025 20:21:47 GMT</pubDate>
            <atom:updated>2025-10-02T20:21:43.575Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tLTKmYbHcwzzJv162WZAlg.png" /><figcaption>#FeizaData Thumbnail.</figcaption></figure><p>If you’ve been around the DataFam for any length of time, you’ve probably heard of <a href="https://makeovermonday.co.uk/">Makeover Monday</a>. It’s a weekly challenge that provides a new dataset every week and invites you to reimagine the story it tells. For the last 2 years, I’ve used this community project as a commitment device to hold me accountable to practicing Tableau. Over time, those quick projects start to stack up into a portfolio of ideas I can share or revisit myself later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*Fis73Mdstq4hjcIeiwMpPQ.png" /><figcaption>Makeover Monday community project logo.</figcaption></figure><p>For a recent challenge (<a href="https://www.kaggle.com/datasets/sahilislam007/ai-impact-on-job-market-20242030?">2025 Week 37</a>), the dataset focused on AI’s impact on the job market. The topic felt immediately relevant, and I was excited to boot up Tableau and dig into the data.</p><p>That excitement didn’t last long, though. While initially exploring the records, a few things stuck out to me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*-HqGpwyRdCPpBNXjDBj8Ig.png" /><figcaption>Sample records from the Makeover Monday dataset.</figcaption></figure><ul><li>An economist with an associate’s degree earning $150K? In the retail industry? Apparently, I picked the wrong career path…</li><li>A dentist categorized in the entertainment industry? That sounds like an Impractical Jokers bit.</li><li>I’m still not sure what kind of fitness manager belongs in the transportation industry.</li></ul><p>The more I explored, the harder it was to trust any patterns I saw. I even went back to the Kaggle dataset to see what the author (and others) had to say.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DllS-AyJKms_pFFC7bxv9w.png" /><figcaption>Dataset author’s response to a user’s Kaggle comment.</figcaption></figure><p>…I think we might have a slightly different perspective on how far from “factual” a synthetic dataset should deviate. Nevertheless, I was stuck. I didn’t want to create and share something misleading, especially with a topic I felt passionate about.</p><p>What if I used this dataset as a starting point and rebuilt the fields myself? Not by hand, of course, but with help from our good pal AI.</p><h3>Rebuilding the Dataset with Databricks Community</h3><p>A few weeks ago, I heard some updates had been made to <a href="https://accounts.cloud.databricks.com/">Databricks’ Free Edition</a> and offered daily free compute to users. I knew they had some AI features baked into their ecosystem, and I’ve been wanting to give them a shot. This project became the perfect opportunity. Here’s how I got started:</p><h4>Importing the Dataset</h4><ul><li>On the left side of the Databricks workspace, go to Data Ingestion &gt; Create or modify table.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l-mPeM0v7MmmQsoWMx9ZEQ.png" /><figcaption>Navigating to the Databricks file upload page.</figcaption></figure><ul><li>Drag and drop your CSV (in this case, the Makeover Monday data). I created a schema called tableau and named my table ai_job_trends.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*krptNIYKbDBB-cggmnhQ5g.png" /><figcaption>Page to assign a schema, table name, and rename columns before creating the table.</figcaption></figure><blockquote>Column renaming is an option here, but I left that for later. The goal was just to get the data staged and ready.</blockquote><h4>Setting Up the Notebook</h4><p>I created a new notebook and ran my standard boilerplate imports.</p><pre># Set up Spark and Python Environment<br>from pyspark.sql.functions import *<br>from pyspark.sql.types import *<br>from pyspark.sql import Window<br>from datetime import date, datetime, timedelta</pre><p>For context, I tend to work more regularly in PySpark, so this blog post mixes PySpark and SQL quite a bit. It’s not the only way to transform your data, but this rhythm helps me move quickly.</p><p>To create the foundation for my new dataset, I first wanted to know how many job titles were actually in the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*07RcpOqqkzyktIzAcXk7ng.png" /><figcaption>Checking distinct job titles in the dataset.</figcaption></figure><p>After removing duplicates, I landed on 639 unique job titles (compared to the nearly 30,000 records in the original dataset). This single-column dataframe became the spine for my AI prompts to follow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7L7PySLP8A0SI-FuKBcJnw.png" /><figcaption>Creating a temporary view for SQL querying.</figcaption></figure><h4>Using AI to Rebuild the Metadata</h4><p>The goal was to generate accurate fields for each job title, including the correct industry, the required education level, and other context from the original dataset. I wanted to turn the simple list of jobs into a structured table that conceptually matched the original Kaggle data, but told the accurate story that we were missing.</p><p>With the isolated 639 job titles, it was time to explore Databricks’ <a href="https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query"><strong><em>ai_query()</em></strong></a> function. After a few iterations, I found an approach that would generate all the fields I wanted (i.e., industry, education, AI impact level) in a single column. The structured output would run quickly and yield data that I could easily split in a subsequent step.</p><p>The final prompt I landed on consists of a few key learnings and decisions I made along the way:</p><ul><li><strong>Use numbered/bulleted instructions</strong>: LLMs tend to perform better with structured steps, so I used them wherever possible.</li><li><strong>Provide strict formatting rules and examples</strong>: This reduced noise in the responses and ensured results were predictable and usable.</li><li><strong>Separate attributes with pipes (|)</strong>: Condensing all the attributes into a single pipe-delimited column significantly improved runtime compared to running a separate <strong><em>ai_query()</em></strong> for each attribute.</li><li><strong>Constrain industry categories</strong>: Confining “Industry” to a fixed set of eight kept categories consistent, but some jobs stretched those boundaries. There’s no perfect approach here, but I opted to stay close to the original dataset.</li></ul><p>Here’s the query (including the prompt) that I used to generate my new data:</p><pre>%sql<br>SELECT <br>    job, <br>    ai_query(<br>        &quot;databricks-meta-llama-3-3-70b-instruct&quot;,<br>        CONCAT(<br>            &quot;You are a structured data generator.<br>            Given a job title in the United States, return exactly 6 attributes separated by the pipe character (|). <br>            The output must contain no explanations, labels, or extra words — only the 6 attributes in order. <br>            <br>            Attributes (always return in this order):<br>            1. Industry → choose ONE from {Retail, Education, Manufacturing, IT, Finance, Transportation, Entertainment, Healthcare}.<br>            2. Job Status → choose ONE from {Increasing, Decreasing}. Base this on whether demand is rising or falling due to AI expansion.<br>            3. AI Impact Level → choose ONE from {High, Medium, Low}. Reflect the degree of AI disruption on this role.<br>            4. Median Salary (numeric USD, no symbols or commas, e.g., 55000).<br>            5. Experience Required (integer years, e.g., 5).<br>            6. Required Education → choose ONE from {High School, Associate, Bachelor Degree, Master Degree, PhD}.<br><br>            Formatting rules:<br>            - Separate attributes ONLY with &#39;|&#39;.<br>            - Do NOT include units, words like &#39;USD&#39; or &#39;years&#39;.<br>            - Do NOT add any commentary, explanation, or text before/after the output.<br>            - Output must be exactly one line.<br><br>            Example valid output:<br>            Entertainment|Increasing|Medium|55000|5|Bachelor Degree<br><br>            Job Title: &quot;,<br>            CAST(`job` AS STRING)<br>        )<br>    ) AS ai_metadata<br>FROM jobs</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muVU4sZAtuXLyqwqFFIF4w.png" /><figcaption>Running the AI prompt query in Databricks and previewing output.</figcaption></figure><p>And the code I used to split the pipe-delimited output column into something easier to work with:</p><pre>improved_df = (<br>    _sqldf<br>    .withColumn(&#39;industry&#39;, split(col(&#39;ai_metadata&#39;), &#39;\|&#39;).getItem(0))<br>    .withColumn(&#39;job_status&#39;, split(col(&#39;ai_metadata&#39;), &#39;\|&#39;).getItem(1))<br>    .withColumn(&#39;ai_impact_level&#39;, split(col(&#39;ai_metadata&#39;), &#39;\|&#39;).getItem(2))<br>    .withColumn(&#39;median_salary&#39;, split(col(&#39;ai_metadata&#39;), &#39;\|&#39;).getItem(3))<br>    .withColumn(&#39;experience_required&#39;, split(col(&#39;ai_metadata&#39;), &#39;\|&#39;).getItem(4))<br>    .withColumn(&#39;required_education&#39;, split(col(&#39;ai_metadata&#39;), &#39;\|&#39;).getItem(5))<br>    .drop(&#39;ai_metadata&#39;)<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fWDDwcrLgbYFC6zeEdVqFw.png" /><figcaption>Resultant dataframe after splitting the metadata column.</figcaption></figure><p>This output was solid, but I didn’t want to stop here. How could I be so sure that these new records were accurate? I couldn’t trust the original synthetic data. Why should I blindly trust this newly generated table?</p><p>To strengthen confidence, I created a second <a href="https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query"><strong><em>ai_query()</em></strong></a> to serve as a validation agent. This step checked every row’s results and flagged any anomalies based on rules I outlined. There are a few things I found especially helpful when writing the validation prompt:</p><ul><li><strong>Forcing a simple output</strong>: Being explicit about returning exactly one character (“1” or “0”) ensured the output was easy to use.</li><li><strong>General validation rules</strong>: These caught obvious issues like invalid categories or bad data type formatting.</li><li><strong>Contextual Rules</strong>: Grounding the model with real-world expectations (e.g., a dentist belonging in Healthcare, not Entertainment) helped surface edge cases that might otherwise slip through (see Validation Rules #7 below).</li></ul><pre>%sql<br>SELECT<br>  ai_query(<br>    &quot;databricks-meta-llama-3-3-70b-instruct&quot;,<br>    CONCAT(<br>      &quot;You are a validator for structured US job metadata. Input is one record with fields in this order, pipe-delimited: <br>      <br>      Job Title | Industry | Job Status | AI Impact Level | Median Salary | Experience Years | Required Education. <br>      <br>      Return exactly one character: &#39;1&#39; if the record is plausible and consistent, or &#39;0&#39; if it is not. No explanations. No spaces. No extra text. One character only.<br>      <br>      Validation rules (if any fail, return &#39;0&#39;):<br>        1) Industry must be in Retail, Education, Manufacturing, IT, Finance, Transportation, Entertainment, or Healthcare. <br>        2) Job Status must be in Increasing or Decreasing. <br>        3) AI Impact Level must be in High, Medium, or Low. <br>        4) Median Salary must be an integer only and directionally plausible for the job. <br>        5) Experience Years must be an integer only.<br>        6) Required Education must be in High School, Associate, Bachelor Degree, Master Degree, or PhD. <br>        7) Context checks: <br>          a) Education cannot undershoot obvious professional requirements. Examples: <br>            - Dentist, Physician, Pharmacist, Psychologist require at least Master Degree or PhD. <br>            - Lawyer requires at least Bachelor Degree. <br>            - Registered Nurse requires at least Associate. <br>            - Engineer, Accountant, Auditor require at least Bachelor Degree. <br>          b) Industry must be a reasonable fit for the job. Example: Dentist must be in Healthcare, not Entertainment. <br>          c) Salary must be plausible for the seniority and education level. Example: Economist with Associate and salary over 100000 is implausible. <br>          d) If Required Education is Associate and Median Salary is greater than 100000, return &#39;0&#39; unless the job is a known high-wage exception such as Pilot, Air Traffic Controller, Elevator Mechanic, Power Lineworker, or Radiation Therapist. <br>          e) Senior titles (Senior, Lead, Principal, Director, VP, Chief) normally require at least 3 years of experience. → Junior or Intern titles normally require 0 to 2 years of experience and salaries should not be executive-level. <br>          f) No field may be empty or null. <br>          g) If the job title is extremely ambiguous or contradicts other fields, return &#39;0&#39;. → If all rules pass, return &#39;1&#39;. <br>          <br>      Record: &quot;,<br>      CAST(job AS STRING), &quot;|&quot;,<br>      CAST(industry AS STRING), &quot;|&quot;,<br>      CAST(job_status AS STRING), &quot;|&quot;,<br>      CAST(ai_impact_level AS STRING), &quot;|&quot;,<br>      CAST(median_salary AS STRING), &quot;|&quot;,<br>      CAST(experience_required AS STRING), &quot;|&quot;,<br>      CAST(required_education AS STRING)<br>    )<br>  ) AS qa_flag,<br>  *<br>FROM improved_df</pre><blockquote>It’s worth noting that I iterated quite a bit on the two prompts above. I actually asked ChatGPT to act as an expert prompt engineer, reviewing my “junior” prompts and suggesting improvements. That feedback loop made these prompts significantly more robust.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F8KYG5jjRFxazSdtOydacg.png" /><figcaption>Validation prompt and preview of output.</figcaption></figure><p>The outcome of this QA prompt: 94% of the rows passed our test. It’s not a perfect 100%, but from a practical standpoint, I wouldn’t trust a human to be 100% accurate in labeling hundreds of jobs either.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*o1-P6Qgs2S-oz4702LiEDQ.png" /><figcaption>Of the 639 job titles, our second agent labelled 94% as fully accurate.</figcaption></figure><p>I took the ~5% of jobs that failed our QA test and filtered them out in Tableau. The remaining job records felt much more trustworthy and ready to be visualized.</p><p>I aggregated the jobs by industry and AI impact level (low/medium/high). Now I had a clear insight that felt grounded, rather than questionable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s4gpt6lriE5HOef2f8EHUg.png" /><figcaption><a href="https://public.tableau.com/app/profile/blake.feiza/viz/AIImpactonDifferentIndustriesMakeoverMonday/MakeoverMonday">Final Visualization Shared on Tableau Public</a></figcaption></figure><h4>Wrapping Up</h4><p>This Makeover Monday dataset wasn’t what I expected, but I’m happy that it pushed me into a new workflow. <a href="https://community.databricks.com/">Databricks Community</a> offered me a way to both reassemble a messy dataset and rethink my approach to validation.</p><p>There are plenty of ways to iterate and refine this process, but I found it encouraging that AI can serve as a builder and validator inside a single notebook. I’d love to hear how you would push this further. What would you try? What would you change?</p><p>Thank you for reading, I’ll see you back here next month with another project!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7d440813c18" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataai/making-over-makeover-monday-e7d440813c18">Making Over “Makeover Monday”</a> was originally published in <a href="https://medium.com/dataai">DataAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>