<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nathansundararajan on Medium]]></title>
        <description><![CDATA[Stories by Nathansundararajan on Medium]]></description>
        <link>https://medium.com/@nathansundararajan?source=rss-ea0ea6c0f203------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3rijJiUbcDNYw9t4XtAXcA@2x.jpeg</url>
            <title>Stories by Nathansundararajan on Medium</title>
            <link>https://medium.com/@nathansundararajan?source=rss-ea0ea6c0f203------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 14 Nov 2026 02:13:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nathansundararajan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Part2 : Lakeflow Declarative pipeline Events..|(When]]></title>
            <link>https://medium.com/@nathansundararajan/part2-lakeflow-declarative-pipeline-events-when-f8ba40dc58a4?source=rss-ea0ea6c0f203------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8ba40dc58a4</guid>
            <category><![CDATA[lakeflow-designer]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Nathansundararajan]]></dc:creator>
            <pubDate>Sat, 03 Oct 2026 22:55:25 GMT</pubDate>
            <atom:updated>2026-10-03T22:55:25.099Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part2 : Lakeflow Declarative pipeline Events..|(When a Databricks Pipeline Fails, Retries — or Just Stops Showing Up)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-YlIt2Xmq9nC00SR.png" /></figure><p>Part 1 covered system.lakeflow_pipeline_events_preview.pipeline_events, the recently announced beta table, and the traps to watch for when querying it.</p><p>This part is about what to build once you trust the data.</p><p>I started with a straightforward question: which pipelines are producing the most errors? That led to repeated failures across flows, a dev pipeline spending money while failing, and pipelines that had stopped running without raising an error.</p><p>That last one is easy to miss. If nothing runs, nothing fails. Your error dashboard can look perfectly calm.</p><p>One reminder before the SQL: <strong>apply the unique-event-ID filtering from Part 1 before trusting these counts.</strong> The beta deduplication caveat applies throughout this post.</p><h3>Start with the fleet error leaderboard</h3><p>Finding the noisiest pipelines used to mean opening individual logs and piecing the results together. Now there is a single place to start:</p><pre>SELECT<br>    pipeline_id,<br>    origin.pipeline_name AS pipeline_name,<br>    COUNT(*) AS error_count,<br>    MIN(event_time) AS first_occurrence,<br>    MAX(event_time) AS last_occurrence<br>FROM system.lakeflow_pipeline_events_preview.pipeline_events<br>WHERE level = &#39;ERROR&#39;<br>  AND event_time &gt;= current_timestamp() - INTERVAL 7 DAYS<br>GROUP BY pipeline_id, origin.pipeline_name<br>ORDER BY error_count DESC, last_occurrence DESC<br>LIMIT 20;</pre><p>A few rows from my workspace:</p><pre>pipeline_id  pipeline_name      error_count  first_occurrence      last_occurrence<br>pipeline_a   [dev] pipeline-a   255          2026-09-09 00:16:02   2026-09-15 16:20:11<br>pipeline_b   [prod] pipeline-b  96           2026-09-09 00:13:09   2026-09-14 14:25:41<br>pipeline_c   [dev] pipeline-c   50           2026-09-10 15:39:43   2026-09-14 05:20:15</pre><p>Some ERROR rows have no populated origin.pipeline_name. That currently looks like a platform bug, although I have not confirmed the cause. If you see it, a specific event ID and pipeline ID would make a useful report to Databricks.</p><p>The missing name does not stop you from identifying the pipeline. You still have pipeline_id.</p><h3>Then look at what keeps failing</h3><p>The leaderboard tells you where to look. Grouping by message tells you what keeps happening.</p><pre>SELECT<br>    message,<br>    COUNT(*) AS error_count,<br>    COUNT(<br>        DISTINCT COALESCE(<br>            origin.flow_id,<br>            NULLIF(regexp_extract(message, &quot;&#39;([^&#39;]+)&#39;&quot;, 1), &#39;&#39;)<br>        )<br>    ) AS distinct_flows_affected<br>FROM system.lakeflow_pipeline_events_preview.pipeline_events<br>WHERE level = &#39;ERROR&#39;<br>  AND event_time &gt;= current_timestamp() - INTERVAL 7 DAYS<br>GROUP BY message<br>ORDER BY error_count DESC<br>LIMIT 20;</pre><p>For example:</p><pre>message                                           error_count  distinct_flows_affected<br>Failed to resolve flow: &#39;prod.pipeline_b.flow_1&#39;.  48           1<br>Flow &#39;dev.pipeline_a.flow_1&#39; has FAILED.            45           1<br>Flow &#39;dev.pipeline_a.flow_2&#39; has FAILED.            45           1</pre><p>origin.flow_id is frequently null on ERROR rows. This query falls back to the quoted name in message.</p><p>There is also a flow-identity detail worth keeping in mind throughout this table: within one pipeline, retain (flow_id, flow_name) together. A full refresh can change flow_id. When comparing flows across pipelines, include pipeline_id as well.</p><p>In my results, four flows in the same pipeline each had exactly 45 errors.</p><p>That was worth a closer look.</p><h3>Keep update-level history in the timeline table</h3><p>Before going further, there is no reason to rebuild everything from event logs.</p><p>system.lakeflow.pipeline_update_timeline already gives you update-level pass/fail history. The event table is useful when you need to look inside an update.</p><p>This query gives a starting point for that investigation:</p><pre>SELECT<br>    update_id,<br>    MIN(event_time) AS update_start,<br>    MAX(event_time) AS update_end,<br>    SUM(CASE WHEN level = &#39;ERROR&#39; THEN 1 ELSE 0 END) AS error_count<br>FROM system.lakeflow_pipeline_events_preview.pipeline_events<br>WHERE event_time &gt;= current_timestamp() - INTERVAL 7 DAYS<br>GROUP BY update_id<br>ORDER BY update_start DESC<br>LIMIT 20;</pre><p>Use the timeline table for “Did the update succeed?”</p><p>Use the event table for “Which flow had a problem, and what kind of error was it?”</p><h3>Put throughput and compute next to each other</h3><p>Flow metrics arrive at two levels. Output rows are on INFO events; executor time is on METRICS events.</p><p>You can bring both into the same query:</p><pre>SELECT<br>    origin.flow_id AS flow_id,<br>    origin.flow_name AS flow_name,<br>    SUM(<br>        CASE WHEN level = &#39;INFO&#39;<br>            THEN details:flow_progress:metrics:num_output_rows::BIGINT<br>        END<br>    ) AS output_rows,<br>    SUM(<br>        CASE WHEN level = &#39;METRICS&#39;<br>            THEN details:flow_progress:metrics:executor_time_ms::BIGINT<br>        END<br>    ) AS executor_time_ms,<br>    COUNT(*) AS events<br>FROM system.lakeflow_pipeline_events_preview.pipeline_events<br>WHERE event_type = &#39;flow_progress&#39;<br>  AND level IN (&#39;INFO&#39;, &#39;METRICS&#39;)<br>  AND event_time &gt;= current_timestamp() - INTERVAL 7 DAYS<br>GROUP BY origin.flow_id, origin.flow_name<br>ORDER BY executor_time_ms DESC<br>LIMIT 20;</pre><p>In my workspace, a <strong>dev bronze feed</strong> topped executor time, ahead of every production flow.</p><p>That is the sort of result I would probably have missed while checking pipelines one at a time.</p><p>Watch for pipelines.flowTimeMetrics.missingFlowName, though. It is a placeholder the platform emits when a METRICS row has no flow name attached. Filter it out or label it clearly. Otherwise, your “biggest flow” might be a bucket of unattributed metrics.</p><h3>The four flows with 45 errors</h3><p>Messages are useful for reading an incident, but they make awkward alert keys. Table names and versions can change the text even when the underlying error is the same.</p><p>error.exceptions[].error_class gives you a more consistent way to group failures:</p><pre>SELECT<br>    error.fatal AS fatal,<br>    exc.error_class,<br>    exc.sql_state,<br>    COUNT(*) AS error_count<br>FROM system.lakeflow_pipeline_events_preview.pipeline_events<br>LATERAL VIEW OUTER EXPLODE(error.exceptions) e AS exc<br>WHERE level = &#39;ERROR&#39;<br>  AND event_time &gt;= current_timestamp() - INTERVAL 7 DAYS<br>GROUP BY error.fatal, exc.error_class, exc.sql_state<br>ORDER BY error_count DESC;</pre><p>Back to those four flows.</p><p>They shared the same error class across the same 15 updates. Within a single update, all four failed within about 70 seconds of one another.</p><p>The 45 events per flow worked out to an average of three error events per update. Those events included repeated failures and summary rows, so <strong>45 error events should not be read as 45 separate attempts</strong>.</p><p>The useful finding was the shared pattern: one pipeline, one error class, 15 updates, and four flows failing together.</p><p>That gave me a reason to investigate one common cause instead of treating the flows as four unrelated incidents. I did not need to open the pipeline UI to see it.</p><p>For deeper investigation, Genie’s code-oriented skills — including the system-tables-specific skill — can help work through this kind of query chain without writing every join by hand.</p><h3>Look for retries, even without a retry field</h3><p>There is no dedicated retry field here. Repeated failure events within the same update and flow give you somewhere to start:</p><pre>SELECT<br>    pipeline_id,<br>    origin.pipeline_name AS pipeline_name,<br>    update_id,<br>    origin.flow_id AS flow_id,<br>    origin.flow_name AS flow_name,<br>    COUNT(*) AS failure_events,<br>    MIN(event_time) AS first_failure,<br>    MAX(event_time) AS last_failure<br>FROM system.lakeflow_pipeline_events_preview.pipeline_events<br>WHERE level = &#39;ERROR&#39;<br>  AND event_time &gt;= current_timestamp() - INTERVAL 7 DAYS<br>GROUP BY<br>    pipeline_id,<br>    origin.pipeline_name,<br>    update_id,<br>    origin.flow_id,<br>    origin.flow_name<br>HAVING COUNT(*) &gt; 1<br>ORDER BY failure_events DESC;</pre><p>Treat this as a list of candidates to inspect. Repeated errors can include summary events; the count alone is not an exact retry count.</p><p>A pipeline that succeeds only on its third attempt every night still deserves attention. An update-level pass/fail summary can hide that pattern. The events inside the update give you a chance to see it.</p><h3>Catalog and schema chargeback hits a gap</h3><p>I also tried grouping events by catalog and schema:</p><pre>SELECT<br>    origin.catalog_name,<br>    origin.schema_name,<br>    COUNT(*) AS event_count,<br>    SUM(CASE WHEN level = &#39;ERROR&#39; THEN 1 ELSE 0 END) AS error_count<br>FROM system.lakeflow_pipeline_events_preview.pipeline_events<br>WHERE event_time &gt;= current_timestamp() - INTERVAL 7 DAYS<br>GROUP BY origin.catalog_name, origin.schema_name<br>ORDER BY event_count DESC;</pre><p>The result was mostly empty.</p><p>Of roughly 652,000 rows, <strong>644,171 had no catalog or schema</strong>, including every ERROR row. origin.catalog_name and origin.schema_name populate only for specific dataset-scoped event types.</p><p>There is no clean fallback in the other tables used here. Neither system.lakeflow.pipelines nor pipeline_update_timeline carries a target catalog/schema.</p><p>For chargeback by domain, you currently need your own pipeline metadata, joined on pipeline_id.</p><h3>Join cost to reliability</h3><p>This was the query I most wanted to run.</p><p>Billing can tell you what a pipeline costs. Event logs can tell you how often it fails. Joining them lets you see both at once.</p><pre>WITH prices AS (<br>    SELECT<br>        sku_name,<br>        usage_unit,<br>        price_start_time,<br>        COALESCE(<br>            price_end_time,<br>            DATE_ADD(CURRENT_DATE(), 1)<br>        ) AS price_end_time,<br>        try_variant_get(<br>            to_variant_object(pricing),<br>            &#39;$.effective_list.default&#39;,<br>            &#39;decimal(38,18)&#39;<br>        ) AS unit_px<br>    FROM system.billing.list_prices<br>    WHERE currency_code = &#39;USD&#39;<br>      AND cloud = &#39;AZURE&#39;<br>),<br>pipeline_cost AS (<br>    SELECT<br>        p.pipeline_id,<br>        p.name AS pipeline_name,<br>        SUM(<br>            COALESCE(pr.unit_px * u.usage_quantity, 0)<br>        ) AS total_cost<br>    FROM system.billing.usage u<br>    LEFT JOIN prices pr<br>        ON u.sku_name = pr.sku_name<br>       AND u.usage_unit = pr.usage_unit<br>       AND u.usage_end_time &gt;= pr.price_start_time<br>       AND u.usage_end_time &lt; pr.price_end_time<br>    LEFT JOIN (<br>        SELECT<br>            pipeline_id,<br>            name,<br>            workspace_id<br>        FROM system.lakeflow.pipelines<br>        QUALIFY ROW_NUMBER() OVER (<br>            PARTITION BY pipeline_id<br>            ORDER BY change_time DESC<br>        ) = 1<br>    ) p<br>        ON u.usage_metadata.dlt_pipeline_id = p.pipeline_id<br>       AND u.workspace_id = p.workspace_id<br>    WHERE u.billing_origin_product = &#39;DLT&#39;<br>      AND u.record_type = &#39;ORIGINAL&#39;<br>      AND u.usage_date &gt;= current_date() - INTERVAL 7 DAYS<br>    GROUP BY p.pipeline_id, p.name<br>)<br>SELECT<br>    c.pipeline_name,<br>    ROUND(c.total_cost, 2) AS total_cost,<br>    SUM(CASE WHEN e.level = &#39;ERROR&#39; THEN 1 ELSE 0 END) AS error_events<br>FROM pipeline_cost c<br>JOIN system.lakeflow_pipeline_events_preview.pipeline_events e<br>    ON e.pipeline_id = c.pipeline_id<br>WHERE e.event_time &gt;= current_timestamp() - INTERVAL 7 DAYS<br>GROUP BY c.pipeline_name, c.total_cost<br>ORDER BY c.total_cost DESC<br>LIMIT 5;</pre><p>My top five looked like this:</p><pre>pipeline_name      total_cost  error_events<br>[prod] pipeline-d  332.33      0<br>[prod] pipeline-e  303.11      0<br>[prod] pipeline-f  157.03      0<br>[prod] pipeline-g  144.00      0<br>[dev] pipeline-h   98.64       50</pre><p>The four most expensive pipelines had zero errors.</p><p>The fifth was a dev pipeline with <strong>$98.64 in list-price compute and 50 error events in a week</strong>.</p><p>The cost was already in billing. The failures were already in the logs. Putting them together made that dev pipeline much harder to overlook.</p><h3>Read the state changes inside flow_progress</h3><p>An error count tells you how much noise a flow produced. Ordering its flow_progress events tells you more about how the run unfolded.</p><p>For each update and flow, take the current status and the next status, then count the pairs.</p><p>There is an important correction in the query below. My original version partitioned on flow_id alone. Some pipeline types have null flow_id values, which merged unrelated flows into one window and created transitions that never happened.</p><p>This version falls back to flow_name. It also excludes pipeline-level events and the missingFlowName placeholder.</p><pre>WITH s AS (<br>    SELECT<br>        update_id,<br>        COALESCE(origin.flow_id, origin.flow_name) AS flow_key,<br>        event_time,<br>        details:flow_progress:status::STRING AS status<br>    FROM system.lakeflow_pipeline_events_preview.pipeline_events<br>    WHERE event_type = &#39;flow_progress&#39;<br>      AND event_time &gt;= current_timestamp() - INTERVAL 7 DAYS<br>      AND details:flow_progress:status IS NOT NULL<br>      AND origin.flow_name IS NOT NULL<br>      AND origin.flow_name != &#39;pipelines.flowTimeMetrics.missingFlowName&#39;<br>),<br>t AS (<br>    SELECT<br>        update_id,<br>        flow_key,<br>        status AS from_status,<br>        LEAD(status) OVER (<br>            PARTITION BY update_id, flow_key<br>            ORDER BY event_time<br>        ) AS to_status<br>    FROM s<br>)<br>SELECT<br>    from_status,<br>    to_status,<br>    COUNT(*) AS transitions<br>FROM t<br>WHERE to_status IS NOT NULL<br>GROUP BY from_status, to_status<br>ORDER BY transitions DESC;</pre><p><strong>The exact transition counts are still under re-verification.</strong> The original write-up reported 26 distinct edges and specific failure-path counts using the earlier query. Those figures should not be treated as final.</p><p>The patterns below are useful directions for investigation, but the corrected query needs to confirm their frequency.</p><h3>Repeated status is mostly noise</h3><p>Pairs such as QUEUED → QUEUED and RUNNING → RUNNING mean the flow is announcing the same state again.</p><p>Raw event counts therefore reflect repeated reporting and time spent in a state, rather than throughput. To count actual state changes, keep only rows where:</p><pre>from_status &lt;&gt; to_status</pre><h3>Where a flow fails changes the investigation</h3><p>In the original results, failures from STARTING substantially outnumbered failures from RUNNING.</p><p>That points toward resolution, dependency, or permission problems before execution begins. It is a useful lead, although the exact ratio — and the cause of each failure — still needs confirmation.</p><h3>A failure followed by activity can reveal a retry</h3><p>Transitions such as FAILED → STARTING or FAILED → RUNNING make retry behavior visible without a dedicated retry field.</p><p>That is a stronger signal to inspect than a raw count of error rows.</p><h3>Unfamiliar transitions are worth reviewing</h3><p>Keep a baseline of the transitions your fleet normally produces.</p><p>An unseen edge, or one below a low count threshold, gives you a small set of events to review. Something like RUNNING → STOPPED or SKIPPED → PLANNING could reflect unusual behavior or a platform change.</p><p>It does not have to mean an incident. It does mean something changed.</p><h3>What if the pipeline stops showing up?</h3><p>Everything so far assumes the pipeline ran.</p><p>The failure I am more concerned about is the one that produces nothing: no update, no flow event, and no error for an alert to match.</p><p>You can look for that by measuring each pipeline’s normal gap between updates, then comparing it with the time since its last update.</p><pre>WITH e AS (<br>    SELECT<br>        pipeline_id,<br>        event_time,<br>        LAG(event_time) OVER (<br>            PARTITION BY pipeline_id<br>            ORDER BY event_time<br>        ) AS prev_time<br>    FROM system.lakeflow_pipeline_events_preview.pipeline_events<br>    WHERE event_type = &#39;create_update&#39;<br>      AND event_time &gt;= current_timestamp() - INTERVAL 7 DAYS<br>),<br>gaps AS (<br>    SELECT<br>        pipeline_id,<br>        (<br>            unix_timestamp(event_time)<br>            - unix_timestamp(prev_time)<br>        ) / 3600.0 AS gap_hours<br>    FROM e<br>    WHERE prev_time IS NOT NULL<br>),<br>cadence AS (<br>    SELECT<br>        pipeline_id,<br>        COUNT(*) + 1 AS update_count,<br>        percentile_approx(gap_hours, 0.5) AS median_gap_hours<br>    FROM gaps<br>    GROUP BY pipeline_id<br>),<br>latest AS (<br>    SELECT<br>        pipeline_id,<br>        (<br>            unix_timestamp(current_timestamp())<br>            - unix_timestamp(MAX(event_time))<br>        ) / 3600.0 AS hours_since<br>    FROM e<br>    GROUP BY pipeline_id<br>)<br>SELECT<br>    c.pipeline_id,<br>    c.update_count,<br>    c.median_gap_hours,<br>    l.hours_since<br>FROM cadence c<br>JOIN latest l<br>    ON c.pipeline_id = l.pipeline_id<br>ORDER BY l.hours_since DESC;</pre><p>In my workspace, 123 pipelines had enough updates within seven days to derive a cadence.</p><p>Across the top results, the median gap was 6.0 hours, almost to the second. Yet the pipelines at the top had gone <strong>157 hours since their last update</strong> — roughly 26 times their usual gap.</p><p>There was no error to alert on.</p><p>Nothing had failed because nothing had run.</p><p>That is the value of measuring cadence. Each pipeline gives you a starting point for its own normal behavior. You can alert when the time since its last update exceeds a chosen multiple of that gap.</p><h3>Do not alert on every registered pipeline</h3><p>I also tried listing registered pipelines in system.lakeflow.pipelines that had no events.</p><p>That returned:</p><ul><li><strong>601</strong> registered pipelines.</li><li><strong>75</strong> active pipelines.</li><li><strong>526</strong> apparently “silent” pipelines.</li></ul><p>Taken at face value, that would have made a terrible alert.</p><p>The table includes pipeline history and entries across workspaces. Before using it to detect silence, filter to your workspace_id and to pipelines your team considers active.</p><p>Otherwise, you will flag pipelines that were never part of your operating fleet.</p><h3>What I would build first</h3><p>After working through these queries, I would start with three things:</p><ol><li><strong>Alert when an active pipeline is overdue.</strong> Use its median update gap as the baseline, then choose a multiple that fits its schedule.</li><li><strong>Review new state transitions.</strong> Keep a baseline of known edges and flag unfamiliar ones for investigation.</li><li><strong>Group incidents by </strong><strong>update_id.</strong> Start with unique event IDs, then use update-level grouping to avoid treating every error row as a separate incident.</li></ol><p>The overdue alert would come first for me.</p><p>I can investigate an error once I know it happened. A pipeline that quietly stops running is easier to miss — and an empty error dashboard does very little to help.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8ba40dc58a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Last Gap in system.* Just Closed: Lakeflow Pipeline Events]]></title>
            <link>https://medium.com/@nathansundararajan/the-last-gap-in-system-just-closed-lakeflow-pipeline-events-21d9ac2b8313?source=rss-ea0ea6c0f203------2</link>
            <guid isPermaLink="false">https://medium.com/p/21d9ac2b8313</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[databricks-lakeflow]]></category>
            <category><![CDATA[unity-catalog]]></category>
            <dc:creator><![CDATA[Nathansundararajan]]></dc:creator>
            <pubDate>Sat, 26 Sep 2026 23:27:44 GMT</pubDate>
            <atom:updated>2026-09-26T23:33:31.112Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqDLsxAlv6SLkH9keDzm7A.png" /></figure><h3>Lakeflow pipeline events just landed in system.* as a beta. What it fixes, how to turn it on, and a few ways it’ll quietly give you wrong answers if you’re not careful.</h3><p>If you’ve spent any time digging around a Databricks account, system tables have probably changed how you work. Billing lives in system.billing.usage. Lineage lives in system.access.table_lineage. Every query anyone runs ends up in system.query.history. Jobs have system.lakeflow.jobs and job_run_timeline.</p><p>Pipelines were the one thing missing. Not anymore.</p><h3>What was broken</h3><p>Every Lakeflow Declarative Pipeline keeps its own event log. You could get at it with the event_log table-valued function or by going to its storage location, but only one pipeline at a time. That&#39;s fine for debugging one pipeline and useless for a fleet. If you wanted to know which pipelines failed the most this month, you were building (and babysitting) your own job to union them all together.</p><p>There were also gaps you just couldn’t fix from the outside. If a cluster never came up, the startup events were gone, and those are usually the failures you care about most. Delete a pipeline and its history went with it. And since cost was in system.* but reliability wasn&#39;t, you couldn&#39;t join the two.</p><p>One thing I want to call out: if all you need is update-level pass/fail trends, system.lakeflow.pipeline_update_timeline already does that and it&#39;s simpler. The new table is about depth. It goes down to individual flows inside an update and has micro-batch metrics the update table never had. Use it when your question is &quot;which flow broke and why,&quot; not &quot;did the update succeed.&quot;</p><h3>What the new table gives you</h3><p>The table is system.lakeflow_pipeline_events_preview.pipeline_events. Per the <a href="https://docs.databricks.com/aws/en/admin/system-tables/pipeline-events">docs</a>, it:</p><ul><li>Captures every pipeline’s event log in one place, even when the cluster doesn’t start.</li><li>Covers all pipelines and updates you have access to.</li><li>Keeps data after a pipeline is deleted, for 13 months by default (configurable).</li></ul><h3>What the docs say vs. what I actually checked</h3><p>Central capture, no per-pipeline setup. Confirmed. 652,368 events across 125 pipelines after seven days, and I didn’t touch a single pipeline.</p><p>Cross-workspace coverage. Couldn’t test it. We only have one workspace_id. The schema has account_id and workspace_id though, so the join key is there if you have several.</p><p>13 months of retention. Can’t see that yet. The table only fills forward from when you turn it on, and my oldest event is 2026–09–08.</p><p>Survives pipeline deletion. Haven’t tested. I didn’t want to delete a pipeline just to find out.</p><p>The retention one matters most in practice. History starts the day you enable it, so every week you put it off is a week you won’t get back.</p><h3>Turning it on</h3><p>It’s an account-level preview, so you need an account admin:</p><ol><li>Go to Account settings from any workspace.</li><li>Click Previews.</li><li>Search for Lakeflow Pipeline Events System Table and turn it on.</li></ol><p>Your pipelines don’t change at all. Events just start showing up from then on.</p><p>Heads up: the beta doesn’t fully dedupe events yet. Filter down to unique event IDs before you trust any count.</p><h3>What’s in it</h3><ul><li>Identity: account_id, workspace_id, pipeline_id, update_id, pipeline_event_id</li><li>What happened: event_type, level (INFO, WARN, ERROR, METRICS), message, event_time</li><li>Where it happened: an origin struct with pipeline name, flow name, dataset name, cluster ID, catalog and schema</li><li>Payload: an error struct, plus a details column (VARIANT) whose shape changes with event_type</li><li>Something new: maturity_level (STABLE, EVOLVING, DEPRECATED), basically a per-event promise about how stable the schema is</li></ul><p>About half the rows are flow_progress (335,593 in everything I had so far). After that it&#39;s flow_definition, dataset_definition, dataset_life_cycle, stream_progress, update_progress, and then a long tail. Errors barely show up by volume, just 419 rows at level = &#39;ERROR&#39; out of 650K+. That&#39;s exactly why having them in something you can GROUP BY is such a big deal.</p><p>About flow_id: it’s null for some pipeline types (database table sync pipelines, for one). Inside a single pipeline, identify a flow by (flow_id, flow_name) together. A full refresh can change the flow_id, so flow_name is the thing that sticks. Across pipelines, add pipeline_id too, because flow names get reused.</p><p>Two things before you build anything on it:</p><ul><li>The schema name is going to change. It’s lakeflow_pipeline_events_preview in beta and becomes system.lakeflow.pipeline_events at GA. Put a view in front of it now, point everything at the view, and just update the view on GA day.</li><li>Read the docs, not just the announcement. It’s beta, and EVOLVING events can change shape on you.</li></ul><h3>Querying it without fooling yourself</h3><p>Everything below ran against a live Azure workspace on 2026–09–15: 125 pipelines, seven days, about 652K rows.</p><p>Gotcha #1: details is VARIANT, so a bad path gives you NULL instead of an error.</p><p>For flow_progress, the INFO rows have status and, once a flow completes, metrics.num_output_rows / num_upserted_rows / num_deleted_rows. The METRICS rows have executor_time_ms / executor_cpu_time_ms. Those live on different rows, so if you want both in one query you have to handle both levels.</p><pre>SELECT COUNT(*) AS flow_progress_rows,</pre><pre>COUNT(details:flow_progress:metrics:num_output_rows::BIGINT)  AS num_output_rows_nonnull,</pre><pre>COUNT(details:flow_progress:metrics:num_output_bytes::BIGINT) AS num_output_bytes_nonnull</pre><pre>FROM system.lakeflow_pipeline_events_preview.pipeline_events</pre><pre>WHERE event_type = &#39;flow_progress&#39;</pre><pre>AND event_time &gt;= current_timestamp() - INTERVAL 7 DAYS</pre><p>Result:</p><pre>flow_progress_rows  num_output_rows_nonnull  num_output_bytes_nonnull</pre><pre>335,561             27,766                   0</pre><p>num_output_rows was filled in on 27,766 rows. I spot-checked three against the Pipeline UI (206,965,340 / 13,222,758 / 1,698,774, which the UI shows as 207M / 13M / 1.7M) and they matched. num_output_bytes was null on every single row in our workspace. Don&#39;t take that as a rule though. Which metrics get populated depends on your pipeline type, so check it against your own stuff.</p><p>A typo in a VARIANT path also just returns NULL. Same symptom, no warning either way.</p><p>So my rule: before anything unattended depends on a VARIANT path, run COUNT(path) against rows you know should have a value.</p><p>Gotcha #2: the status values aren’t what you’d expect.</p><pre>SELECT DISTINCT details:flow_progress:status::STRING AS status</pre><pre>FROM system.lakeflow_pipeline_events_preview.pipeline_events</pre><pre>WHERE event_type = &#39;flow_progress&#39;</pre><p>I got ten: blank, COMPLETED, EXCLUDED, FAILED, PLANNING, QUEUED, RUNNING, SKIPPED, STARTING, STOPPED. If you treat &quot;not COMPLETED or FAILED&quot; as &quot;still running,&quot; you&#39;ll get STOPPED, SKIPPED and EXCLUDED wrong.</p><p>Gotcha #3: join on pipeline_id, never on the name.</p><p>Pipeline names get reused and renamed mid-month. Join on origin.pipeline_name and you&#39;ll double count in one place and drop rows in another, and nothing will error. Use pipeline_id (plus workspace_id once you have more than one workspace) and only pull the name in at the very end for display.</p><h3>Coming up in Part 2</h3><p>What to actually build on top of this: fleet-wide error triage, spotting retries, joining cost with reliability, and the state machine hiding inside flow_progress, including an alert that doesn&#39;t need a threshold at all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21d9ac2b8313" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Configure Genie one Home Page.]]></title>
            <link>https://medium.com/@nathansundararajan/how-to-configure-genie-one-home-page-0c6bd1c40e08?source=rss-ea0ea6c0f203------2</link>
            <guid isPermaLink="false">https://medium.com/p/0c6bd1c40e08</guid>
            <dc:creator><![CDATA[Nathansundararajan]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 15:34:37 GMT</pubDate>
            <atom:updated>2026-09-03T15:35:56.174Z</atom:updated>
            <content:encoded><![CDATA[<p>A quick walkthrough of Databricks Genie&#39;s homepage customization — and two things I&#39;d love to see added next</p><p>One small feature in Databricks Genie that makes a real difference for adoption: the Genie One homepage is fully customizable, and it only takes one click to get there. If you&#39;re rolling Genie out to a team that&#39;s never touched a natural-language analytics tool before, this five-minute setup step is worth doing before you send out the link.</p><p>Here&#39;s how it works.</p><p># Step 1: Open the homepage settings</p><p>From the Genie One homepage, hit Edit next to the homepage settings panel.</p><p>Genie One homepage with Edit button</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pD6G0VMXxDPbd95qkS4r6w.png" /></figure><p>That&#39;s the only entry point you need — everything else happens on the page it opens.</p><p># Step 2: Customize brand, logo, and welcome text</p><p>Clicking Edit opens the customization page, where three things are editable in one place: brand color, logo, and a markdown welcome message.</p><p>Customization page with color picker, logo section, and markdown text area</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Htfx8FnBkoJjL5FsgvE0rQ.png" /></figure><p>The markdown block is the part that matters most, and it&#39;s easy to underinvest in. A blank chat box is intimidating — most people don&#39;t know what a natural-language analytics assistant can actually answer until they&#39;ve seen a few good examples. A well-written welcome message closes that gap on day one.</p><p>The pattern I&#39;ve landed on for that text:</p><p>• Show 2–3 example questions specific to the space — generic examples don&#39;t build confidence the way domain-specific ones do</p><p>• One question at a time — compound asks get muddled answers, and that&#39;s the single biggest cause of &quot;this thing doesn&#39;t work&quot; complaints</p><p>• Name your dimensions explicitly — &quot;by branch,&quot; &quot;by customer segment,&quot; &quot;by region,&quot; rather than leaving grouping up to the model</p><p>• Give a time frame — &quot;trailing 30 days,&quot; &quot;this quarter,&quot; &quot;FY2026 YTD&quot;</p><p>• Filter explicitly — call out the segment, region, product, or category by name rather than assuming context</p><p>For a banking-industry space, that might look like:</p><p>• Loan approval rate by branch this quarter.</p><p>• Average account balance by customer segment.</p><p>• Flag transactions over $10,000 by region, trailing 30 days.</p><p>Small UX investment, real reduction in &quot;I don&#39;t know what to ask it&quot; friction.</p><p># Step 3: Save and verify</p><p>Once you hit save, anyone on the team can confirm the change by checking the Admin wiki button in the top bar — it&#39;s a quick way to spot-check that your updates went live.</p><p>Top bar with Admin wiki button visible in the Genie one main Page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*6I2Ll-KB8LFwBchOb1gTxw.png" /></figure><p># Two things I&#39;d love to see next</p><p>1. A configurable Admin wiki link. Right now the label and destination are fixed to &quot;Admin wiki.&quot; I&#39;d like to point that button at whatever doc or resource actually makes sense for a given space, with whatever label fits.</p><p>2. A footer notes area on the main Genie One homepage. A dedicated spot for team-wide links and announcements would round out the customization story — right now there&#39;s no equivalent home for content that isn&#39;t specific to one space.</p><p>Both are small asks, but they&#39;d close the gap between &quot;this space is customizable&quot; and &quot;this whole homepage is customizable.&quot;</p><p>---</p><p>If you&#39;re rolling out Databricks Genie to your team, don&#39;t skip the homepage setup — it&#39;s a five-minute step that changes how confidently people use the tool on day one.</p><p>#Databricks #DatabricksGenie #DataEngineering #AnalyticsEngineering #GenAI</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0c6bd1c40e08" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>